From daemon@optimus.ietf.org  Fri Jan  4 12:37:33 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA09265
	for <ipoverib-archive@odin.ietf.org>; Fri, 4 Jan 2002 12:37:33 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id MAA00693
	for ipoverib-archive@odin.ietf.org; Fri, 4 Jan 2002 12:37:34 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA00460;
	Fri, 4 Jan 2002 12:30:47 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id IAA22171
	for <ipoverib@optimus.ietf.org>; Fri, 4 Jan 2002 08:51:22 -0500 (EST)
Received: from CNRI.Reston.VA.US (localhost [127.0.0.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA01914;
	Fri, 4 Jan 2002 08:51:20 -0500 (EST)
Message-Id: <200201041351.IAA01914@ietf.org>
Mime-Version: 1.0
Content-Type: Multipart/Mixed; Boundary="NextPart"
To: IETF-Announce: ;
Cc: ipoverib@ietf.org
From: Internet-Drafts@ietf.org
Reply-to: Internet-Drafts@ietf.org
Date: Fri, 04 Jan 2002 08:51:20 -0500
Subject: [Ipoverib] I-D ACTION:draft-ietf-ipoib-architecture-00.txt
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the IP over InfiniBand Working Group of the IETF.

	Title		: IP over InfiniBand(IPoIB) Architecture
	Author(s)	: V. Kashyap
	Filename	: draft-ietf-ipoib-architecture-00.txt
	Pages		: 21
	Date		: 03-Jan-02
	
InfiniBand is a high speed, channel based interconnect between
systems and devices.
This memo presents an overview of the InfiniBand architecture.
It further describes the requirements and guidelines for the
transmission of IP over InfiniBand. The encapsulation of IP
over InfiniBand and the mechanism for IP address resolution on
IB fabrics will be described in separate documents.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-ipoib-architecture-00.txt

To remove yourself from the IETF Announcement list, send a message to 
ietf-announce-request with the word unsubscribe in the body of the message.

Internet-Drafts are also available by anonymous FTP. Login with the username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-ietf-ipoib-architecture-00.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt


Internet-Drafts can also be obtained by e-mail.

Send a message to:
	mailserv@ietf.org.
In the body type:
	"FILE /internet-drafts/draft-ietf-ipoib-architecture-00.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.
		
		
Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Multipart/Alternative; Boundary="OtherAccess"

--OtherAccess
Content-Type: Message/External-body;
	access-type="mail-server";
	server="mailserv@ietf.org"

Content-Type: text/plain
Content-ID:	<20020103142304.I-D@ietf.org>

ENCODING mime
FILE /internet-drafts/draft-ietf-ipoib-architecture-00.txt

--OtherAccess
Content-Type: Message/External-body;
	name="draft-ietf-ipoib-architecture-00.txt";
	site="ftp.ietf.org";
	access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID:	<20020103142304.I-D@ietf.org>

--OtherAccess--

--NextPart--




_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan  4 14:13:39 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA11604
	for <ipoverib-archive@odin.ietf.org>; Fri, 4 Jan 2002 14:13:39 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id OAA03773
	for ipoverib-archive@odin.ietf.org; Fri, 4 Jan 2002 14:13:41 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA03700;
	Fri, 4 Jan 2002 14:09:17 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA03670
	for <ipoverib@ns.ietf.org>; Fri, 4 Jan 2002 14:09:16 -0500 (EST)
Received: from rwcrmhc51.attbi.com (rwcrmhc51.attbi.com [204.127.198.38])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA11481
	for <ipoverib@ietf.org>; Fri, 4 Jan 2002 14:09:12 -0500 (EST)
Received: from spike ([12.224.213.32]) by rwcrmhc51.attbi.com
          (InterMail vM.4.01.03.27 201-229-121-127-20010626) with SMTP
          id <20020104190843.NPJH20119.rwcrmhc51.attbi.com@spike>
          for <ipoverib@ietf.org>; Fri, 4 Jan 2002 19:08:43 +0000
From: "Bill Swortwood" <billiam33@attbi.com>
To: <ipoverib@ietf.org>
Subject: Some comments RE:  [Ipoverib] Meeting minutes from the 12/10/01 meeting
Date: Fri, 4 Jan 2002 11:10:09 -0800
Message-ID: <HFEHJGAKENECEAHPNJECOEBNCBAA.billiam33@attbi.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0)
Importance: Normal
In-Reply-To: <HHEDIJGOEFLPCPNILMPAAEFLCEAA.bill@strahm.net>
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit


All -

 Apologies for not being able to make the last meeting. Thanks to Mr.
Jim Pinkerton for his fine notes.  A few comments on the proceedings:

 1) Exposure of the M_Key in the SM MIB.

 - I agree with Jeff Young that exposure of the M_Key is a concern.
The statement that "the M_Key is sent in the clear" is somehat of a
misnomer,
if one views an InfiniBand subnet more as a PCI bus or system backplane,
rather than a snoopable exposed network media - the M_Key is not
exposed outside of the subnet currently. As InfiniBand management
communications
are point to point switched, and SM communications are not allowed to be
routed outside of the subnet, while it is true the M_Key is unencrypted,
it is also currently unexposed.

Carl's need to be able to share the M_Key is also quite understandable.
If one is communicating between SMs using SNMP via a secured ethernet
management network, across a VPN, or some other methodologies, use of
SNMP communications seems as good as any secured XML or RPC mechanisim
for such exchange.

A possible compromise solution would be to add in a IsMKeyExposed boolean,
and allow the SM itself determine if it allows M_Key transferance outside
of the subnet.


 2) The device management MIB not having baseboard management.

- This is intentional. The InfiniBand device management class does not
include baseboard management either. Further, unless the object in
question is an InfiniBand module, full baseboard management is not
required. I will be more than happy to work with anyone wanting to
model InfiniBand baseboard management in an SNMP MIB. However, InfiniBand
device management does not require baseboard management support beyond
the VPD data as currently specified in Vol2 of of the IB specification.


Thanks to all working the MIBs !

-------------------------------
Bill Swortwood
(Office)(503) 690 8578
(CELL)  (503) 819-5281
--------------------------------


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan  4 23:35:14 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA20983
	for <ipoverib-archive@odin.ietf.org>; Fri, 4 Jan 2002 23:35:14 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id XAA18744
	for ipoverib-archive@odin.ietf.org; Fri, 4 Jan 2002 23:35:16 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id XAA18447;
	Fri, 4 Jan 2002 23:30:20 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id XAA18406
	for <ipoverib@ns.ietf.org>; Fri, 4 Jan 2002 23:30:18 -0500 (EST)
Received: from pdxpo.dsl-only.net (pdxpo.dsl-only.net [63.105.16.3])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA20596
	for <ipoverib@ietf.org>; Fri, 4 Jan 2002 23:30:15 -0500 (EST)
Received: from bonfire (unverified [63.105.18.201]) by pdxpo.dsl-only.net
 (Rockliffe SMTPRA 4.5.4) with SMTP id <B0008963438@pdxpo.dsl-only.net> for <ipoverib@ietf.org>;
 Fri, 4 Jan 2002 20:29:44 -0800
From: "Bill Strahm" <bill@strahm.net>
To: <ipoverib@ietf.org>
Subject: RE: [Ipoverib] Meeting minutes from the 12/10/01 meeting As ASCII text
Date: Fri, 4 Jan 2002 20:37:51 -0800
Message-ID: <HHEDIJGOEFLPCPNILMPAIEJACEAA.bill@strahm.net>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0)
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
In-Reply-To: <Roam.SIMC.2.0.6.1010006714.8270.nordmark@bebop.france>
Importance: Normal
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Two requests for this... so here are minutes from the Monday meeting in
ASCII text.

Thanks
Bill


Minutes from IPoIB meeting on 12/10/01.  Reported by Jim Pinkerton -
Microsoft Corp.


Bill reviewed the agenda.

Shawn has taken over from the original author for the IPoIB MIBS.
IANA has defined an IBMIB as experimental 117.The IB MIBs need to be
assigned hierarchically from there.

Bill Strahm asked whether IANA object Ids can be informally assigned or does
each need to go to IANA. Bill recommends Shawn be the designee for assigning
IANA MIB numbers. There were no objections.

Bill summarizes the IPoIB MIBS that are required:
	- Bill comments that the sample counters are not useful.
	- Channel adapter MIB
	- specify  baseboard management MIB
	- device and switch MIBs

Bill asked that the textual conventions be pulled out separately. This would
allow the conventions to be defined independently of the MIBs. Bill walked
briefly through the high level textual conventions to give a flavor for the
approach. Comment from the floor was that the language in RFC2578 is
"SHOULD" language, so if there is a good reason to do zero-based indexing
you can. The concern is that using a one-based offset doesn't match the
hardware. Thus zero-based indexing is now deemed "acceptable".

Bill then walked through the subnet interface MIB. Talked about the
iblfStatGroup and iblfVLTrafficGroup and the types of stats in each. Textual
convention MIB was removed from this document. Authors have transitioned on
this doc. Bill then walked through the main differences in the update to
the -00 version. Open issues:

- should 32-bit error counters be made 64-bit (counter wrap frequency
analysis)? Jeff Young claims that software stacks are implementing 64 bit
counters. Point was made that the analysis could be led by what is the
fastest rate that these counters can increment? RFC2578 has some insight
into this.

- consensus question on this mib - they are concerned about the level of
review that has occurred. Bill is not ready to call consensus on this
document because of the level of review. Bill believes the document is
close, but needs additional review.


Carl Yang presented the IB Subnet Management MIB.

He first reviewed the possible roles that SNMP can play in managing an IB
network. Missed first option. Second option is to enable device management
over SNMP by allowing the subnet manager to change the values. But not all
variables are defined through the current MIB (e.g. multicast group creation
and deletion). Third is to enable SNMP to configure and monitor IB subnets
by communicating with the SM. The third approach is the primary focus of
this MIB. Point was made that with the SMA MIB, only one variable is
actually writable.

One of the main differences between the SM MIB and the SMA MIB is that the
SMA only deals with one value of a variable whereas the SM MIB deals with a
series of values. There was some discussion of how to document this. One
suggestion was to document the MIB for the SMA, but use an index and only
allow the SMA to use index zero, for example. Then the SM can reference
this.

There were some questions on the exact transport mechanism. Does the SM
proxy this. Bill commented that he'd like to enable a range of
implementations. There was concern that there not be a requirement for an IB
out-of-band management solution. Bill pointed out that there are a variety
of solutions possible - from doing it on top of UDP on IB, to tunneled, to
many other solutions.

There is some concern about whether the management key should be exposed or
should it be a secret? Bill claims since the management key is sent in the
clear, it's not a crypto key, and thus it's okay to expose it. Bill asks for
consensus. Jeff Young was concerned. There was a proposal that the SNMP
interface could be to have the variable read/write, where the read always
returns a zero. Bill pointed out that it's fairly easy to snoop the value,
thus not clear it needs to be protected. Another statement pointed out that
if SNMP is used over the internet and this really is a sensitive value,
maybe it shouldn't be visible by SNMP. Jeff said that clearly there needs to
be a mechanism to set the M_KEY. Not clear that it needs to be through SNMP.

On the configuration table, the number for row status - what is the
persistence of this information? Walked through the three possibilities - no
persistence, persistence, and allowing another variable to control the
persistence. Bill asked the crowd on why we would want SM state to not be
persistent across. No one came up with a reason or objected (after some
discussion and a walk through of a few scenarios).

Carl continued to do a brief overview of the ID. Comment Switch Info
PortStateChange should be a Boolean - only one value is currently defined.
Bill asked whether exposing P_Keys is a problem. Jeff Young said he has the
opposite opinion. They'll take it off line.

There is a problem with the trap. <insert text here>. See RFC2576.

Next steps
	- modify to use IB MIB textual convention
	- can we consildate SMA MIAB and SM MIB?
	-	- some MIBS are identical, other than their index
	- add missing MIBs
		- ServiceRecord, LinkRecord, and RenageRecord
	- Security consdertions
	- more trap definitions
	- compliance statements
Bill comments that because of the outstanding issues, there needs to be at
least one more draft. This draft is currently an individual contribution.
Bill's concern is whether this MIB could be collapsed with the next MIB
(SMA) being presented (from Bill Swortwood).

Bill Strahm presented Bill Swortwood's SMA proposal.

This is a working group item. Next steps are:
	- creation of compliance/required/optional groups
		- HCAs normally do not need wswitch groups
	- Addition of some references
	- <other stuff>

Point was made that this MIB has the same problem around TRAPs as the last
one.
Bill points out that this does not allow management - it allows monitoring
the network because you can't set the values of the variables.

Question was asked why you wouldn't just do this through the SA? It was
pointed out that there was really two proposals on the table - Bill's
proposal and Carl's proposal. Bill's addresses the SMA, Carl's addresses the
SA and SM.

Bill outlined three options - keep it as it is, single MIB that has
compliance statements separately for SM and SMA, third is to have a single
SMA MIB, with an SM MIB that includes the SMA and extends it for SM unique
issues.  A fourth option is to leverage the existing IB Management
infrastructure to the SA and map that to SNMP.


Bill Strahm presented Bill Swortwood's Device management MIB.

Bill overviews the proposal. Focused on monitoring hosts with IB device
management agents. It is missing baseboard management.

Next steps:
	- Creation of compliance statements
	- Addition of some references
	- Any other rational changes

Bill Strahm presented Connection MIB.

This is just an initial draft. Missing security consideration section and
need better explanatory text. The purpose is to model IB connections, but
this is not directly modeled after connection manager.

The point was made that it should be either an "all or none" access to
sensitive tables. A partial controlled access is much more complex.

Questions:
	- Objects that aren't in the IB spec
	- Is this a WG item
	- Does this cover the WG Item for Channel adapter MIB - should we  rename
the WG item?

Bill stated his opinion that he'd prefer to keep the separate MIBs separate
because it allows focused discussion and focused iteration. It also allows
the initial MIBs to get out to last call faster.

There was a proposal to have an interim meeting co-located with the next
IBTA management-working group to get better attendance. It was noted that
the management-working group is currently meeting monthly. Another
possibility is to request the IBTA management-working group to co-locate at
the next IETF. Bill will formally ask the Management co-chairs to work this
out with him.


-----Original Message-----
From: Erik Nordmark [mailto:Erik.Nordmark@eng.sun.com]
Sent: Wednesday, January 02, 2002 1:25 PM
To: Bill Strahm
Subject: Re: [Ipoverib] Meeting minutes from the 12/10/01 meeting



Bill,

Do you have the minutes in the form of plain text instead of a word
document?
Much easier for me to read.

  Erik


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Sun Jan  6 05:17:33 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA22326
	for <ipoverib-archive@odin.ietf.org>; Sun, 6 Jan 2002 05:17:33 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id FAA17947
	for ipoverib-archive@odin.ietf.org; Sun, 6 Jan 2002 05:17:37 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id FAA17909;
	Sun, 6 Jan 2002 05:14:57 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id FAA17879
	for <ipoverib@optimus.ietf.org>; Sun, 6 Jan 2002 05:14:56 -0500 (EST)
Received: from e1.ny.us.ibm.com (e1.ny.us.ibm.com [32.97.182.101])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA22309;
	Sun, 6 Jan 2002 05:14:51 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e1.ny.us.ibm.com (8.9.3/8.9.3) with ESMTP id FAA332132;
	Sun, 6 Jan 2002 05:11:26 -0500
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g06AENe131786;
	Sun, 6 Jan 2002 05:14:23 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id g06AEMg16935;
	Sun, 6 Jan 2002 02:14:22 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g06AELN19368;
	Sun, 6 Jan 2002 02:14:21 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201061014.g06AELN19368@eng4.beaverton.ibm.com>
To: minutes@ietf.org, ipoverib@ietf.org
Date: Sun, 6 Jan 2002 02:14:21 -0800 (PST)
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Subject: [Ipoverib] IPoIB WG Slides
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

The slides presented at the IPoIB WG meeting at IETF-52 can be found at:

http://www.geocities.com/vivk36/IETF_files/ipoib_52_ietf.ppt

Vivek
-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Fri Jan 11 15:31:18 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA28529
	for <ipoverib-archive@odin.ietf.org>; Fri, 11 Jan 2002 15:31:17 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id PAA14954
	for ipoverib-archive@odin.ietf.org; Fri, 11 Jan 2002 15:31:20 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA14475;
	Fri, 11 Jan 2002 15:28:10 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA14441
	for <ipoverib@optimus.ietf.org>; Fri, 11 Jan 2002 15:28:08 -0500 (EST)
Received: from vieo.com (vieo.com [216.30.79.131])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA28343
	for <ipoverib@ietf.org>; Fri, 11 Jan 2002 15:28:03 -0500 (EST)
Received: (from root@localhost)
	by vieo.com (8.11.2/8.11.2) id g0BKRa530642
	for ipoverib@ietf.org; Fri, 11 Jan 2002 14:27:36 -0600 (CST)
	(envelope-from rronald@VIEO.com)
Received: from Fishhead (dhcp-10-2-15-151.vieo.com [10.2.15.151])
	by vieo.com (8.11.2/8.11.2) with SMTP id g0BKRZw30508
	for <ipoverib@ietf.org>; Fri, 11 Jan 2002 14:27:35 -0600 (CST)
	(envelope-from rronald@VIEO.com)
Message-ID: <00e501c19ade$5f2ff440$970f020a@Fishhead>
From: "Roger Ronald" <rronald@VIEO.com>
To: <ipoverib@ietf.org>
Date: Fri, 11 Jan 2002 14:27:20 -0600
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 5.50.4133.2400
x-mimeole: Produced By Microsoft MimeOLE V5.50.4133.2400
X-scanner: scanned by Inflex 0.1.5c+ on vieo.com
Content-Transfer-Encoding: 7bit
Subject: [Ipoverib] Bridging of  IP over IB
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Howdy,

I'll apologize for subscribing and immediately diving in to a heavy
duty topic. I've noticed text in the IP over IB Architecture draft and 
IPv4 over IB drafts that imply that bridging/IB routing of IP over IB is 
stuck until the IBTA makes changes. I don't believe that this needs to be true.

In "IB routing" between IB subnets, the same issues arise as with 
bridging IP over IB to Ethernet. While it's certainly not easy to do
this IB routing or Ethernet bridging, I don't believe that it's all that
different than other bridging problem. There are a few spec issues
that need resolution, but these aren't too tough. I'm convinced that
there is a need to allow such bridging.

To start, I will postulate a bridge that joins an Ethernet and an IB
fabric. To simplify, my hypothetical bridge will only be concerned with
IP traffic. To move traffic, it will rely on MAC layer addresses. 

Each packet that is forwarded through the bridge must have source 
and destination addresses replaced with addresses appropriate to
the destination side's address format (Ethernet devices cannot
be changed to recognize anything but 48 bit ULAs and IB devices
probably cannot be persuaded to deal with addresses other than
LIDs or GIDs. These addresses must be assigned by the bridge and 
remembered for subsequent use. They must be unique and have 
no possibility of colliding with addresses already in the address domain.

The fact that each side of the bridge has a well known broadcast 
(or multicast) address and a protocol for learning addresses 
allows resolving addresses so that the address resolution is known
to the requestor AND an address mapping can be established in the bridge.

Given address translation and packet format translation in the 
bridge for the MAC header, the balance of the bridging problem
is pretty well covered in the IEEE 802.1d spec.
Despite the address swizzling, the Spanning Tree Algorithm and
Protocol seems applicable and usable to prevent topology
loops (although there might need to be an extension to prevent
multiple IB subnets from using the same 48 bit ULAs and 
GIDs for the proxied addresses).

I have some additional input on the direct text of the two affected 
IP over IB internet drafts, but this email is already too long. I'll save them
for a bit. I will say that the only protocol changes to these documents
that I envision involves increasing the requirement for use of the GRH 
to cover bridged messages as well as the multicast address.

Roger Ronald
VIEO, Inc.



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Fri Jan 11 15:54:42 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA29223
	for <ipoverib-archive@odin.ietf.org>; Fri, 11 Jan 2002 15:54:42 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id PAA15227
	for ipoverib-archive@odin.ietf.org; Fri, 11 Jan 2002 15:54:45 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA15205;
	Fri, 11 Jan 2002 15:53:36 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA15175
	for <ipoverib@optimus.ietf.org>; Fri, 11 Jan 2002 15:53:35 -0500 (EST)
Received: from patan.sun.com (patan.Sun.COM [192.18.98.43])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA29213
	for <ipoverib@ietf.org>; Fri, 11 Jan 2002 15:53:31 -0500 (EST)
Received: from sunmail1.Sun.COM ([129.145.1.2])
	by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id NAA12069;
	Fri, 11 Jan 2002 13:53:31 -0700 (MST)
Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.89.50])
	by sunmail1.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1-Sun.COM.mod.2) with ESMTP id MAA00455;
	Fri, 11 Jan 2002 12:54:38 -0800 (PST)
Received: from taipei (taipei.Eng.Sun.COM [129.146.85.178])
	by jurassic.eng.sun.com (8.12.2.Beta5+Sun/8.12.2.Beta5) with SMTP id g0BKrSS3649220;
	Fri, 11 Jan 2002 12:53:28 -0800 (PST)
Message-Id: <200201112053.g0BKrSS3649220@jurassic.eng.sun.com>
Date: Fri, 11 Jan 2002 12:51:07 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Subject: Re: [Ipoverib] Bridging of  IP over IB
To: rronald@VIEO.com
Cc: ipoverib@ietf.org
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: yHDd90m0Wc2anHQH3cvn/g==
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.6_06 SunOS 5.8 sun4u sparc 
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Vivek and I are submitting a new draft (it should show up anytime now)
covering IPoIB link and multicast that removes the contraint you alluded
to below. We believe that IPoIB links can be defined abstractly without
requiring all the IB routing/bridging details. This separation allows
us to proceed and focus on IP in IETF while leaving all the incomplete
IB work for IBTA to handle. Your comments are much welcomed.

Jerry Chu
Sr. Staff Engineer
Solaris Networking
Sun Microsystems, Inc.
(650) 786-5146
 
>From: "Roger Ronald" <rronald@VIEO.com>
>To: <ipoverib@ietf.org>
>Date: Fri, 11 Jan 2002 14:27:20 -0600
>MIME-Version: 1.0
>Content-Transfer-Encoding: 7bit
>X-Priority: 3
>X-MSMail-Priority: Normal
>x-mimeole: Produced By Microsoft MimeOLE V5.50.4133.2400
>X-scanner: scanned by Inflex 0.1.5c+ on vieo.com
>Content-Transfer-Encoding: 7bit
>Subject: [Ipoverib] Bridging of  IP over IB
>X-Mailman-Version: 1.0
>List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
>X-BeenThere: ipoverib@ietf.org
>
>Howdy,
>
>I'll apologize for subscribing and immediately diving in to a heavy
>duty topic. I've noticed text in the IP over IB Architecture draft and 
>IPv4 over IB drafts that imply that bridging/IB routing of IP over IB is 
>stuck until the IBTA makes changes. I don't believe that this needs to be true.
>
>In "IB routing" between IB subnets, the same issues arise as with 
>bridging IP over IB to Ethernet. While it's certainly not easy to do
>this IB routing or Ethernet bridging, I don't believe that it's all that
>different than other bridging problem. There are a few spec issues
>that need resolution, but these aren't too tough. I'm convinced that
>there is a need to allow such bridging.
>
>To start, I will postulate a bridge that joins an Ethernet and an IB
>fabric. To simplify, my hypothetical bridge will only be concerned with
>IP traffic. To move traffic, it will rely on MAC layer addresses. 
>
>Each packet that is forwarded through the bridge must have source 
>and destination addresses replaced with addresses appropriate to
>the destination side's address format (Ethernet devices cannot
>be changed to recognize anything but 48 bit ULAs and IB devices
>probably cannot be persuaded to deal with addresses other than
>LIDs or GIDs. These addresses must be assigned by the bridge and 
>remembered for subsequent use. They must be unique and have 
>no possibility of colliding with addresses already in the address domain.
>
>The fact that each side of the bridge has a well known broadcast 
>(or multicast) address and a protocol for learning addresses 
>allows resolving addresses so that the address resolution is known
>to the requestor AND an address mapping can be established in the bridge.
>
>Given address translation and packet format translation in the 
>bridge for the MAC header, the balance of the bridging problem
>is pretty well covered in the IEEE 802.1d spec.
>Despite the address swizzling, the Spanning Tree Algorithm and
>Protocol seems applicable and usable to prevent topology
>loops (although there might need to be an extension to prevent
>multiple IB subnets from using the same 48 bit ULAs and 
>GIDs for the proxied addresses).
>
>I have some additional input on the direct text of the two affected 
>IP over IB internet drafts, but this email is already too long. I'll save them
>for a bit. I will say that the only protocol changes to these documents
>that I envision involves increasing the requirement for use of the GRH 
>to cover bridged messages as well as the multicast address.
>
>Roger Ronald
>VIEO, Inc.
>
>
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Fri Jan 11 16:07:00 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA29615
	for <ipoverib-archive@odin.ietf.org>; Fri, 11 Jan 2002 16:06:55 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id QAA16129
	for ipoverib-archive@odin.ietf.org; Fri, 11 Jan 2002 16:06:59 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA15972;
	Fri, 11 Jan 2002 16:03:58 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA15942
	for <ipoverib@optimus.ietf.org>; Fri, 11 Jan 2002 16:03:56 -0500 (EST)
Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA29523
	for <ipoverib@ietf.org>; Fri, 11 Jan 2002 16:03:51 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e31.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id QAA106172;
	Fri, 11 Jan 2002 16:00:49 -0500
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0BL3ki30708;
	Fri, 11 Jan 2002 14:03:46 -0700
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id g0BL3jg15494;
	Fri, 11 Jan 2002 13:03:45 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0BL3iZ10417;
	Fri, 11 Jan 2002 13:03:44 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201112103.g0BL3iZ10417@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] Bridging of  IP over IB
To: rronald@VIEO.com (Roger Ronald)
Date: Fri, 11 Jan 2002 13:03:43 -0800 (PST)
Cc: ipoverib@ietf.org
In-Reply-To: <00e501c19ade$5f2ff440$970f020a@Fishhead> from "Roger Ronald" at Jan 11, 2002 01:27:20 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Roger,
        Bridging/proxying for the subnet beyond will work but that is
        not the same as stating that IP subnets span multiple IB
        subnets. It is a specialised solution.

        To accommodate exactly what you have just said I have included
        the GID in the link layer address (see the slides I presented
        at IETF, http://www.geocities.com/vivk36/IETF_files/ipoib_52_ietf.ppt) 
        or see the first message of the thread 'IPoIB
        architecture/encapsulation'. 

        The 'IPoIB-bridge' responds to ARP requests with its own LID
        but the remote node's GID. Note that it is not an IB router
        that is responding though you can always put IPoIB component
        in your device. The IB router is the 'official' bridge between
        the IB subnets.

        The source can then send the packet out normally with the
        assuarance that the packet will be received by the peer after
        tunneling through this bridge. The use of GRH thus becomes a
        configuration parameter, set by the administrator, to ensure
        that the IPoIB stacks use the full GRH. This bridge will have
        to keep track of the multicast groups across the two subnets
        and pass the packets through on either side. The details are
        not in the domain of the IPoIB specificaiton. It all is an
        implementation and subnet setup choice of the administrator.

        Such a solution allows those that want across the IB subnet
        solutions without making things difficult for those that would
        just stay within an IB subnet. Of course, we could always
        define the MAC addresses as having a variable length - if 6
        bytes(48 bits) then it is LID,capability flags and the QPN. If
        22 bytes then it is LID+capability flags+QPN+GID.

        For a generic implementation we do need the necessary IB
        infrastructure for multicast support. Frankly though, I have
        not seen any compelling reason for IP to span multiple
        subnets. However, if it can be done we can do it when IB
        provides the necessary hooks. 

	Please do send your comments on the drafts.

Vivek



> 
> Howdy,
> 
> I'll apologize for subscribing and immediately diving in to a heavy
> duty topic. I've noticed text in the IP over IB Architecture draft and 
> IPv4 over IB drafts that imply that bridging/IB routing of IP over IB is 
> stuck until the IBTA makes changes. I don't believe that this needs to be true.
> 
> In "IB routing" between IB subnets, the same issues arise as with 
> bridging IP over IB to Ethernet. While it's certainly not easy to do
> this IB routing or Ethernet bridging, I don't believe that it's all that
> different than other bridging problem. There are a few spec issues
> that need resolution, but these aren't too tough. I'm convinced that
> there is a need to allow such bridging.
> 
> To start, I will postulate a bridge that joins an Ethernet and an IB
> fabric. To simplify, my hypothetical bridge will only be concerned with
> IP traffic. To move traffic, it will rely on MAC layer addresses. 
> 
> Each packet that is forwarded through the bridge must have source 
> and destination addresses replaced with addresses appropriate to
> the destination side's address format (Ethernet devices cannot
> be changed to recognize anything but 48 bit ULAs and IB devices
> probably cannot be persuaded to deal with addresses other than
> LIDs or GIDs. These addresses must be assigned by the bridge and 
> remembered for subsequent use. They must be unique and have 
> no possibility of colliding with addresses already in the address domain.
> 
> The fact that each side of the bridge has a well known broadcast 
> (or multicast) address and a protocol for learning addresses 
> allows resolving addresses so that the address resolution is known
> to the requestor AND an address mapping can be established in the bridge.
> 
> Given address translation and packet format translation in the 
> bridge for the MAC header, the balance of the bridging problem
> is pretty well covered in the IEEE 802.1d spec.
> Despite the address swizzling, the Spanning Tree Algorithm and
> Protocol seems applicable and usable to prevent topology
> loops (although there might need to be an extension to prevent
> multiple IB subnets from using the same 48 bit ULAs and 
> GIDs for the proxied addresses).
> 
> I have some additional input on the direct text of the two affected 
> IP over IB internet drafts, but this email is already too long. I'll save them
> for a bit. I will say that the only protocol changes to these documents
> that I envision involves increasing the requirement for use of the GRH 
> to cover bridged messages as well as the multicast address.
> 
> Roger Ronald
> VIEO, Inc.
> 
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Fri Jan 11 16:09:25 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA29693
	for <ipoverib-archive@odin.ietf.org>; Fri, 11 Jan 2002 16:09:25 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id QAA16210
	for ipoverib-archive@odin.ietf.org; Fri, 11 Jan 2002 16:09:28 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA16109;
	Fri, 11 Jan 2002 16:06:53 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA16079
	for <ipoverib@optimus.ietf.org>; Fri, 11 Jan 2002 16:06:51 -0500 (EST)
Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA29606
	for <ipoverib@ietf.org>; Fri, 11 Jan 2002 16:06:46 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id PAA71596;
	Fri, 11 Jan 2002 15:02:42 -0600
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0BL6jZ167570;
	Fri, 11 Jan 2002 16:06:45 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id g0BL6jg15558;
	Fri, 11 Jan 2002 13:06:45 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0BL6iH10574;
	Fri, 11 Jan 2002 13:06:44 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201112106.g0BL6iH10574@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] Bridging of  IP over IB
To: Jerry.Chu@eng.sun.com
Date: Fri, 11 Jan 2002 13:06:43 -0800 (PST)
Cc: rronald@VIEO.com, ipoverib@ietf.org
In-Reply-To: <200201112053.g0BKrSS3649220@jurassic.eng.sun.com> from "H.K. Jerry Chu" at Jan 11, 2002 11:51:07 AM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Oh yes! and I forgot to mention that as far as IPoIB multicast is
concerned, but for the scope bits, there is no other restriction in
defining it for spanning multiple IB subnets. However it will work
only if IB provides the necessary underlying infrastructure.

Vivek

> 
> Vivek and I are submitting a new draft (it should show up anytime now)
> covering IPoIB link and multicast that removes the contraint you alluded
> to below. We believe that IPoIB links can be defined abstractly without
> requiring all the IB routing/bridging details. This separation allows
> us to proceed and focus on IP in IETF while leaving all the incomplete
> IB work for IBTA to handle. Your comments are much welcomed.
> 
> Jerry Chu
> Sr. Staff Engineer
> Solaris Networking
> Sun Microsystems, Inc.
> (650) 786-5146
>  
> >From: "Roger Ronald" <rronald@VIEO.com>
> >To: <ipoverib@ietf.org>
> >Date: Fri, 11 Jan 2002 14:27:20 -0600
> >MIME-Version: 1.0
> >Content-Transfer-Encoding: 7bit
> >X-Priority: 3
> >X-MSMail-Priority: Normal
> >x-mimeole: Produced By Microsoft MimeOLE V5.50.4133.2400
> >X-scanner: scanned by Inflex 0.1.5c+ on vieo.com
> >Content-Transfer-Encoding: 7bit
> >Subject: [Ipoverib] Bridging of  IP over IB
> >X-Mailman-Version: 1.0
> >List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
> >X-BeenThere: ipoverib@ietf.org
> >
> >Howdy,
> >
> >I'll apologize for subscribing and immediately diving in to a heavy
> >duty topic. I've noticed text in the IP over IB Architecture draft and 
> >IPv4 over IB drafts that imply that bridging/IB routing of IP over IB is 
> >stuck until the IBTA makes changes. I don't believe that this needs to be true.
> >
> >In "IB routing" between IB subnets, the same issues arise as with 
> >bridging IP over IB to Ethernet. While it's certainly not easy to do
> >this IB routing or Ethernet bridging, I don't believe that it's all that
> >different than other bridging problem. There are a few spec issues
> >that need resolution, but these aren't too tough. I'm convinced that
> >there is a need to allow such bridging.
> >
> >To start, I will postulate a bridge that joins an Ethernet and an IB
> >fabric. To simplify, my hypothetical bridge will only be concerned with
> >IP traffic. To move traffic, it will rely on MAC layer addresses. 
> >
> >Each packet that is forwarded through the bridge must have source 
> >and destination addresses replaced with addresses appropriate to
> >the destination side's address format (Ethernet devices cannot
> >be changed to recognize anything but 48 bit ULAs and IB devices
> >probably cannot be persuaded to deal with addresses other than
> >LIDs or GIDs. These addresses must be assigned by the bridge and 
> >remembered for subsequent use. They must be unique and have 
> >no possibility of colliding with addresses already in the address domain.
> >
> >The fact that each side of the bridge has a well known broadcast 
> >(or multicast) address and a protocol for learning addresses 
> >allows resolving addresses so that the address resolution is known
> >to the requestor AND an address mapping can be established in the bridge.
> >
> >Given address translation and packet format translation in the 
> >bridge for the MAC header, the balance of the bridging problem
> >is pretty well covered in the IEEE 802.1d spec.
> >Despite the address swizzling, the Spanning Tree Algorithm and
> >Protocol seems applicable and usable to prevent topology
> >loops (although there might need to be an extension to prevent
> >multiple IB subnets from using the same 48 bit ULAs and 
> >GIDs for the proxied addresses).
> >
> >I have some additional input on the direct text of the two affected 
> >IP over IB internet drafts, but this email is already too long. I'll save them
> >for a bit. I will say that the only protocol changes to these documents
> >that I envision involves increasing the requirement for use of the GRH 
> >to cover bridged messages as well as the multicast address.
> >
> >Roger Ronald
> >VIEO, Inc.
> >
> >
> >
> >_______________________________________________
> >IPoverIB mailing list
> >IPoverIB@ietf.org
> >http://www.ietf.org/mailman/listinfo/ipoverib
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Fri Jan 11 16:17:32 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA29902
	for <ipoverib-archive@odin.ietf.org>; Fri, 11 Jan 2002 16:17:25 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id QAA16361
	for ipoverib-archive@odin.ietf.org; Fri, 11 Jan 2002 16:17:27 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA16335;
	Fri, 11 Jan 2002 16:16:24 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA16303
	for <ipoverib@optimus.ietf.org>; Fri, 11 Jan 2002 16:16:22 -0500 (EST)
Received: from vieo.com (vieo.com [216.30.79.131])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA29889
	for <ipoverib@ietf.org>; Fri, 11 Jan 2002 16:16:20 -0500 (EST)
Received: (from root@localhost)
	by vieo.com (8.11.2/8.11.2) id g0BLFrF56141
	for ipoverib@ietf.org; Fri, 11 Jan 2002 15:15:53 -0600 (CST)
	(envelope-from rronald@VIEO.com)
Received: from Fishhead (dhcp-10-2-15-151.vieo.com [10.2.15.151])
	by vieo.com (8.11.2/8.11.2) with SMTP id g0BLFow56013;
	Fri, 11 Jan 2002 15:15:50 -0600 (CST)
	(envelope-from rronald@VIEO.com)
Message-ID: <012e01c19ae5$1cc90950$970f020a@Fishhead>
From: "Roger Ronald" <rronald@VIEO.com>
To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Cc: <ipoverib@ietf.org>
References: <200201112053.g0BKrSS3649220@jurassic.eng.sun.com>
Subject: Re: [Ipoverib] Bridging of  IP over IB
Date: Fri, 11 Jan 2002 15:15:35 -0600
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 5.50.4133.2400
x-mimeole: Produced By Microsoft MimeOLE V5.50.4133.2400
X-scanner: scanned by Inflex 0.1.5c+ on vieo.com
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Great. I'm completely in accord with leaving those details
out of the spec. 

As long as the GID gets included when needed and
the current draft comments that claim it can't be done (section 3.1
in the IPv4 over IB and 3.3 in the IP over IB Architecture draft)
are cleaned up, the details can be documented elsewhere.

RR
----- Original Message ----- 
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
To: <rronald@vieo.com>
Cc: <ipoverib@ietf.org>
Sent: Friday, January 11, 2002 2:51 PM
Subject: Re: [Ipoverib] Bridging of IP over IB


> Vivek and I are submitting a new draft (it should show up anytime now)
> covering IPoIB link and multicast that removes the contraint you alluded
> to below. We believe that IPoIB links can be defined abstractly without
> requiring all the IB routing/bridging details. This separation allows
> us to proceed and focus on IP in IETF while leaving all the incomplete
> IB work for IBTA to handle. Your comments are much welcomed.
> 
> Jerry Chu
> Sr. Staff Engineer
> Solaris Networking
> Sun Microsystems, Inc.
> (650) 786-5146
>  
> >From: "Roger Ronald" <rronald@VIEO.com>
> >To: <ipoverib@ietf.org>
> >Date: Fri, 11 Jan 2002 14:27:20 -0600
> >MIME-Version: 1.0
> >Content-Transfer-Encoding: 7bit
> >X-Priority: 3
> >X-MSMail-Priority: Normal
> >x-mimeole: Produced By Microsoft MimeOLE V5.50.4133.2400
> >X-scanner: scanned by Inflex 0.1.5c+ on vieo.com
> >Content-Transfer-Encoding: 7bit
> >Subject: [Ipoverib] Bridging of  IP over IB
> >X-Mailman-Version: 1.0
> >List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
> >X-BeenThere: ipoverib@ietf.org
> >
> >Howdy,
> >
> >I'll apologize for subscribing and immediately diving in to a heavy
> >duty topic. I've noticed text in the IP over IB Architecture draft and 
> >IPv4 over IB drafts that imply that bridging/IB routing of IP over IB is 
> >stuck until the IBTA makes changes. I don't believe that this needs to be true.
> >
> >In "IB routing" between IB subnets, the same issues arise as with 
> >bridging IP over IB to Ethernet. While it's certainly not easy to do
> >this IB routing or Ethernet bridging, I don't believe that it's all that
> >different than other bridging problem. There are a few spec issues
> >that need resolution, but these aren't too tough. I'm convinced that
> >there is a need to allow such bridging.
> >
> >To start, I will postulate a bridge that joins an Ethernet and an IB
> >fabric. To simplify, my hypothetical bridge will only be concerned with
> >IP traffic. To move traffic, it will rely on MAC layer addresses. 
> >
> >Each packet that is forwarded through the bridge must have source 
> >and destination addresses replaced with addresses appropriate to
> >the destination side's address format (Ethernet devices cannot
> >be changed to recognize anything but 48 bit ULAs and IB devices
> >probably cannot be persuaded to deal with addresses other than
> >LIDs or GIDs. These addresses must be assigned by the bridge and 
> >remembered for subsequent use. They must be unique and have 
> >no possibility of colliding with addresses already in the address domain.
> >
> >The fact that each side of the bridge has a well known broadcast 
> >(or multicast) address and a protocol for learning addresses 
> >allows resolving addresses so that the address resolution is known
> >to the requestor AND an address mapping can be established in the bridge.
> >
> >Given address translation and packet format translation in the 
> >bridge for the MAC header, the balance of the bridging problem
> >is pretty well covered in the IEEE 802.1d spec.
> >Despite the address swizzling, the Spanning Tree Algorithm and
> >Protocol seems applicable and usable to prevent topology
> >loops (although there might need to be an extension to prevent
> >multiple IB subnets from using the same 48 bit ULAs and 
> >GIDs for the proxied addresses).
> >
> >I have some additional input on the direct text of the two affected 
> >IP over IB internet drafts, but this email is already too long. I'll save them
> >for a bit. I will say that the only protocol changes to these documents
> >that I envision involves increasing the requirement for use of the GRH 
> >to cover bridged messages as well as the multicast address.
> >
> >Roger Ronald
> >VIEO, Inc.
> >
> >
> >
> >_______________________________________________
> >IPoverIB mailing list
> >IPoverIB@ietf.org
> >http://www.ietf.org/mailman/listinfo/ipoverib
> 
> 


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Fri Jan 11 16:52:37 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA00912
	for <ipoverib-archive@odin.ietf.org>; Fri, 11 Jan 2002 16:52:32 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id QAA17662
	for ipoverib-archive@odin.ietf.org; Fri, 11 Jan 2002 16:52:34 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA17642;
	Fri, 11 Jan 2002 16:51:38 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA17610
	for <ipoverib@optimus.ietf.org>; Fri, 11 Jan 2002 16:51:37 -0500 (EST)
Received: from e32.co.us.ibm.com (e32.co.us.ibm.com [32.97.110.130])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA00883
	for <ipoverib@ietf.org>; Fri, 11 Jan 2002 16:51:34 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e32.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id QAA79728;
	Fri, 11 Jan 2002 16:48:36 -0500
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0BLpZi56436;
	Fri, 11 Jan 2002 14:51:35 -0700
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0BLpYV22670;
	Fri, 11 Jan 2002 13:51:34 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0BLpXj12128;
	Fri, 11 Jan 2002 13:51:33 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201112151.g0BLpXj12128@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] Bridging of  IP over IB
To: rronald@VIEO.com (Roger Ronald)
Date: Fri, 11 Jan 2002 13:51:32 -0800 (PST)
Cc: ipoverib@ietf.org
In-Reply-To: <012e01c19ae5$1cc90950$970f020a@Fishhead> from "Roger Ronald" at Jan 11, 2002 02:15:35 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

> 
> Great. I'm completely in accord with leaving those details
> out of the spec. 
> 
> As long as the GID gets included when needed and
> the current draft comments that claim it can't be done (section 3.1
> in the IPv4 over IB and 3.3 in the IP over IB Architecture draft)

        You need the LID to the bridge too. You need the GID of the
        host on the remote subnet. So LID+GID but the usage of the GRH
        cannot be mandated. It is upto the implementation and the
        administrator to set it up.

        See my earlier note on this subject. The architecture draft is
        correct in what it says - we can't implement IP subnets across
        IB subnets at this point. A specialised IPoIB-bridge is not
        part of the IB architecture. IB architecture has to inherently
        provide the necessary infrastructure.

        However the specification in the drafts is such that when the
        infrastrucuture comes into being things will work.

Vivek

> are cleaned up, the details can be documented elsewhere.
> 
> RR
> ----- Original Message ----- 
> From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
> To: <rronald@vieo.com>
> Cc: <ipoverib@ietf.org>
> Sent: Friday, January 11, 2002 2:51 PM
> Subject: Re: [Ipoverib] Bridging of IP over IB
> 
> 
> > Vivek and I are submitting a new draft (it should show up anytime now)
> > covering IPoIB link and multicast that removes the contraint you alluded
> > to below. We believe that IPoIB links can be defined abstractly without
> > requiring all the IB routing/bridging details. This separation allows
> > us to proceed and focus on IP in IETF while leaving all the incomplete
> > IB work for IBTA to handle. Your comments are much welcomed.
> > 
> > Jerry Chu
> > Sr. Staff Engineer
> > Solaris Networking
> > Sun Microsystems, Inc.
> > (650) 786-5146
> >  
> > >From: "Roger Ronald" <rronald@VIEO.com>
> > >To: <ipoverib@ietf.org>
> > >Date: Fri, 11 Jan 2002 14:27:20 -0600
> > >MIME-Version: 1.0
> > >Content-Transfer-Encoding: 7bit
> > >X-Priority: 3
> > >X-MSMail-Priority: Normal
> > >x-mimeole: Produced By Microsoft MimeOLE V5.50.4133.2400
> > >X-scanner: scanned by Inflex 0.1.5c+ on vieo.com
> > >Content-Transfer-Encoding: 7bit
> > >Subject: [Ipoverib] Bridging of  IP over IB
> > >X-Mailman-Version: 1.0
> > >List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
> > >X-BeenThere: ipoverib@ietf.org
> > >
> > >Howdy,
> > >
> > >I'll apologize for subscribing and immediately diving in to a heavy
> > >duty topic. I've noticed text in the IP over IB Architecture draft and 
> > >IPv4 over IB drafts that imply that bridging/IB routing of IP over IB is 
> > >stuck until the IBTA makes changes. I don't believe that this needs to be true.
> > >
> > >In "IB routing" between IB subnets, the same issues arise as with 
> > >bridging IP over IB to Ethernet. While it's certainly not easy to do
> > >this IB routing or Ethernet bridging, I don't believe that it's all that
> > >different than other bridging problem. There are a few spec issues
> > >that need resolution, but these aren't too tough. I'm convinced that
> > >there is a need to allow such bridging.
> > >
> > >To start, I will postulate a bridge that joins an Ethernet and an IB
> > >fabric. To simplify, my hypothetical bridge will only be concerned with
> > >IP traffic. To move traffic, it will rely on MAC layer addresses. 
> > >
> > >Each packet that is forwarded through the bridge must have source 
> > >and destination addresses replaced with addresses appropriate to
> > >the destination side's address format (Ethernet devices cannot
> > >be changed to recognize anything but 48 bit ULAs and IB devices
> > >probably cannot be persuaded to deal with addresses other than
> > >LIDs or GIDs. These addresses must be assigned by the bridge and 
> > >remembered for subsequent use. They must be unique and have 
> > >no possibility of colliding with addresses already in the address domain.
> > >
> > >The fact that each side of the bridge has a well known broadcast 
> > >(or multicast) address and a protocol for learning addresses 
> > >allows resolving addresses so that the address resolution is known
> > >to the requestor AND an address mapping can be established in the bridge.
> > >
> > >Given address translation and packet format translation in the 
> > >bridge for the MAC header, the balance of the bridging problem
> > >is pretty well covered in the IEEE 802.1d spec.
> > >Despite the address swizzling, the Spanning Tree Algorithm and
> > >Protocol seems applicable and usable to prevent topology
> > >loops (although there might need to be an extension to prevent
> > >multiple IB subnets from using the same 48 bit ULAs and 
> > >GIDs for the proxied addresses).
> > >
> > >I have some additional input on the direct text of the two affected 
> > >IP over IB internet drafts, but this email is already too long. I'll save them
> > >for a bit. I will say that the only protocol changes to these documents
> > >that I envision involves increasing the requirement for use of the GRH 
> > >to cover bridged messages as well as the multicast address.
> > >
> > >Roger Ronald
> > >VIEO, Inc.
> > >
> > >
> > >
> > >_______________________________________________
> > >IPoverIB mailing list
> > >IPoverIB@ietf.org
> > >http://www.ietf.org/mailman/listinfo/ipoverib
> > 
> > 
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Fri Jan 11 16:57:13 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA01125
	for <ipoverib-archive@odin.ietf.org>; Fri, 11 Jan 2002 16:57:08 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id QAA17872
	for ipoverib-archive@odin.ietf.org; Fri, 11 Jan 2002 16:57:10 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA17850;
	Fri, 11 Jan 2002 16:56:30 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA17801
	for <ipoverib@optimus.ietf.org>; Fri, 11 Jan 2002 16:56:28 -0500 (EST)
Received: from vieo.com (vieo.com [216.30.79.131])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA01098
	for <ipoverib@ietf.org>; Fri, 11 Jan 2002 16:56:25 -0500 (EST)
Received: (from root@localhost)
	by vieo.com (8.11.2/8.11.2) id g0BLtuH78265
	for ipoverib@ietf.org; Fri, 11 Jan 2002 15:55:56 -0600 (CST)
	(envelope-from rronald@VIEO.com)
Received: from Fishhead (dhcp-10-2-15-151.vieo.com [10.2.15.151])
	by vieo.com (8.11.2/8.11.2) with SMTP id g0BLtmK78178;
	Fri, 11 Jan 2002 15:55:48 -0600 (CST)
	(envelope-from rronald@VIEO.com)
Message-ID: <015601c19aea$b22839d0$970f020a@Fishhead>
From: "Roger Ronald" <rronald@VIEO.com>
To: "Vivek Kashyap" <kashyapv@us.ibm.com>
Cc: <ipoverib@ietf.org>
References: <200201112103.g0BL3iZ10417@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] Bridging of  IP over IB
Date: Fri, 11 Jan 2002 15:55:33 -0600
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 5.50.4133.2400
x-mimeole: Produced By Microsoft MimeOLE V5.50.4133.2400
X-scanner: scanned by Inflex 0.1.5c+ on vieo.com
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit


----- Original Message ----- 
From: "Vivek Kashyap" <kashyapv@us.ibm.com>
To: "Roger Ronald" <rronald@vieo.com>

<snip> Please do send your comments on the drafts.
> 
> Vivek

1) In your IPv4 over IB draft, section 3.1, I suggest deleting the last two paragraphs:

       "The process is complex and unnecessary. The additional
        complexity introduced by IPv4 subnets spanning IB subnets
        gains very little if anything.

        IPv4 subnets therefore MUST be fully contained within an IB
        subnet."

2) In the ARP hardware address format, (section 3.2.2), I would prefer GID to GUID.
This would change 3.2.2.4 as well (the description of GUID). GID is more
complete, if you are crossing IB subnet boundaries the additional data is
useful (but not essential since you can capture the inbound port #).

3) In section 4.0, the text that reads:
        "The InfiniBand specification requires the use of GRH when
        multicasting. A unicast packet need not include the GRH. It is
        RECOMMENDED that a GRH be included in the interest of symmetry.
        It is RECOMMENDED that for unicast packets the destination GID
        be formed from the GUID in the ARP cache and the local IB
        subnet prefix. The GID at GID index 0 MUST be used as the
        source GID.
should be changed to either require the GRH at all times (my favorite) 
or changed to read:
       "The InfiniBand specification requires the use of GRH when
        multicasting. Any unicast packet that will be bridged or IB
        routed also requires the GRH. A unicast packet destined for the local
        IB fabric need not include the GRH. In the interest of symmetry, 
        it is RECOMMENDED that a GRH be included for unicast packets 
        destined for the local  IB fabric by forming the destination GID 
        from the GUID in  the ARP cache and the local IB subnet prefix. The 
        GID at  GID index 0 MUST be used as the source GID."

4) In the IP over IB Architecture document, section 3.3 reads:
       "3.3 IP subnets across IB subnets ?

        Some implementations may desire to support multiple clusters
        of machines in their own IB subnets but otherwise part of a
        common IP subnet. For such a solution the IB specification
        needs multiple upgrades:

        1) A method for creating IB multicast GIDs that span multiple
           IB subntes. The partition keys and other parameters need to
           be consistent across IB subnets.

        2) Develop IB routing protocol to determine the IB topology
           across IB subnets.

        3) Define the process and protocols needed between IB nodes
           and IB routers

        Until the above conditions are met it is not possible to
        define IPoIB subnets that span IB subnets. The IPoIB
        architecture however is capable of providing IP subnets across
        IB subnets if the underlying IB fabric provides the
        infrastructure.

I would delete the above section or (at least) remove the text that
claims the solution must be provided by changing the IBTA
specification. I don't believe that the IB specification
provides the answers, but I don't think they prevent a solution
either. The IETF or the IEEE could choose to take this on or a
proprietary solution could work. 

As to list item #1, there needs to be a method for supporting multicast 
across IB subnets, but partition keys and other parameters might 
NOT be consistent. Instead, they might be substituted as appropriate 
by a bridging device.

The first sentence of that last paragraph sounds like a challenge :-)

RR




_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Fri Jan 11 17:07:45 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA01460
	for <ipoverib-archive@odin.ietf.org>; Fri, 11 Jan 2002 17:07:45 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id RAA18706
	for ipoverib-archive@odin.ietf.org; Fri, 11 Jan 2002 17:07:47 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA18637;
	Fri, 11 Jan 2002 17:06:47 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA18599
	for <ipoverib@optimus.ietf.org>; Fri, 11 Jan 2002 17:06:45 -0500 (EST)
Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA01407
	for <ipoverib@ietf.org>; Fri, 11 Jan 2002 17:06:42 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id QAA68982;
	Fri, 11 Jan 2002 16:02:39 -0600
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0BM6iZ136898;
	Fri, 11 Jan 2002 17:06:44 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0BM6gV23086;
	Fri, 11 Jan 2002 14:06:43 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0BM6gj12883;
	Fri, 11 Jan 2002 14:06:42 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201112206.g0BM6gj12883@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] Bridging of  IP over IB
To: rronald@VIEO.com (Roger Ronald)
Date: Fri, 11 Jan 2002 14:06:41 -0800 (PST)
Cc: kashyapv@us.ibm.com (Vivek Kashyap), ipoverib@ietf.org
In-Reply-To: <015601c19aea$b22839d0$970f020a@Fishhead> from "Roger Ronald" at Jan 11, 2002 02:55:33 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

> 
> 
> ----- Original Message ----- 
> From: "Vivek Kashyap" <kashyapv@us.ibm.com>
> To: "Roger Ronald" <rronald@vieo.com>
> 
> <snip> Please do send your comments on the drafts.
> > 
> > Vivek
> 
> 1) In your IPv4 over IB draft, section 3.1, I suggest deleting the last two paragraphs:
> 
>        "The process is complex and unnecessary. The additional
>         complexity introduced by IPv4 subnets spanning IB subnets
>         gains very little if anything.
> 
>         IPv4 subnets therefore MUST be fully contained within an IB
>         subnet."
> 

        The draft is no longer under consideration. However, the
        process would be quite a bit difficult. In the newer draft
        there in no discussion of this issue except in the
        architecture draft that we just discussed.

> 2) In the ARP hardware address format, (section 3.2.2), I would prefer GID to GUID.

	See my previous mail. I have done that.


> This would change 3.2.2.4 as well (the description of GUID). GID is more
> complete, if you are crossing IB subnet boundaries the additional data is
> useful (but not essential since you can capture the inbound port #).
> 
> 3) In section 4.0, the text that reads:
>         "The InfiniBand specification requires the use of GRH when
>         multicasting. A unicast packet need not include the GRH. It is
>         RECOMMENDED that a GRH be included in the interest of symmetry.
>         It is RECOMMENDED that for unicast packets the destination GID
>         be formed from the GUID in the ARP cache and the local IB
>         subnet prefix. The GID at GID index 0 MUST be used as the
>         source GID.
> should be changed to either require the GRH at all times (my favorite) 
> or changed to read:
>        "The InfiniBand specification requires the use of GRH when
>         multicasting. Any unicast packet that will be bridged or IB
>         routed also requires the GRH. A unicast packet destined for the local
>         IB fabric need not include the GRH. In the interest of symmetry, 
>         it is RECOMMENDED that a GRH be included for unicast packets 
>         destined for the local  IB fabric by forming the destination GID 
>         from the GUID in  the ARP cache and the local IB subnet prefix. The 
>         GID at  GID index 0 MUST be used as the source GID."

        IT has been decided to leave it to the implementation to use
        or not use GRH. However, I would like to recommend using the
        GID formed by the IB subnet prefix and the GUID in the ARP/ND
        packets. 

> 
> 4) In the IP over IB Architecture document, section 3.3 reads:
>        "3.3 IP subnets across IB subnets ?
> 
>         Some implementations may desire to support multiple clusters
>         of machines in their own IB subnets but otherwise part of a
>         common IP subnet. For such a solution the IB specification
>         needs multiple upgrades:
> 
>         1) A method for creating IB multicast GIDs that span multiple
>            IB subntes. The partition keys and other parameters need to
>            be consistent across IB subnets.
> 
>         2) Develop IB routing protocol to determine the IB topology
>            across IB subnets.
> 
>         3) Define the process and protocols needed between IB nodes
>            and IB routers
> 
>         Until the above conditions are met it is not possible to
>         define IPoIB subnets that span IB subnets. The IPoIB
>         architecture however is capable of providing IP subnets across
>         IB subnets if the underlying IB fabric provides the
>         infrastructure.
> 
> I would delete the above section or (at least) remove the text that
> claims the solution must be provided by changing the IBTA
> specification. I don't believe that the IB specification
> provides the answers, but I don't think they prevent a solution
> either. The IETF or the IEEE could choose to take this on or a
> proprietary solution could work. 

	See my earlier response to your mail.

> 
> As to list item #1, there needs to be a method for supporting multicast 
> across IB subnets, but partition keys and other parameters might 
> NOT be consistent. Instead, they might be substituted as appropriate 
> by a bridging device.

        A bridging device is a non-IB solution. IP over IB is
        concerned with the IB specification. In IB the P_Keys etc.
        have to be kept consistent.

Vivek

> 
> The first sentence of that last paragraph sounds like a challenge :-)
> 
> RR
> 
> 
> 
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Fri Jan 11 19:45:15 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA03851
	for <ipoverib-archive@odin.ietf.org>; Fri, 11 Jan 2002 19:45:15 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id TAA22649
	for ipoverib-archive@odin.ietf.org; Fri, 11 Jan 2002 19:45:17 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA22525;
	Fri, 11 Jan 2002 19:33:44 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA22495
	for <ipoverib@optimus.ietf.org>; Fri, 11 Jan 2002 19:33:40 -0500 (EST)
Received: from e23.esmtp.ibm.com (e23.nc.us.ibm.com [32.97.136.229])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA03728
	for <ipoverib@ietf.org>; Fri, 11 Jan 2002 19:33:37 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e23.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id SAA44730
	for <ipoverib@ietf.org>; Fri, 11 Jan 2002 18:30:39 -0600
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0C0XcZ68906
	for <ipoverib@ietf.org>; Fri, 11 Jan 2002 19:33:38 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0C0XZV27657;
	Fri, 11 Jan 2002 16:33:35 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0C0XY600925;
	Fri, 11 Jan 2002 16:33:34 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201120033.g0C0XY600925@eng4.beaverton.ibm.com>
To: ipoverib@ietf.org
Date: Fri, 11 Jan 2002 16:33:33 -0800 (PST)
Cc: kashyapv@us.ibm.com
In-Reply-To: <200112072134.fB7LYVF02070@eng4.beaverton.ibm.com> from "Vivek Kashyap" at Dec 07, 2001 12:34:30 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Subject: [Ipoverib] IPoIB advanced capabilities
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Folks,
        We have discussed IP over non-UD links a few times on this
        mailing list.The capability-flags in the address-resolution
        packets provide the necessary hook to implementing such
        solutions in an interoperable way. I presented a solution at
        the IPoIB WG meeting at SLC earlier this month.

        Attached is a draft that I submitted a few days back. It
        hasn't been announced yet but I thought that we need to move
        forward. The draft and the solution is completely disjoint
        from the encapsulation draft except for the capability-bits
        being included in the address resolution packet.

	I look forward to your comments. 

Vivek

-------------------------------------------------------------------------------









INTERNET DRAFT                                                      V. Kashyap
<draft-kashyap-ipoib-advanced-capabilities-00.txt>                         IBM
Expiration Date: June 15, 2002                               December 15, 2001

                IP over InfiniBand(IPoIB): Advanced Capabilities


Status of this memo

        This document is an Internet-Draft and is in full conformance
        with all provisions of Section 10 of RFC 2026.

        Internet-Drafts are working documents of the Internet
        Engineering Task Force (IETF), its areas, and its working
        groups. Note that other groups may also distribute working
        documents as Internet- Drafts.

        Internet-Drafts are draft documents valid for a maximum of six
        months and may be updated, replaced, or obsoleted by other
        documents at any time. It is inappropriate to use
        Internet-Drafts as Reference material or to cite them other
        than as ``work in progress''.

        The list of current Internet-Drafts can be accessed at
        http://www.ietf.org/ietf/1id-abstracts.txt

        The list of Internet-Draft Shadow Directories can be accessed
        at http://www.ietf.org/shadow.html

        This memo provides information for the Internet community.
        This memo does not specify an Internet standard of any kind.
        Distribution of this memo is unlimited.

Copyright Notice

   Copyright (C) The Internet Society (2001).  All Rights Reserved.

Abstract

        InfiniBand Architecture(IBA) defines a high speed, channel
        based interconnect between systems and devices. The InfiniBand
        architecture provides multiple modes of transport services
        with differing characteristics. IPoIB may be implemented over
        any or multiple such modes to take advantage of the underlying
        mode's features. IPoIB by default is implemented over the UD
        mode of IBA. This document describes IPoIB over IBA's non-UD
        transport modes.



Kashyap                                                         [Page 1]





INTERNET-DRAFT              Advanced IPoIB             December 15, 2001


Table of Contents

        1.0     IPoIB advanced capability links
        2.0     Capability flags and interoperability
        3.0     Advanced capability IPoIB implementation
        3.1     IPoIB-QPN
        3.2     IPoIB-RC/UC/RD
        3.3     IP encapsulation
        4.0     Service ID
        4.1     Protocol specific subfield
        4.2     Service ID examples
        5.0     Security Considerations
        6.0     References
        7.0     Author's address

1.0 IPoIB advanced capability links

        IBA provides a broad set of capabilities to choose from when
        implementing IP over InfiniBand architecture.

        The datagram modes are limited by the link MTU. In contrast,
        the connected modes can offer significant benefits by offering
        a large MTU. Reliability is also enhanced if the underlying
        feature of automatic path migration of connected modes is
        utilised. An implementation MAY choose to provide IP over
        non-UD transport modes in addition to the mandatory IP over UD
        function.

        The InfiniBand architecture specification requires Unreliable
        Datagram mode to be supported by all the IB nodes. The host
        channel adapters(HCAs) are additionally required to support
        Reliable connected and Unreliable connected modes but not
        target channel adapters(TCAs). Support for the two Raw
        Datagram modes is entirely optional.

        For the sake of simplicity, ease of implementation and
        integration with existing stacks, it is desirable that the
        fabric support multicasting. This is possible only in
        Unreliable datagram (UD) and IB's Raw datagram modes.

        Given these conditions IPoIB over IBA's UD mode is mandatory.
        IPoIB over other modes of IB transport is optional. It is a
        MUST that IPoIB implementations with any combination of
        capabilities interoperate seamlessly.

        An interface MAY be associated with multiple QPNs. This
        provides a mode of implementation wherein a single IP address
        is associated with different QPNs. Such an association may be



Kashyap                                                         [Page 2]





INTERNET-DRAFT              Advanced IPoIB             December 15, 2001


        used to demultiplex the incoming packets in the HCA hardware,
        based on the QPN, avoiding or reducing the upper-layer port
        based lookup. This amounts to there being multiple MAC
        addresses associated with an endpoint. Any process for
        providing resolution and support of multiple QPNs per IP
        address must provide for interoperability with the default
        version of a single QPN per IPoIB interface. This document
        presents such a process.

        An IP implementation will therefore always support IPoIB over
        unreliable datagram mode (IPoIB-UD). It may choose to support
        IPoIB over the other modes. These modes will be referred to as
        IPoIB-RC(reliable connected), IPoIB-UC(unreliable connected),
        IPoIB-RD(reliable datagram).

        An implementation may also map different upper layer services
        to different QPNs while supporting IPoIB-UD. Such an
        implementation will be referred to as IPoIB-QPN.

        These capabilities are indicated to the other end by way of
        capability flags included in the address resolution
        packets[IPoIB_ENCAP]. Note that address resolution is always
        attempted over IPoIB-UD since that is the only mode that
        allows multicasting and is guaranteed to be supported by all
        IPoIB stacks.

2.0 Capability flags and interoperability

        [ Note to WG: The exact format of the link-layer address will
        depend on the IP encapsulation draft.]

        The link-layer address is defined as follows:

        +-------+-------+--------+-------+
        |      LID      |  Flags |  QPN  |
        +-------+-------+--------+-------+
        |       QPN     |     GID        |
        +-------+-------+--------+-------+
        |            GID                 |
        +-------+-------+--------+-------+
        |            GID                 |
        +-------+-------+--------+-------+
        |            GID                 |
        +-------+-------+--------+-------+
        |       GID     |
        +-------+-------+

        The flags are defined as follows:



Kashyap                                                         [Page 3]





INTERNET-DRAFT              Advanced IPoIB             December 15, 2001


        +-----+-----+-----+-----+---+---+---+---+
        | QPN | RC  | UC  | RD  | 0 | 0 | 0 | 0 |
        +-----+-----+-----+-----+---+---+---+---+

        QPN: Indicates IPoIB-QPN capability
        RC : Indicates IPoIB-RC capability
        UC : Indicates IPoIB-UC capability
        RD : Indicates IPoIB-RD capability

        All implementations of IPoIB MUST be able to interoperate.
        This is achieved by requiring an IPoIB stack to set only those
        flags that correspond to the capabilities it supports. A stack
        must ignore capability flags it doesn't support. Thus if a
        capability is not supported the rule is 'transmit as zero,
        ignore on receive'.

        Therefore the address resolution process between the peers
        informs them of each other's support of IP over specific modes
        of transport. The default case implementation i.e. IPoIB-UD
        only, ignores all the flags received and does not send any
        flags. Its peers therefore always communicate with it using
        IPoIB-UD.

3.0 Advanced capability IPoIB implementation

        The address resolution (ARP/ND) is always implemented over the
        IPoIB-UD mode. As noted earlier the capability flags are
        exchanged between the peers as part of the address
        resolution.

        It is an implementation decision to accept the capabilities
        offered by the peer. The method by which these decisions are
        made or conveyed to the IPoIB stack are beyond the scope of
        this document.

        Once the IP address is resolved the sender requests for the
        advanced capability QPN using the relevant InfiniBand
        connection or service ID resolution request. Once the relevant
        QPN is received the communication can continue over the
        specified QPN.

        It is an implemenation choice on how these connections are
        reflected to the user and how or whether these connections are
        reflected through a separate IPoIB interface.

3.1 IPoIB-QPN

        The IPoIB-QPN method of communication is implemented over the



Kashyap                                                         [Page 4]





INTERNET-DRAFT              Advanced IPoIB             December 15, 2001


        UD mode of IB. In IPoIB-QPN the two peers may associate
        separate QPNs with different services. The IB specification
        associates a 'serviceID' with the relevant service. This
        service ID is used in the IB specific Service ID
        resolution(SIDR) protocol. See the IB specification for more
        details[IB_ARCH]. See section 4.0 for the ServiceIDs IPoIB
        will use.

        The IB service ID relevant to IPoIB is derived from the
        protocol and the port or other information relevant to the
        protocol or the service.

        Thus the IPoIB-QPN interface resolves the relevant service ID
        and then uses the QPN so determined for all IP communication to
        the service.

        The QPN, which is returned in the address resolution protocol,
        may or may not be used for a particular service. The
        implementation must internally manage the multiple QPNs that
        may refer to the same IP address(but different upper layer
        protocol's ports).

        InfiniBand's SIDR protocol includes an application specific
        opaque block of data called the 'private data' field. This
        field MUST include the following information:

   +--------+--------+--------+--------+--------+--------+--------+--------+
   |  QPN received in ARP/ND  | Pad    | QPN send in ARP/ND       | Pad    |
   +--------+--------+--------+--------+--------+--------+--------+--------+

        The QPNs exchanged are the values that were received/sent
        during IP address resolution. These values will allow the
        recipient to disambiguate the IP interface for which the
        resolution is needed. The pad bytes must be set to 0.

        [ A note to WG. One could use the private data field to
        exchange large blocks/default QPN values but I don't think the
        gain is much.]

3.2 IPoIB-RC/UC/RD

        Similarly, as a result of the address resolution the two peers
        let the other know of their support of one or more of
        IPoIB-RC/UC/RD. The corresponding connection at the IB layer
        MAY then be setup using the communication establishment
        messages defined by IB. The method by which a host decides it
        wants to use the IPoIB-RC/UC/RD services is beyond the scope
        of this specification.



Kashyap                                                         [Page 5]





INTERNET-DRAFT              Advanced IPoIB             December 15, 2001


        The serviceID used in these messages is derived from the
        protocol and the port to be connected to. The serviceID is
        derived from these values as defined in section 4.0.

        Every communication establishment message includes an
        application specific 'private data' field. The private data
        field in IPoIB-RC/UC/RD is defined as follows:

   +--------+--------+--------+--------+--------+--------+--------+--------+
   |  QPN received in ARP/ND  |  Pad   |  QPN sent in ARP/ND      | Pad    |
   +--------+--------+--------+--------+--------+--------+--------+--------+
   | Maximum IP packet size(IP MTU)    |
   +--------+--------+--------+--------+

        The IB specification allows a message size of up to 2^31
        bytes. However the IPoIB implementations may set lower limits.
        This limit is an implementation choice based on the maximum
        packet the CA can accept and other factors.

        On receiving the IB connection request the peer may accept or
        reject the request based on IB rules and other implementation
        choices. If the connection is accepted the reply message(as
        per IB protocol) will include the MTU that the receiver
        prefers. It may be less, same or more than the MTU present in
        the request. The two peers however MUST use the smaller value
        as the MTU of the newly formed IPoIB-RC/UC/RD link.

        The remote peer may reject the connection due to InfiniBand
        related errors. If the connection is being rejected because
        the MTU value is required to be of a certain value then
        InfiniBand's reject (REJ) message is sent. The REJ message
        must include the MTU that would be acceptable and use the
        error code 3 (No resources available). In contrast if the
        requester doesn't find the resultant MTU suitable it can
        disconnect the IB connection.

3.3 IP encapsulation

        The IP encapsulation will be done as defined in the IPoIB
        encapsulation standard[IPoIB_ENCAP].

        Packets destined for any upper layer endpoint MAY be sent over
        any of the IPoIB links (UD/RC/RD/UC/QPN). Any IP packet
        received over IPoIB-UD link MUST be processed correctly by the
        receiving IPoIB even if there is an advanced capability link
        associated with that particular packet.

        IP multicast cannot be done over the advanced mode since it is



Kashyap                                                         [Page 6]





INTERNET-DRAFT              Advanced IPoIB             December 15, 2001


        inherently not supported. All Address Resolution Protocol(ARP)
        packets MUST only be exchanged over IPoIB-UD mode.

4.0 Service ID

        The InfiniBand specification defines a block of service IDs
        for IETF use. The InfiniBand specification has left the
        definition and management of this block to the IETF. The
        64-bit block is:

+--------+--------+--------+--------+--------+--------+--------+--------+
|00000001|<-------------------IETF use--------------------------------->|
+--------+--------+--------+--------+--------+--------+--------+--------+

        The ServiceIDs used by IPoIB will use the following format:

+--------+--------+--------+--------+--------+--------+--------+--------+
|00000001|Reserved|Reserved|Protocol|Reserved|Reserved|Protocol subfield|
+--------+--------+--------+--------+--------+--------+--------+--------+

        The Reserved fields MUST be transmitted as zeroes. They are
        ignored on reception. The Protocol field takes the values that
        may be included in the 'protocol' field of IPv4.

4.1 Protocol specific subfield

        The 16-bit field carries any information that uniquely
        identifies the protocol specific endpoint. For example this
        field will hold the TCP port numbers for TCP. This document
        specifies the protocol subfield for TCP/SCTP/UDP only. If any
        other value is found useful it will be described in a separate
        document.

        Rule 1: The 'protocol subfield' may be set to zero. This
                corresponds to the wild-card or 'all' case unless the
                relevant protocol description says otherwise.

        Rule 2: For the protocols of TCP/SCTP/UDP the protocol
                subfield corresponds to the port number associated
                with the service.


4.2 Service ID examples

        1) Service ID for TCP port 40

        The Protocol value for TCP is 6. Port 40 corresponds to 0x28.
        The service ID will therefore be:



Kashyap                                                         [Page 7]





INTERNET-DRAFT              Advanced IPoIB             December 15, 2001


        1:0:0:6:0:0:0:28

        2) Service ID for all UDP communication

        1:0:0:11:0:0:0:0

5.0 Security Considerations

        A node may be returned a false set of capability flags by an
        imposter. This may cause unnecessary attempts and some
        delay/disruption in IPoIB communication. An imposter can do
        more damage by replying with its own or otherwise wrong
        LID/GID. It is the responsibility of the higher layers or
        applications to implement suitable counter-measures if this is
        a problem.

6.0 References

[IB_ARCH]       InfiniBand Architecture Specification
[IPoIB_ARCH]    IPoIB Architecture
[IPoIB_ENCAP]   IP encapsulation over InfiniBand

7.0 Author's Address

Vivek Kashyap

15450, SW Koll Parkway
Beaverton, OR 97006

Phone: +1 503 578 3422
Email: vivk@us.ibm.com

Full Copyright Statement

        Copyright (C) The Internet Society (2001). All Rights Reserved.

        This document and translations of it may be copied and
        furnished to others, and derivative works that comment on or
        otherwise explain it or assist in its implementation may be
        prepared, copied, published and distributed, in whole or in
        part, without restriction of any kind, provided that the above
        copyright notice and this paragraph are included on all such
        copies and derivative works. However, this document itself may
        not be modified in any way, such as by removing the copyright
        notice or references to the Internet Society or other Internet
        organizations, except as needed for the purpose of developing
        Internet standards in which case the procedures for copyrights
        defined in the Internet Standards process must be followed, or



Kashyap                                                         [Page 8]





INTERNET-DRAFT              Advanced IPoIB             December 15, 2001


        as required to translate it into languages other than
        English.

        The limited permissions granted above are perpetual and will
        not be revoked by the Internet Society or its successors or
        assigns.

        This document and the information contained herein is provided
        on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET
        ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE
        USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR
        ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A
        PARTICULAR PURPOSE.





































Kashyap                                                         [Page 9]



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Fri Jan 11 21:11:39 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA04527
	for <ipoverib-archive@odin.ietf.org>; Fri, 11 Jan 2002 21:11:39 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id VAA24701
	for ipoverib-archive@odin.ietf.org; Fri, 11 Jan 2002 21:11:42 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA24684;
	Fri, 11 Jan 2002 21:10:48 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA23238
	for <ipoverib@optimus.ietf.org>; Fri, 11 Jan 2002 20:09:53 -0500 (EST)
Received: from mail2.microsoft.com (mail2.microsoft.com [131.107.3.124])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA04030
	for <ipoverib@ietf.org>; Fri, 11 Jan 2002 20:09:50 -0500 (EST)
Received: from INET-VRS-02.redmond.corp.microsoft.com ([157.54.8.110]) by mail2.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Fri, 11 Jan 2002 17:09:21 -0800
Received: from 157.54.8.109 by INET-VRS-02.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Fri, 11 Jan 2002 17:09:21 -0800
Received: from red-imc-04.redmond.corp.microsoft.com ([157.54.2.168]) by inet-hub-02.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Fri, 11 Jan 2002 17:09:21 -0800
Received: from win-imc-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.39]) by red-imc-04.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Fri, 11 Jan 2002 17:09:20 -0800
Received: from win-msg-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.52]) by win-imc-01.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3588.0);
	 Fri, 11 Jan 2002 17:07:20 -0800
X-MimeOLE: Produced By Microsoft Exchange V6.0.6115.0
Content-Class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/mixed;
	boundary="------------InterScan_NT_MIME_Boundary"
Subject: RE: [Ipoverib] IPoIB advanced capabilities
Date: Fri, 11 Jan 2002 17:07:20 -0800
Message-ID: <2E33960095B58E40A4D3345AB9F65EC10226183E@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
Thread-Topic: [Ipoverib] IPoIB advanced capabilities
thread-index: AcGbAIkEP3qkwf1gQq25QJC8xk/ODgAAqZ6A
From: "Arvind Murching" <arvindm@windows.microsoft.com>
To: "Vivek Kashyap" <kashyapv@us.ibm.com>, <ipoverib@ietf.org>
Cc: <Jerry.Chu@eng.sun.com>
X-OriginalArrivalTime: 12 Jan 2002 01:07:20.0554 (UTC) FILETIME=[7C4B5CA0:01C19B05]
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

This is a multi-part message in MIME format.

--------------InterScan_NT_MIME_Boundary
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C19B05.7C49D496"

------_=_NextPart_001_01C19B05.7C49D496
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Vivek, two issues regarding the capabilities flags:

(a) From a purist point of view, the link layer address is not the right
place to specify end-node capabilities.

(b) Section 3.2 says:

	  The method by which a host decides it wants to use
	  the IPoIB-RC/UC/RD services is beyond the scope
        of this specification.

However, this means that two nodes that conform to this specification,
and decide to use the capability bits, can easily fail to interoperate.
This is because this document does not specify how they, for example,
set up RC/UC connections (what service ID is used etc). We should either
fully specify the encapsulation and connection mechanisms for these
other transport methods (if time permits), or simply label the bits used
for this purpose as "Reserved" in this spec, and require all conformant
implementations to set this to 0 on transmit and ignore on receive.
Later versions of this spec can define these bits, along with details of
how they are used.

- Arvind

-----Original Message-----
From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]=20
Sent: Friday, January 11, 2002 4:34 PM
To: ipoverib@ietf.org
Cc: kashyapv@us.ibm.com
Subject: [Ipoverib] IPoIB advanced capabilities

Folks,
        We have discussed IP over non-UD links a few times on this
        mailing list.The capability-flags in the address-resolution
        packets provide the necessary hook to implementing such
        solutions in an interoperable way. I presented a solution at
        the IPoIB WG meeting at SLC earlier this month.

        Attached is a draft that I submitted a few days back. It
        hasn't been announced yet but I thought that we need to move
        forward. The draft and the solution is completely disjoint
        from the encapsulation draft except for the capability-bits
        being included in the address resolution packet.

	I look forward to your comments.=20

Vivek
<snip>

------_=_NextPart_001_01C19B05.7C49D496
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.0.6115.0">
<TITLE>RE: [Ipoverib] IPoIB advanced capabilities</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">Vivek,</FONT><FONT SIZE=3D2 FACE=3D"Courier =
New"></FONT></SPAN><SPAN LANG=3D"en-us"> <FONT SIZE=3D2 FACE=3D"Courier =
New">two issues</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Courier New"> regarding the capabilities =
flags</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">(a) From a purist point of view, the link layer address is not the =
right place to specify end-node capabilities.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">(</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">b</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">)</FONT> <FONT SIZE=3D2 FACE=3D"Courier New">S</FONT></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier New">ection 3.2 =
says:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN =
LANG=3D"en-us">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <FONT SIZE=3D2 =
FACE=3D"Courier New">&nbsp;</FONT></SPAN><SPAN LANG=3D"en-us"> <FONT =
SIZE=3D2 FACE=3D"Courier New">The method by which a host decides =
it</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New"></FONT></SPAN><SPAN LANG=3D"en-us"> <FONT SIZE=3D2 FACE=3D"Courier =
New">wants to use</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN =
LANG=3D"en-us">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <FONT SIZE=3D2 =
FACE=3D"Courier New">&nbsp;</FONT></SPAN><SPAN LANG=3D"en-us"> <FONT =
SIZE=3D2 FACE=3D"Courier New">the IPoIB-RC/UC/RD services is beyond the =
scope</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; of this =
specification.</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">However, this means that two</FONT></SPAN><SPAN LANG=3D"en-us"> =
<FONT SIZE=3D2 FACE=3D"Courier New">nodes</FONT></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier New"> that conform to =
this</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New"> specification,</FONT> <FONT SIZE=3D2 FACE=3D"Courier =
New">and</FONT> <FONT SIZE=3D2 FACE=3D"Courier New">decide to use the =
capability</FONT><FONT SIZE=3D2 FACE=3D"Courier New"> =
bits,</FONT></SPAN><SPAN LANG=3D"en-us"> <FONT SIZE=3D2 FACE=3D"Courier =
New">can easily</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Courier New"> fail to interoperate</FONT></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier New">. This =
is</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New"> because this document does not specify how</FONT></SPAN><SPAN =
LANG=3D"en-us"> <FONT SIZE=3D2 FACE=3D"Courier =
New">th</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">ey</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">, for example, set up RC/UC connections</FONT></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier New"> (what service ID is =
used etc)</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Courier New">.</FONT></SPAN><SPAN LANG=3D"en-us"> <FONT SIZE=3D2 =
FACE=3D"Courier New">We should either fully specify the =
encapsulation</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Courier New"> and</FONT></SPAN><SPAN LANG=3D"en-us"> <FONT =
SIZE=3D2 FACE=3D"Courier New">connection mechanisms for these other =
transport methods</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Courier New"> (if time permits)</FONT></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier New">, =
or</FONT></SPAN><SPAN LANG=3D"en-us"> <FONT SIZE=3D2 FACE=3D"Courier =
New">simply label the bits used for this purpose as &quot;Reserved&quot; =
in this spec, and require all conformant implementations to set this to =
0 on tr</FONT><FONT SIZE=3D2 FACE=3D"Courier New">ansmit and ignore on =
receive.</FONT></SPAN><SPAN LANG=3D"en-us"> <FONT SIZE=3D2 =
FACE=3D"Courier New">Later versions of this spec can define these bits, =
along with details of how they are used.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">- Arvind</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">-----Original Message-----<BR>
</FONT><FONT SIZE=3D2 FACE=3D"Courier New">From:</FONT><FONT SIZE=3D2 =
FACE=3D"Courier New"> Vivek Kashyap [<A =
HREF=3D"mailto:kashyapv@us.ibm.com">mailto:kashyapv@us.ibm.com</A>]<BR>
</FONT><FONT SIZE=3D2 FACE=3D"Courier New">Sent:</FONT><FONT SIZE=3D2 =
FACE=3D"Courier New"> Friday, January 11, 2002 4:34 PM<BR>
</FONT><FONT SIZE=3D2 FACE=3D"Courier New">To:</FONT><FONT SIZE=3D2 =
FACE=3D"Courier New"> ipoverib@ietf.org<BR>
</FONT><FONT SIZE=3D2 FACE=3D"Courier New">Cc:</FONT><FONT SIZE=3D2 =
FACE=3D"Courier New"> kashyapv@us.ibm.com<BR>
</FONT><FONT SIZE=3D2 FACE=3D"Courier New">Subject:</FONT><FONT SIZE=3D2 =
FACE=3D"Courier New"> [Ipoverib] IPoIB advanced =
capabilities</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">Folks,</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; We have discussed IP =
over non-UD links a few times on this</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; mailing list.The =
capability-flags in the address-resolution</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; packets provide the =
necessary hook to implementing such</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; solutions in an =
inter</FONT><FONT SIZE=3D2 FACE=3D"Courier New">operable way. I =
presented a solution at</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; the IPoIB WG meeting at =
SLC earlier this month.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Attached is a draft that =
I submitted a few days back. It</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; hasn't been announced =
yet but I thought that we need to move</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; forward. The =
draf</FONT><FONT SIZE=3D2 FACE=3D"Courier New">t and the solution is =
completely disjoint</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; from the encapsulation =
draft except for the capability-bits</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; being included in the =
address resolution packet.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN =
LANG=3D"en-us">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <FONT SIZE=3D2 =
FACE=3D"Courier New">I look forward to your comments. </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">Vivek</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">&lt;snip&gt;</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

</BODY>
</HTML>
------_=_NextPart_001_01C19B05.7C49D496--

--------------InterScan_NT_MIME_Boundary--



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Sat Jan 12 01:33:36 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA09166
	for <ipoverib-archive@odin.ietf.org>; Sat, 12 Jan 2002 01:33:36 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id BAA09157
	for ipoverib-archive@odin.ietf.org; Sat, 12 Jan 2002 01:33:37 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id BAA08931;
	Sat, 12 Jan 2002 01:30:27 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id BAA08901
	for <ipoverib@optimus.ietf.org>; Sat, 12 Jan 2002 01:30:25 -0500 (EST)
Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA09137
	for <ipoverib@ietf.org>; Sat, 12 Jan 2002 01:30:22 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e31.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id BAA76812
	for <ipoverib@ietf.org>; Sat, 12 Jan 2002 01:27:24 -0500
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0C6UNi135354
	for <ipoverib@ietf.org>; Fri, 11 Jan 2002 23:30:24 -0700
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0C6UMV05949
	for <ipoverib@ietf.org>; Fri, 11 Jan 2002 22:30:22 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0C6ULx13044
	for ipoverib@ietf.org; Fri, 11 Jan 2002 22:30:21 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201120630.g0C6ULx13044@eng4.beaverton.ibm.com>
Subject: RE: [Ipoverib] IPoIB advanced capabilities (fwd)
To: ipoverib@ietf.org
Date: Fri, 11 Jan 2002 22:30:20 -0800 (PST)
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Forwarding to mailing list. I erroneously sent it to a
non-existent address rather than to ipoverib@ietf.org.

Vivek


Forwarded message:
> From ARVINDM@windows.microsoft.com  Fri Jan 11 17:27:33 2002
> X-MimeOLE: Produced By Microsoft Exchange V6.0.6115.0
> Content-Class: urn:content-classes:message
> MIME-Version: 1.0
> Content-Type: text/plain;
> 	charset="us-ascii"
> Subject: RE: [Ipoverib] IPoIB advanced capabilities
> Date: Fri, 11 Jan 2002 17:25:27 -0800
> Message-ID: <2E33960095B58E40A4D3345AB9F65EC10353E1D3@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
> X-MS-Has-Attach: 
> X-MS-TNEF-Correlator: 
> Thread-Topic: [Ipoverib] IPoIB advanced capabilities
> thread-index: AcGbB2uP7LMdmQs8SI+CjzxrhR0N9AAAJkog
> From: "Arvind Murching" <arvindm@windows.microsoft.com>
> To: "Vivek Kashyap" <kashyapv@us.ibm.com>
> Cc: <ipoverib@us.ibm.com>
> X-OriginalArrivalTime: 12 Jan 2002 01:25:27.0758 (UTC) FILETIME=[045186E0:01C19B08]
> Content-Transfer-Encoding: 8bit
> X-MIME-Autoconverted: from quoted-printable to 8bit by eng4.beaverton.ibm.com id g0C1RXV03039
> 
> Okay, I should read this in more detail. I got the impression that the
> draft does not specify details of encapsulation/communication for the
> other modes.
> 
> Thanks
> Arvind
> 
> -----Original Message-----
> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com] 
> Sent: Friday, January 11, 2002 5:23 PM
> To: Arvind Murching
> Cc: ipoverib@us.ibm.com
> Subject: Re: [Ipoverib] IPoIB advanced capabilities
> 
>         No, service IDs are explained in section 4.0. If it is unclear
>         I can add more details.
> 
>         Wrt decision of the host I have said that if A and B want to
>         talk. A gets to know B has the capability and vice versa after
>         address resolution. However, it is A's (say the client)
>         decision on whether it talks to B (say the server) using a
>         particular mode. I don't think we should mandate a particular
>         choice- that is upto the application and the user.
> 
> Vivek
> 
> > 
> > This is a multi-part message in MIME format.
> > 
> > --------------InterScan_NT_MIME_Boundary
> > Content-Type: multipart/alternative;
> > 	boundary="----_=_NextPart_001_01C19B05.7C49D496"
> > 
> > ------_=_NextPart_001_01C19B05.7C49D496
> > Content-Type: text/plain;
> > 	charset="us-ascii"
> > Content-Transfer-Encoding: quoted-printable
> > 
> > Vivek, two issues regarding the capabilities flags:
> > 
> > (a) From a purist point of view, the link layer address is not the
> right
> > place to specify end-node capabilities.
> > 
> > (b) Section 3.2 says:
> > 
> > 	  The method by which a host decides it wants to use
> > 	  the IPoIB-RC/UC/RD services is beyond the scope
> >         of this specification.
> > 
> > However, this means that two nodes that conform to this specification,
> > and decide to use the capability bits, can easily fail to
> interoperate.
> > This is because this document does not specify how they, for example,
> > set up RC/UC connections (what service ID is used etc). We should
> either
> > fully specify the encapsulation and connection mechanisms for these
> > other transport methods (if time permits), or simply label the bits
> used
> > for this purpose as "Reserved" in this spec, and require all
> conformant
> > implementations to set this to 0 on transmit and ignore on receive.
> > Later versions of this spec can define these bits, along with details
> of
> > how they are used.
> > 
> > - Arvind
> > 
> > -----Original Message-----
> > From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]=20
> > Sent: Friday, January 11, 2002 4:34 PM
> > To: ipoverib@ietf.org
> > Cc: kashyapv@us.ibm.com
> > Subject: [Ipoverib] IPoIB advanced capabilities
> > 
> > Folks,
> >         We have discussed IP over non-UD links a few times on this
> >         mailing list.The capability-flags in the address-resolution
> >         packets provide the necessary hook to implementing such
> >         solutions in an interoperable way. I presented a solution at
> >         the IPoIB WG meeting at SLC earlier this month.
> > 
> >         Attached is a draft that I submitted a few days back. It
> >         hasn't been announced yet but I thought that we need to move
> >         forward. The draft and the solution is completely disjoint
> >         from the encapsulation draft except for the capability-bits
> >         being included in the address resolution packet.
> > 
> > 	I look forward to your comments.=20
> > 
> > Vivek
> > <snip>
> > 
> > ------_=_NextPart_001_01C19B05.7C49D496
> > Content-Type: text/html;
> > 	charset="us-ascii"
> > Content-Transfer-Encoding: quoted-printable
> > 
> > <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
> > <HTML>
> > <HEAD>
> > <META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
> > charset=3Dus-ascii">
> > <META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
> > 6.0.6115.0">
> > <TITLE>RE: [Ipoverib] IPoIB advanced capabilities</TITLE>
> > </HEAD>
> > <BODY>
> > <!-- Converted from text/rtf format -->
> > 
> > <P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
> > New">Vivek,</FONT><FONT SIZE=3D2 FACE=3D"Courier =
> > New"></FONT></SPAN><SPAN LANG=3D"en-us"> <FONT SIZE=3D2
> FACE=3D"Courier =
> > New">two issues</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
> > FACE=3D"Courier New"> regarding the capabilities =
> > flags</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier
> =
> > New">:</FONT></SPAN></P>
> > 
> > <P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
> > New">(a) From a purist point of view, the link layer address is not
> the =
> > right place to specify end-node capabilities.</FONT></SPAN></P>
> > 
> > <P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
> > New">(</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2
> FACE=3D"Courier =
> > New">b</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2
> FACE=3D"Courier =
> > New">)</FONT> <FONT SIZE=3D2 FACE=3D"Courier New">S</FONT></SPAN><SPAN
> =
> > LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier New">ection 3.2 =
> > says:</FONT></SPAN></P>
> > 
> > <P ALIGN=3DLEFT><SPAN =
> > LANG=3D"en-us">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <FONT
> SIZE=3D2 =
> > FACE=3D"Courier New">&nbsp;</FONT></SPAN><SPAN LANG=3D"en-us"> <FONT =
> > SIZE=3D2 FACE=3D"Courier New">The method by which a host decides =
> > it</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
> > New"></FONT></SPAN><SPAN LANG=3D"en-us"> <FONT SIZE=3D2
> FACE=3D"Courier =
> > New">wants to use</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN></P>
> > 
> > <P ALIGN=3DLEFT><SPAN =
> > LANG=3D"en-us">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <FONT
> SIZE=3D2 =
> > FACE=3D"Courier New">&nbsp;</FONT></SPAN><SPAN LANG=3D"en-us"> <FONT =
> > SIZE=3D2 FACE=3D"Courier New">the IPoIB-RC/UC/RD services is beyond
> the =
> > scope</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN></P>
> > 
> > <P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
> > New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; of this =
> > specification.</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN></P>
> > 
> > <P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
> > New">However, this means that two</FONT></SPAN><SPAN LANG=3D"en-us"> =
> > <FONT SIZE=3D2 FACE=3D"Courier New">nodes</FONT></SPAN><SPAN =
> > LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier New"> that conform to =
> > this</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier
> =
> > New"> specification,</FONT> <FONT SIZE=3D2 FACE=3D"Courier =
> > New">and</FONT> <FONT SIZE=3D2 FACE=3D"Courier New">decide to use the
> =
> > capability</FONT><FONT SIZE=3D2 FACE=3D"Courier New"> =
> > bits,</FONT></SPAN><SPAN LANG=3D"en-us"> <FONT SIZE=3D2
> FACE=3D"Courier =
> > New">can easily</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
> > FACE=3D"Courier New"> fail to interoperate</FONT></SPAN><SPAN =
> > LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier New">. This =
> > is</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
> > New"> because this document does not specify how</FONT></SPAN><SPAN =
> > LANG=3D"en-us"> <FONT SIZE=3D2 FACE=3D"Courier =
> > New">th</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2
> FACE=3D"Courier =
> > New">ey</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2
> FACE=3D"Courier =
> > New">, for example, set up RC/UC connections</FONT></SPAN><SPAN =
> > LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier New"> (what service ID
> is =
> > used etc)</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
> > FACE=3D"Courier New">.</FONT></SPAN><SPAN LANG=3D"en-us"> <FONT
> SIZE=3D2 =
> > FACE=3D"Courier New">We should either fully specify the =
> > encapsulation</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
> > FACE=3D"Courier New"> and</FONT></SPAN><SPAN LANG=3D"en-us"> <FONT =
> > SIZE=3D2 FACE=3D"Courier New">connection mechanisms for these other =
> > transport methods</FONT></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
> > FACE=3D"Courier New"> (if time permits)</FONT></SPAN><SPAN =
> > LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier New">, =
> > or</FONT></SPAN><SPAN LANG=3D"en-us"> <FONT SIZE=3D2 FACE=3D"Courier =
> > New">simply label the bits used for this purpose as
> &quot;Reserved&quot; =
> > in this spec, and require all conformant implementations to set this
> to =
> > 0 on tr</FONT><FONT SIZE=3D2 FACE=3D"Courier New">ansmit and ignore on
> =
> > receive.</FONT></SPAN><SPAN LANG=3D"en-us"> <FONT SIZE=3D2 =
> > FACE=3D"Courier New">Later versions of this spec can define these
> bits, =
> > along with details of how they are used.</FONT></SPAN></P>
> > 
> > <P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
> > New">- Arvind</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN></P>
> > 
> > <P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
> > New">-----Original Message-----<BR>
> > </FONT><FONT SIZE=3D2 FACE=3D"Courier New">From:</FONT><FONT SIZE=3D2
> =
> > FACE=3D"Courier New"> Vivek Kashyap [<A =
> >
> HREF=3D"mailto:kashyapv@us.ibm.com">mailto:kashyapv@us.ibm.com</A>]<BR>
> > </FONT><FONT SIZE=3D2 FACE=3D"Courier New">Sent:</FONT><FONT SIZE=3D2
> =
> > FACE=3D"Courier New"> Friday, January 11, 2002 4:34 PM<BR>
> > </FONT><FONT SIZE=3D2 FACE=3D"Courier New">To:</FONT><FONT SIZE=3D2 =
> > FACE=3D"Courier New"> ipoverib@ietf.org<BR>
> > </FONT><FONT SIZE=3D2 FACE=3D"Courier New">Cc:</FONT><FONT SIZE=3D2 =
> > FACE=3D"Courier New"> kashyapv@us.ibm.com<BR>
> > </FONT><FONT SIZE=3D2 FACE=3D"Courier New">Subject:</FONT><FONT
> SIZE=3D2 =
> > FACE=3D"Courier New"> [Ipoverib] IPoIB advanced =
> > capabilities</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN></P>
> > 
> > <P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
> > New">Folks,</FONT></SPAN></P>
> > 
> > <P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
> > New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; We have discussed IP =
> > over non-UD links a few times on this</FONT></SPAN></P>
> > 
> > <P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
> > New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; mailing list.The =
> > capability-flags in the address-resolution</FONT></SPAN></P>
> > 
> > <P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
> > New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; packets provide the =
> > necessary hook to implementing such</FONT></SPAN></P>
> > 
> > <P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
> > New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; solutions in an =
> > inter</FONT><FONT SIZE=3D2 FACE=3D"Courier New">operable way. I =
> > presented a solution at</FONT></SPAN></P>
> > 
> > <P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
> > New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; the IPoIB WG meeting
> at =
> > SLC earlier this month.</FONT></SPAN></P>
> > 
> > <P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
> > New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Attached is a draft
> that =
> > I submitted a few days back. It</FONT></SPAN></P>
> > 
> > <P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
> > New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; hasn't been announced
> =
> > yet but I thought that we need to move</FONT></SPAN></P>
> > 
> > <P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
> > New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; forward. The =
> > draf</FONT><FONT SIZE=3D2 FACE=3D"Courier New">t and the solution is =
> > completely disjoint</FONT></SPAN></P>
> > 
> > <P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
> > New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; from the encapsulation
> =
> > draft except for the capability-bits</FONT></SPAN></P>
> > 
> > <P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
> > New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; being included in the
> =
> > address resolution packet.</FONT></SPAN></P>
> > 
> > <P ALIGN=3DLEFT><SPAN =
> > LANG=3D"en-us">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <FONT
> SIZE=3D2 =
> > FACE=3D"Courier New">I look forward to your comments.
> </FONT></SPAN></P>
> > 
> > <P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
> > New">Vivek</FONT></SPAN></P>
> > 
> > <P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
> > New">&lt;snip&gt;</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN></P>
> > 
> > </BODY>
> > </HTML>
> > ------_=_NextPart_001_01C19B05.7C49D496--
> > 
> > --------------InterScan_NT_MIME_Boundary--
> > 
> > 
> 
> 
> -- 
> Vivek Kashyap
> Linux Technology Center, IBM
> kashyapv@us.ibm.com 
> vivk@us.ibm.com 
> 503 578 3422 (o)
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Sat Jan 12 11:15:23 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA21708
	for <ipoverib-archive@odin.ietf.org>; Sat, 12 Jan 2002 11:15:23 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id LAA20702
	for ipoverib-archive@odin.ietf.org; Sat, 12 Jan 2002 11:15:26 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id LAA20689;
	Sat, 12 Jan 2002 11:14:39 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id LAA20640
	for <ipoverib@optimus.ietf.org>; Sat, 12 Jan 2002 11:14:36 -0500 (EST)
Received: from mxout1.netvision.net.il (mxout1.netvision.net.il [194.90.9.20])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA21681
	for <ipoverib@ietf.org>; Sat, 12 Jan 2002 11:14:30 -0500 (EST)
Received: from vaio505he ([12.64.129.19]) by mxout1.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GPU00H5Q2F4OY@mxout1.netvision.net.il> for ipoverib@ietf.org;
 Sat, 12 Jan 2002 18:14:02 +0200 (IST)
Date: Sat, 12 Jan 2002 18:14:32 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: RE: [Ipoverib] IPoIB advanced capabilities
In-reply-to: 
 <2E33960095B58E40A4D3345AB9F65EC10226183E@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
To: "'Arvind Murching'" <arvindm@windows.microsoft.com>,
        "'Vivek Kashyap'" <kashyapv@us.ibm.com>, ipoverib@ietf.org
Cc: "'Jim Pinkerton'" <jpink@microsoft.com>
Reply-to: yaronh@voltaire.com
Message-id: <00fd01c19b84$396986d0$0100007f@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: multipart/alternative;
 boundary="Boundary_(ID_MH0w1LW5oe3PQ6xdgF91BA)"
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

This is a multi-part message in MIME format.

--Boundary_(ID_MH0w1LW5oe3PQ6xdgF91BA)
Content-type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7BIT

Arvind,
 
In our case the capability (RC/UD)  is part of the link layer attributes
The host could always use UD if they want to, however if both of them
support RC and for some reason (MTU ..) want to work RC they can
As far as I know Vivek has a proposal for how to set an RC connection, I'm
not sure if he posted it yet
 
Another capability bit I would like to add that would help is and "SDP"
capability bit
Since with the new proposal there is no need to go to the SM for ARP, in
order to know if a node supports SDP I must add an SM lookup to the process
That could slow the process significantly
If we add this bit we could avoid that and quite easily know if we could use
SDP for the connection
Since protocol offloading is an integral (non-optional) part of InfiniBand
it also makes sense to put it as part of the IB ARP
(not to mention that it is specified in SDP that it uses the IP over IB
address resolution)
 
Yaron
 
-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Arvind Murching
Sent: Saturday, January 12, 2002 3:07 AM
To: Vivek Kashyap; ipoverib@ietf.org
Cc: Jerry.Chu@eng.sun.com
Subject: RE: [Ipoverib] IPoIB advanced capabilities
 
Vivek, two issues regarding the capabilities flags:
(a) From a purist point of view, the link layer address is not the right
place to specify end-node capabilities.
(b) Section 3.2 says:
          The method by which a host decides it wants to use
          the IPoIB-RC/UC/RD services is beyond the scope
        of this specification.
However, this means that two nodes that conform to this specification, and
decide to use the capability bits, can easily fail to interoperate. This is
because this document does not specify how they, for example, set up RC/UC
connections (what service ID is used etc). We should either fully specify
the encapsulation and connection mechanisms for these other transport
methods (if time permits), or simply label the bits used for this purpose as
"Reserved" in this spec, and require all conformant implementations to set
this to 0 on transmit and ignore on receive. Later versions of this spec can
define these bits, along with details of how they are used.
- Arvind
-----Original Message-----
From: Vivek Kashyap [ mailto:kashyapv@us.ibm.com]
Sent: Friday, January 11, 2002 4:34 PM
To: ipoverib@ietf.org
Cc: kashyapv@us.ibm.com
Subject: [Ipoverib] IPoIB advanced capabilities
Folks,
        We have discussed IP over non-UD links a few times on this
        mailing list.The capability-flags in the address-resolution
        packets provide the necessary hook to implementing such
        solutions in an interoperable way. I presented a solution at
        the IPoIB WG meeting at SLC earlier this month.
        Attached is a draft that I submitted a few days back. It
        hasn't been announced yet but I thought that we need to move
        forward. The draft and the solution is completely disjoint
        from the encapsulation draft except for the capability-bits
        being included in the address resolution packet.
        I look forward to your comments.
Vivek
<snip>

--Boundary_(ID_MH0w1LW5oe3PQ6xdgF91BA)
Content-type: text/html; charset=us-ascii
Content-Transfer-Encoding: 7BIT

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns="http://www.w3.org/TR/REC-html40">

<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">


<meta name=ProgId content=Word.Document>
<meta name=Generator content="Microsoft Word 9">
<meta name=Originator content="Microsoft Word 9">
<link rel=File-List href="cid:filelist.xml@01C19B94.FB947480">
<title>RE: [Ipoverib] IPoIB advanced capabilities</title>
<!--[if gte mso 9]><xml>
 <o:OfficeDocumentSettings>
  <o:DoNotRelyOnCSS/>
 </o:OfficeDocumentSettings>
</xml><![endif]--><!--[if gte mso 9]><xml>
 <w:WordDocument>
  <w:Zoom>0</w:Zoom>
  <w:DocumentKind>DocumentEmail</w:DocumentKind>
  <w:EnvelopeVis/>
 </w:WordDocument>
</xml><![endif]-->
<style>
<!--
 /* Font Definitions */
@font-face
	{font-family:Tahoma;
	panose-1:2 11 6 4 3 5 4 4 2 4;
	mso-font-charset:0;
	mso-generic-font-family:swiss;
	mso-font-pitch:variable;
	mso-font-signature:553679495 -2147483648 8 0 66047 0;}
 /* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{mso-style-parent:"";
	margin:0in;
	margin-bottom:.0001pt;
	mso-pagination:widow-orphan;
	font-size:12.0pt;
	font-family:"Times New Roman";
	mso-fareast-font-family:"Times New Roman";}
a:link, span.MsoHyperlink
	{color:blue;
	text-decoration:underline;
	text-underline:single;}
a:visited, span.MsoHyperlinkFollowed
	{color:blue;
	text-decoration:underline;
	text-underline:single;}
p.MsoAutoSig, li.MsoAutoSig, div.MsoAutoSig
	{margin:0in;
	margin-bottom:.0001pt;
	mso-pagination:widow-orphan;
	font-size:12.0pt;
	font-family:"Times New Roman";
	mso-fareast-font-family:"Times New Roman";}
p
	{margin-right:0in;
	mso-margin-top-alt:auto;
	mso-margin-bottom-alt:auto;
	margin-left:0in;
	mso-pagination:widow-orphan;
	font-size:12.0pt;
	font-family:"Times New Roman";
	mso-fareast-font-family:"Times New Roman";}
span.EmailStyle18
	{mso-style-type:personal-reply;
	mso-ansi-font-size:10.0pt;
	mso-ascii-font-family:Arial;
	mso-hansi-font-family:Arial;
	mso-bidi-font-family:Arial;
	color:navy;}
@page Section1
	{size:8.5in 11.0in;
	margin:1.0in 1.25in 1.0in 1.25in;
	mso-header-margin:.5in;
	mso-footer-margin:.5in;
	mso-paper-source:0;
	mso-gutter-direction:rtl;}
div.Section1
	{page:Section1;}
-->
</style>
</head>

<body lang=EN-US link=blue vlink=blue style='tab-interval:.5in'>
<!-- Converted from text/rtf format -->

<div class=Section1 dir=RTL>

<p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
Arial'>Arvind,<o:p></o:p></span></font></span></p>

<p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
Arial'><![if !supportEmptyParas]>&nbsp;<![endif]><o:p></o:p></span></font></span></p>

<p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
Arial'>In our case the capability (RC/UD) <span style="mso-spacerun:
yes">&nbsp;</span>is part of the link layer attributes <o:p></o:p></span></font></span></p>

<p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
Arial'>The host could always use UD if they want to, however if both of them
support RC and for some reason (MTU ..) want to work RC they can<o:p></o:p></span></font></span></p>

<p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
Arial'>As far as I know Vivek has a proposal for how to set an RC connection, I&#8217;m
not sure if he posted it yet<o:p></o:p></span></font></span></p>

<p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
Arial'><![if !supportEmptyParas]>&nbsp;<![endif]><o:p></o:p></span></font></span></p>

<p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
Arial'>Another capability bit I would like to add that would help is and &#8220;SDP&#8221;
capability bit<o:p></o:p></span></font></span></p>

<p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
Arial'>Since with the new proposal there is no need to go to the SM for ARP, in
order to know if a node supports SDP I must add an SM lookup to the process<o:p></o:p></span></font></span></p>

<p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
Arial'>That could slow the process significantly<o:p></o:p></span></font></span></p>

<p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
Arial'>If we add this bit we could avoid that and quite easily know if we could
use SDP for the connection<o:p></o:p></span></font></span></p>

<p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
Arial'>Since protocol offloading is an integral (non-optional) part of InfiniBand
it also makes sense to put it as part of the IB ARP<o:p></o:p></span></font></span></p>

<p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
Arial'>(not to mention that it is specified in SDP that it uses the IP over IB
address resolution) <o:p></o:p></span></font></span></p>

<p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
Arial'><![if !supportEmptyParas]>&nbsp;<![endif]><o:p></o:p></span></font></span></p>

<p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
Arial'>Yaron<o:p></o:p></span></font></span></p>

<p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
Arial'><![if !supportEmptyParas]>&nbsp;<![endif]><o:p></o:p></span></font></span></p>

<p class=MsoNormal dir=LTR style='margin-left:.5in'><font size=2 color=black
face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma;color:black'>-----Original
Message-----<br>
<b><span style='font-weight:bold'>From:</span></b> ipoverib-admin@ietf.org
[mailto:ipoverib-admin@ietf.org]<b><span style='font-weight:bold'>On Behalf Of </span></b>Arvind
Murching<br>
<b><span style='font-weight:bold'>Sent:</span></b> Saturday, January 12, 2002
3:07 AM<br>
<b><span style='font-weight:bold'>To:</span></b> Vivek Kashyap;
ipoverib@ietf.org<br>
<b><span style='font-weight:bold'>Cc:</span></b> Jerry.Chu@eng.sun.com<br>
<b><span style='font-weight:bold'>Subject:</span></b> RE: [Ipoverib] IPoIB advanced
capabilities</span></font></p>

<p class=MsoNormal dir=LTR style='margin-left:.5in'><font size=3
face="Times New Roman"><span style='font-size:12.0pt'><![if !supportEmptyParas]>&nbsp;<![endif]><o:p></o:p></span></font></p>

<p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>Vivek,</span></font><font
color=black><span style='color:black'> </span></font><font size=2 color=black
face="Courier New"><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>two issues regarding the capabilities flags:</span></font><font
color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>

<p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>(a) From a
purist point of view, the link layer address is not the right place to specify
end-node capabilities.</span></font><font color=black><span style='color:black;
mso-color-alt:windowtext'><o:p></o:p></span></font></p>

<p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>(b)</span></font><font
color=black><span style='color:black'> </span></font><font size=2 color=black
face="Courier New"><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>Section 3.2 says:</span></font><font color=black><span
style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>

<p dir=LTR style='margin-left:.5in'><font size=3 color=black
face="Times New Roman"><span style='font-size:12.0pt;color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
</span></font><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;</span></font><font
color=black><span style='color:black'> </span></font><font size=2 color=black
face="Courier New"><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>The method by which a host decides it</span></font><font
color=black><span style='color:black'> </span></font><font size=2 color=black
face="Courier New"><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>wants to use</span></font><font color=black><span
style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>

<p dir=LTR style='margin-left:.5in'><font size=3 color=black
face="Times New Roman"><span style='font-size:12.0pt;color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
</span></font><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;</span></font><font
color=black><span style='color:black'> </span></font><font size=2 color=black
face="Courier New"><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>the IPoIB-RC/UC/RD services is beyond the scope</span></font><font
color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>

<p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
of this specification.</span></font><font color=black><span style='color:black;
mso-color-alt:windowtext'><o:p></o:p></span></font></p>

<p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>However, this
means that two</span></font><font color=black><span style='color:black'> </span></font><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt;font-family:
"Courier New";color:black'>nodes that conform to this specification,</span></font><font
color=black><span style='color:black'> </span></font><font size=2 color=black
face="Courier New"><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>and</span></font><font color=black><span style='color:black'> </span></font><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt;font-family:
"Courier New";color:black'>decide to use the capability bits,</span></font><font
color=black><span style='color:black'> </span></font><font size=2 color=black
face="Courier New"><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>can easily fail to interoperate. This is because this document
does not specify how</span></font><font color=black><span style='color:black'> </span></font><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt;font-family:
"Courier New";color:black'>they, for example, set up RC/UC connections (what
service ID is used etc).</span></font><font color=black><span style='color:
black'> </span></font><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>We should either
fully specify the encapsulation and</span></font><font color=black><span
style='color:black'> </span></font><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>connection
mechanisms for these other transport methods (if time permits), or</span></font><font
color=black><span style='color:black'> </span></font><font size=2 color=black
face="Courier New"><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>simply label the bits used for this purpose as
&quot;Reserved&quot; in this spec, and require all conformant implementations
to set this to 0 on transmit and ignore on receive.</span></font><font
color=black><span style='color:black'> </span></font><font size=2 color=black
face="Courier New"><span style='font-size:10.0pt;font-family:"Courier New";
color:black'>Later versions of this spec can define these bits, along with
details of how they are used.</span></font><font color=black><span
style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>

<p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>- Arvind</span></font><font
color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>

<p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>-----Original
Message-----<br>
From: Vivek Kashyap [<a href="mailto:kashyapv@us.ibm.com">mailto:kashyapv@us.ibm.com</a>]<br>
Sent: Friday, January 11, 2002 4:34 PM<br>
To: ipoverib@ietf.org<br>
Cc: kashyapv@us.ibm.com<br>
Subject: [Ipoverib] IPoIB advanced capabilities</span></font><font color=black><span
style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>

<p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>Folks,</span></font><font
color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>

<p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
We have discussed IP over non-UD links a few times on this</span></font><font
color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>

<p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
mailing list.The capability-flags in the address-resolution</span></font><font
color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>

<p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
packets provide the necessary hook to implementing such</span></font><font
color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>

<p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
solutions in an interoperable way. I presented a solution at</span></font><font
color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>

<p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
the IPoIB WG meeting at SLC earlier this month.</span></font><font color=black><span
style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>

<p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
Attached is a draft that I submitted a few days back. It</span></font><font
color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>

<p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
hasn't been announced yet but I thought that we need to move</span></font><font
color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>

<p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
forward. The draft and the solution is completely disjoint</span></font><font
color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>

<p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
from the encapsulation draft except for the capability-bits</span></font><font
color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>

<p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
being included in the address resolution packet.</span></font><font
color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>

<p dir=LTR style='margin-left:.5in'><font size=3 color=black
face="Times New Roman"><span style='font-size:12.0pt;color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
</span></font><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>I look forward
to your comments. </span></font><font color=black><span style='color:black;
mso-color-alt:windowtext'><o:p></o:p></span></font></p>

<p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>Vivek</span></font><font
color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>

<p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
style='font-size:10.0pt;font-family:"Courier New";color:black'>&lt;snip&gt;</span></font><font
color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>

</div>

</body>

</html>

--Boundary_(ID_MH0w1LW5oe3PQ6xdgF91BA)--

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Sat Jan 12 19:52:47 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA25408
	for <ipoverib-archive@odin.ietf.org>; Sat, 12 Jan 2002 19:52:47 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id TAA01338
	for ipoverib-archive@odin.ietf.org; Sat, 12 Jan 2002 19:52:50 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA01315;
	Sat, 12 Jan 2002 19:51:15 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA01287
	for <ipoverib@optimus.ietf.org>; Sat, 12 Jan 2002 19:51:14 -0500 (EST)
Received: from patan.sun.com (patan.Sun.COM [192.18.98.43])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA25400
	for <ipoverib@ietf.org>; Sat, 12 Jan 2002 19:51:10 -0500 (EST)
Received: from sunmail1.Sun.COM ([129.145.1.2])
	by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id RAA00701
	for <ipoverib@ietf.org>; Sat, 12 Jan 2002 17:51:13 -0700 (MST)
Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.84.31])
	by sunmail1.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1-Sun.COM.mod.2) with ESMTP id QAA02562
	for <ipoverib@ietf.org>; Sat, 12 Jan 2002 16:52:20 -0800 (PST)
Received: from sweethome (dsl-194-138.Eng.Sun.COM [129.146.194.138])
	by jurassic.eng.sun.com (8.12.2.Beta5+Sun/8.12.2.Beta5) with SMTP id g0D0p8R8802157
	for <ipoverib@ietf.org>; Sat, 12 Jan 2002 16:51:12 -0800 (PST)
Message-Id: <200201130051.g0D0p8R8802157@jurassic.eng.sun.com>
Date: Sat, 12 Jan 2002 16:51:31 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Subject: Re: [Ipoverib] IPoIB architecture/encapsulation
To: ipoverib@ietf.org
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: BFJ054ZmDEZyYoz8Lp+3lA==
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.2 SunOS 5.8 sun4u sparc 
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Sorry I am late to the party (was stuck on something else most of
Dec.), but like to make some comments regarding EtherType.

While I don't buy the performance arguement, I sympathize with
Vivek on his feeling toward EtherType. Architecturally it is ugly
to force one into the payload. The right approach would have the
framing layer carrying a field identifying the type of data being
encapsulated. But BTH wasn't designed for encapusulation purpose.
That is supposed to be the job and the only purpose of raw datagrams. 
Unfortunately due to various reasons the latter has been considered
by many as not fit for the job of IPoIB encapsulation.

Aside from the architectural correctness, I only see two valid concerns
against not including EtherType:

1. Compatibility issue - how many existing stack implementations
expect EtherType, or part of framing header to be present in the
datastream? Although the driver can always fabricate a pseudo header
if one is needed but is missing, it may incur some performance
penalty since it won't be in-line.

2. Not including Ethertype forbids one from sharing the same QP with
other protocols outside of IPv4/ARP/IPv6. Is this a legitimate concern?
Note that this concerns only applies unicast. For multicast, by
embedding a special IPoIB signature in MGIDs for IPv4/ARP/IPv6 use
only, demuxing can be based on MGID so sharing QPs won't be an issue.

Before the two questions are fully understood, I'm leaning slightly
toward including the EtherType, just to be on the safe side.

Jerry

>From: Vivek Kashyap <kashyapv@us.ibm.com>
>Subject: Re: [Ipoverib] IPoIB architecture/encapsulation
>To: yaronh@voltaire.com (Yaron Haviv)
>Date: Thu, 13 Dec 2001 16:23:24 -0800 (PST)
>Cc: ipoverib@ietf.org
>> Just trying to finalize my arguments 
>> 
>> As for having 4 bits or 4 byes ethertype:
>> Comparing 4 bits takes exactly the same resources as 4 bytes
>> If the header size is constant the header processing is not complicated,
>> when the next header offset is variable (in case of IP options, LLC, etc')
>> you need to parse the packet
>
>        Is it less complicated than not having the header at all ?
>        Sure one can work with the header but why do something when
>        there is no need for it? 
>
>        In my previous mail I explained the advantages of protocol
>        signature over the 'ethertype'. Those advantages are not
>        overcome by your argument. All you are saying is that one can
>        make the receive processing in the presence of 4 byte
>        'ethertype' as good as the 'protocol signature' no-header
>        processing.  
>
>> Its not the case here you just start the IP header processing in offset 4,
>> after you checked the first 4 bytes
>> Putting the 4 bits in pieces of the IB header is less natural
>
>        I am afraid I have lost you. There is no putting of 4 bits of
>        pieces in IB header. The 4 bits I refer to are already part of
>        the packet.
>
>        The IP packet starts with the IP version (a 4 bit field) and
>        the ARP encapsulation starts with hardware type (a 16 bit
>        value). These values are different in the first 4 bits as I
>        explained in an earlier mail about the values. Hence the
>        information is already present in the packet and there is no
>        need to add a header. As noted earlier we are disambiguating
>        between 2 protocols only.
>
>
>> We have done several NIC drivers in different OS's, including an
>IPoIB > driver > So I could tell you based on our experience it is not
>a problem
>
>        Anything can be made to work. I only object to have to make
>        something work that has not been shown to bring anything to
>        the table.
>
>> 
>> As for the demuxing 
>> 1. The network driver doesn't process the ARP's, the OS above it does
>
>        Yes, but the driver or some layer just above it demultiplexes
>        the streams.
>
>> 2. You separate to different queues when A) you want to save context switch,
>> copies or b) when you want to avoid going over the queue and look for your
>> specific information, this is not the case here we are in the same context,
>> the ARP messages are sporadic and could wait in the queue for a while until
>> I will get to them 
>	
>        Sure one can do as you say. Put them all in one queue and then
>        process them on the type. Why would one need to invent a
>        header for that? All one has to do is to look at the protocol
>        signature. 
>
>> 
>> As I mentioned before you are looking for optimizations in the wrong place,
>> there are many other places where we could optimize in future work 
>
>        I am pointing out that if there is no advantage from more
>        work then less work is always better than more work.
>
>Vivek
>
>
>> -----Original Message-----
>> From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
>> Vivek Kashyap
>> Sent: Thursday, December 13, 2001 1:41 AM
>> To: yaronh-voltaire.com@mail.barak.net.il
>> Cc: ipoverib@ietf.org
>> Subject: Re: Re: [Ipoverib] IPoIB architecture/encapsulation
>> 
>> Yaron,
>>         I am afraid we have sidetracked. The comment about performance
>>         was different. The point I was making was that generally you
>>         would get better performance if you had less to do. It doesn't
>>         mean that one cannot by specialisation mitigate the negatives
>>         of a more complicated task.
>> 
>>           delete/add header vs. no such work. I like no work.
>> 
>>         Here is the problem and the reasoning:
>> 
>>         The network stack doesn't change you merely add a network
>>         interface driver. This driver has the task of determining the
>>         upper layer to call. It can call it directly or it can slot
>>         the packet in some type based queue. The interface driver will
>>         determine the type from the packet stream. It can do so in
>>         many ways. One of the ways is by using the protocol signature.
>>         Another is by way of 'ethertype' and a third is by mandating a
>>         QP per protocol.
>> 
>>         The use of 'ethertype' is not universal. A choice by IPoIB to
>>         use ethertype does not mean 'protocol X' over IB will use the
>>         same encapsulation and therefore you cannot rely on
>>         'ethertype' differentiating your packets.
>> 
>>         So the 'ethertype' is of use in differentiating IPoIB
>>         protocols only (IPv4, IPv6 and ARP). For this task we have two
>>         other methods that require no change in the on-wire
>>         encapsulation: use the protocol signatures (4 bits in the
>>         begining) and use separate QPs per protocol. Of the two : use
>>         protocol signatures doesn't require any on-wire change,
>>         doesn't require the driver to strip headers, doesn't require
>>         the TCP layer to keep space for the header. That is why it is
>>         preferable. 
>> 
>>         In essence the protocol signature solution gives us the same
>>         returns with the additional advantage of :
>> 
>>                 - no need to reflect it on the wire
>>                 - no removal/addition of headers
>>        
>> 
>> Vivek
>> 
>> 
>> >
>> > Vivek,
>> >
>> > I generally agrees with Mike, comparing 4 bit vs. 4 bytes has no impact on
>> mem access as well as CPU cycles
>> > My assumption is that OSV's will not change the way their TCP/IP stack is
>> built and separate ARP's from IP's, they will just add IPoIB as another net
>> interface driver
>> > The demultiplexing doesn't add much value since it is done in the same
>> kernel context, there is no urgency in dealing with the ARP's, there aren't
>> many ARP's, it also relates to the stack work (ARP response might release
>> pending IP packets)
>> >
>> > it looks to me like the 4 bit option is not the straightforward approach
>> >
>> > If you want to improve performance significantly, without affecting the
>> stack you should do things similar to what the new NIC's do:
>> > -  Offload checksum (avoid cache trashing)
>> > -  Increase MTU, MSS (avoid UDP fragmentation, TCP segmentation)
>> > -  Suppress Interrupts
>> > -  Improve memory management
>> > These things could be done with RC/RD and using some IB tricks
>> >
>> > Yaron
>> >
>> > Original Message:
>> > -----------------
>> > From: Vivek Kashyap kashyapv@us.ibm.com
>> > Date: Wed, 12 Dec 2001 12:27:20 -0800 (PST)
>> > To: yaronh@voltaire.com, ipoverib@ietf.org
>> > Subject: Re: [Ipoverib] IPoIB architecture/encapsulation
>> >
>> >
>> > Yaron,
>> >
>> > >
>> > > Few questions & suggestions
>> > >
>> > > I'm trying to understand way not to take the straightforward approach
>> for
>> > > the ethrtype field
>> > > And just add that before the IP header in the payload area, if alignment
>> is
>> > > needed so align it
>> >
>> >         Since we already filter the recepients by way of QP to one
>> >         (IPv6) or two (IPv4/ARP) why go to this trouble? The type of
>> >         work is the same whether you see 4 bits or 32 bits. But it is a
>> >         lot less with 4 bits.
>> >
>> >         The interface driver will distinguish the types and call the
>> >         upper layer accordingly.
>> >
>> >                 It is easier with 4 bits than with 4 bytes. Some people
>> >                 have mentioned performance - I would say that if you
>> >                 have less to do you are more performant. With 4 bits
>> >                 you have less to do. Secondly, if you are in
>> >                 'performance' mode switching in hardware then it is
>> >                 even better since you don't have to strip the header
>> >                 but merely have to pass the buffer to the relevant
>> >                 upper layer directly.
>> >
>> >         The interface driver does not have to do the followig when
>> >         using protocol signatures:
>> >
>> >     - It doen't have to add a header when sending
>> >     - It doesn't have to receive header when receiving
>> >
>> >   These tasks are required for if one invents an 'ethertype'.
>> >
>> >         The upper layer doesn't have to keep space for the header, the
>> >         ethertype, when allocating data for sending if the protocol
>> >         signatures are to be used.
>> >
>> >         I don't see any gain if an artificial header is included to
>> >         distinguish protocols in IPoIB. I see more work. We already
>> >         know, by definition, it is IP or ARP. Just test which one it
>> >         is by looking at the first 4 bits.
>> >
>> >
>> > >
>> > > I might be missing something here so forgive me if I do
>> > > It looks to me like TCP stacks are used to get a packet with everything
>> > > including Ethernet header, it is not like we are doing full header/data
>> > > separation that could help the stack
>> > > On the worst case if the TCP implementation has problems with it we
>> could
>> > > have 2 data segments in the receiving side, one for the first word,
>> second
>> > > for the rest
>> >
>> > Things are generally done in simpler ways. The IP stack gets the IP
>> > packet. The lower headers are stripped away. If you don't have the
>> > any header to strip in your interface driver all the better.
>> >
>> > >
>> > > Another thing that was mentioned in the recent emails was that IPoIB is
>> > > defined just for UD
>> >
>> >         If you look at the beginning of this thread (the mail that
>> >         started it all) I have discussed capability flags and how
>> >         RC/RD/UC connections can be setup and work interoperably with
>> >         UD only implementations. IPoIB over UD is the default and
>> >         madatory.
>> >
>> >
>> > Vivek
>> >
>> > _______________________________________________
>> > IPoverIB mailing list
>> > IPoverIB@ietf.org
>> > http://www.ietf.org/mailman/listinfo/ipoverib
>> >
>> > --------------------------------------------------------------------
>> > mail2web - Check your email from the web at
>> > http://mail2web.com/ .
>> >
>> >
>> 
>> 
>> --
>> Vivek Kashyap
>> IBM
>> kashyapv@us.ibm.com
>> vivk@us.ibm.com
>> 503 578 3422 (o)
>> 
>> _______________________________________________
>> IPoverIB mailing list
>> IPoverIB@ietf.org
>> http://www.ietf.org/mailman/listinfo/ipoverib
>> 
>
>
>-- 
>Vivek Kashyap
>Linux Technology Center, IBM
>kashyapv@us.ibm.com 
>vivk@us.ibm.com 
>503 578 3422 (o)
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Sun Jan 13 00:29:08 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA29442
	for <ipoverib-archive@odin.ietf.org>; Sun, 13 Jan 2002 00:29:03 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id AAA07021
	for ipoverib-archive@odin.ietf.org; Sun, 13 Jan 2002 00:29:04 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id AAA07002;
	Sun, 13 Jan 2002 00:28:14 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id AAA06972
	for <ipoverib@optimus.ietf.org>; Sun, 13 Jan 2002 00:28:12 -0500 (EST)
Received: from patan.sun.com (patan.Sun.COM [192.18.98.43])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA29434
	for <ipoverib@ietf.org>; Sun, 13 Jan 2002 00:28:11 -0500 (EST)
Received: from sunmail1.Sun.COM ([129.145.1.2])
	by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id WAA12947
	for <ipoverib@ietf.org>; Sat, 12 Jan 2002 22:28:12 -0700 (MST)
Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.88.130])
	by sunmail1.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1-Sun.COM.mod.2) with ESMTP id VAA25913
	for <ipoverib@ietf.org>; Sat, 12 Jan 2002 21:29:20 -0800 (PST)
Received: from sweethome (dsl-194-138.Eng.Sun.COM [129.146.194.138])
	by jurassic.eng.sun.com (8.12.2.Beta5+Sun/8.12.2.Beta5) with SMTP id g0D5S9R8812864
	for <ipoverib@ietf.org>; Sat, 12 Jan 2002 21:28:11 -0800 (PST)
Message-Id: <200201130528.g0D5S9R8812864@jurassic.eng.sun.com>
Date: Sat, 12 Jan 2002 21:28:29 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
To: ipoverib@ietf.org
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: cIFUSakRGxmyraKYC0IHrQ==
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.2 SunOS 5.8 sun4u sparc 
Subject: [Ipoverib] What's inside an IPoIB link layer address?
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

While proposals beyond the basic IPoIB over UD started showing up,
I still don't think we've reached a rough consensus on some basic
questions such as what the IPoIB link layer address should contain.
Again I'd like to apologize for not being there at the SLC meeting
last month. Perhaps a consensus was reached there but i've missed
the boat. In that case please let me know what it is and I'll shut
up.

What I'd like to see included in the IPoIB link layer address is

GID
QPN

and I'd like to allow QPN field to be 0.

What I don't like to see included is LID.

Discussions:

There has been a consensus of using GID, not LID as the basic
IB address identifying the port hosting an IP address. GID is
sufficiently stable, not subject to change across a node reboot
or subnet merge (IB level) like LID is. Also people don't want
to exclude the possibility of IPoIB link spanning more than
one IB subnet. GID supports scope beyond local link so it's
suitable for that purpose.

Multiple GIDs with different scopes may have been assigned to a CA.
It is required that a CA returns the GID with a scope no less than
the one for the IPoIB link. E.g. if an IPoIB link covers multiple
IB subnets, but a CA uses a link-local GID as part of its IPoIB
link layer address, it's not going to work well.

I don't see any value of including LID besides GID as proposed
by Vivek. I believes a lookup of PathRecord is always needed
in order for a node to retrieve complete path info before sending
unicast pkts to another node. The destination LID is part of
the PathRecord so including it in the link layer address returned
from ARP/ND doesn't save work. But you have all the drawbacks that
swayed people away from using LID - it adds volatility to the link
address. LID changing due to subnet merge or other reasons requires
potentially new mechanism being added to IP/driver implementation
to trigger ARP/ND flush. Lastly, what should be the LID if the IP
node is in a different IB subnet? Obviously it can't be the LID of
an intermediate IB router.

For QPN I'd like to suggest one thing. Since SIDR is a required
feature by IBA, the QPN field is allowed to be zero. What that
means is that the requester must use SIDR to retrieve the remote
QPN, given the GID of a remote IP address.

This feature allows the support of static/persistent "MAC addr"
wrt node reboot by removing the volatile part of the link layer
address - the QPN. In many implementations the QPN allocated for
a CA may change across node reboot. Support of static "MAC addr"
has been considered desirable by a number of people.

I don't like to add additional requirement such as SIDR to IPoIB,
and was trying to make the above feature "optional". Ideally nodes
can express whether they'd like QPN to be returned by embedding
some flags, such as "I don't talk SIDR so please return QPN" in
their IPoIB link layer addresses. But the broadcast nature of
IPv4/ARP forbids the use of different "MAC addrs" for the same IP
address, so my attempt failed.

Comment?

Jerry


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Sun Jan 13 15:07:28 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA14684
	for <ipoverib-archive@odin.ietf.org>; Sun, 13 Jan 2002 15:07:28 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id PAA03937
	for ipoverib-archive@odin.ietf.org; Sun, 13 Jan 2002 15:07:32 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA03887;
	Sun, 13 Jan 2002 15:06:49 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA02729
	for <ipoverib@optimus.ietf.org>; Sun, 13 Jan 2002 14:07:24 -0500 (EST)
Received: from mail1.microsoft.com (mail1.microsoft.com [131.107.3.125])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA14124
	for <ipoverib@ietf.org>; Sun, 13 Jan 2002 14:07:19 -0500 (EST)
Received: from inet-vrs-01.redmond.corp.microsoft.com ([157.54.8.27]) by mail1.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Sun, 13 Jan 2002 11:06:51 -0800
Received: from 157.54.6.150 by inet-vrs-01.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Sun, 13 Jan 2002 11:06:51 -0800
Received: from red-imc-01.redmond.corp.microsoft.com ([157.54.9.102]) by inet-hub-05.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Sun, 13 Jan 2002 11:06:51 -0800
Received: from win-imc-02.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.84]) by red-imc-01.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Sun, 13 Jan 2002 11:06:51 -0800
Received: from win-msg-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.52]) by win-imc-02.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3588.0);
	 Sun, 13 Jan 2002 11:04:49 -0800
X-MimeOLE: Produced By Microsoft Exchange V6.0.6115.0
Content-Class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
Date: Sun, 13 Jan 2002 11:04:48 -0800
Message-ID: <2E33960095B58E40A4D3345AB9F65EC102261842@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
Thread-Topic: [Ipoverib] What's inside an IPoIB link layer address?
thread-index: AcGb9LTUbYOL+DWmSq6fpwVhuJFu+QAbmrNA
From: "Arvind Murching" <arvindm@windows.microsoft.com>
To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>, <ipoverib@ietf.org>
X-OriginalArrivalTime: 13 Jan 2002 19:04:49.0112 (UTC) FILETIME=[2C403580:01C19C65]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id OAA02730
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

IMO, this proposal meshes well with both the IBTA spec (regarding
end-to-end communication between nodes) as well as with IP stack
implementations.

Nodes may use SIDR to resolve "more than the IP address" with partner
nodes - this is what SIDR is good for. For interop reasons, this RFC
should be either silent (preferable in the interests of completing this
RFC) regarding use of SIDR for additional resolution, or specify SIDR
usage completely. It is not a good idea to mention an optional feature
without describing it completely.

Also for interoperability reasons, this RFC should always require a
"default" QPN to be returned by a responding node. As an example,
consider gratuitous ARP packets - can these contain a 0 QPN?

Thanks
Arvind

-----Original Message-----
From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com] 
Sent: Saturday, January 12, 2002 9:28 PM
To: ipoverib@ietf.org
Subject: [Ipoverib] What's inside an IPoIB link layer address?


While proposals beyond the basic IPoIB over UD started showing up, I
still don't think we've reached a rough consensus on some basic
questions such as what the IPoIB link layer address should contain.
Again I'd like to apologize for not being there at the SLC meeting last
month. Perhaps a consensus was reached there but i've missed the boat.
In that case please let me know what it is and I'll shut up.

What I'd like to see included in the IPoIB link layer address is

GID
QPN

and I'd like to allow QPN field to be 0.

What I don't like to see included is LID.

Discussions:

There has been a consensus of using GID, not LID as the basic IB address
identifying the port hosting an IP address. GID is sufficiently stable,
not subject to change across a node reboot or subnet merge (IB level)
like LID is. Also people don't want to exclude the possibility of IPoIB
link spanning more than one IB subnet. GID supports scope beyond local
link so it's suitable for that purpose.

Multiple GIDs with different scopes may have been assigned to a CA. It
is required that a CA returns the GID with a scope no less than the one
for the IPoIB link. E.g. if an IPoIB link covers multiple IB subnets,
but a CA uses a link-local GID as part of its IPoIB link layer address,
it's not going to work well.

I don't see any value of including LID besides GID as proposed by Vivek.
I believes a lookup of PathRecord is always needed in order for a node
to retrieve complete path info before sending unicast pkts to another
node. The destination LID is part of the PathRecord so including it in
the link layer address returned from ARP/ND doesn't save work. But you
have all the drawbacks that swayed people away from using LID - it adds
volatility to the link address. LID changing due to subnet merge or
other reasons requires potentially new mechanism being added to
IP/driver implementation to trigger ARP/ND flush. Lastly, what should be
the LID if the IP node is in a different IB subnet? Obviously it can't
be the LID of an intermediate IB router.

For QPN I'd like to suggest one thing. Since SIDR is a required feature
by IBA, the QPN field is allowed to be zero. What that means is that the
requester must use SIDR to retrieve the remote QPN, given the GID of a
remote IP address.

This feature allows the support of static/persistent "MAC addr" wrt node
reboot by removing the volatile part of the link layer address - the
QPN. In many implementations the QPN allocated for a CA may change
across node reboot. Support of static "MAC addr" has been considered
desirable by a number of people.

I don't like to add additional requirement such as SIDR to IPoIB, and
was trying to make the above feature "optional". Ideally nodes can
express whether they'd like QPN to be returned by embedding some flags,
such as "I don't talk SIDR so please return QPN" in their IPoIB link
layer addresses. But the broadcast nature of IPv4/ARP forbids the use of
different "MAC addrs" for the same IP address, so my attempt failed.

Comment?

Jerry


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Sun Jan 13 15:07:29 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA14699
	for <ipoverib-archive@odin.ietf.org>; Sun, 13 Jan 2002 15:07:29 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id PAA03951
	for ipoverib-archive@odin.ietf.org; Sun, 13 Jan 2002 15:07:33 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA03914;
	Sun, 13 Jan 2002 15:06:51 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA02787
	for <ipoverib@optimus.ietf.org>; Sun, 13 Jan 2002 14:14:34 -0500 (EST)
Received: from mail5.microsoft.com (mail5.microsoft.com [131.107.3.121])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA14220
	for <ipoverib@ietf.org>; Sun, 13 Jan 2002 14:14:30 -0500 (EST)
Received: from inet-vrs-05.redmond.corp.microsoft.com ([157.54.6.148]) by mail5.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Sun, 13 Jan 2002 11:14:02 -0800
Received: from 157.54.6.150 by inet-vrs-05.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Sun, 13 Jan 2002 11:14:02 -0800
Received: from red-imc-01.redmond.corp.microsoft.com ([157.54.9.102]) by inet-hub-05.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Sun, 13 Jan 2002 11:14:02 -0800
Received: from win-imc-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.39]) by red-imc-01.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Sun, 13 Jan 2002 11:14:01 -0800
Received: from win-msg-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.52]) by win-imc-01.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3588.0);
	 Sun, 13 Jan 2002 11:11:59 -0800
X-MimeOLE: Produced By Microsoft Exchange V6.0.6115.0
Content-Class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Subject: RE: [Ipoverib] IPoIB architecture/encapsulation (EtherType or not)
Date: Sun, 13 Jan 2002 11:11:59 -0800
Message-ID: <2E33960095B58E40A4D3345AB9F65EC102261843@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
Thread-Topic: [Ipoverib] IPoIB architecture/encapsulation (EtherType or not)
thread-index: AcGbzCQhqoHDCvQHQ82W0voo+YRoIgAmXAxg
From: "Arvind Murching" <arvindm@windows.microsoft.com>
To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>, <ipoverib@ietf.org>
X-OriginalArrivalTime: 13 Jan 2002 19:11:59.0844 (UTC) FILETIME=[2CFCB240:01C19C66]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id OAA02788
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

FWIW, other IP over FOO RFCs do take care to allow protocols other than
IP to share the link layer connection, typically using an EtherType.

-----Original Message-----
From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com] 
Sent: Saturday, January 12, 2002 4:52 PM
To: ipoverib@ietf.org
Subject: Re: [Ipoverib] IPoIB architecture/encapsulation


Sorry I am late to the party (was stuck on something else most of Dec.),
but like to make some comments regarding EtherType.

While I don't buy the performance arguement, I sympathize with Vivek on
his feeling toward EtherType. Architecturally it is ugly to force one
into the payload. The right approach would have the framing layer
carrying a field identifying the type of data being encapsulated. But
BTH wasn't designed for encapusulation purpose. That is supposed to be
the job and the only purpose of raw datagrams. 
Unfortunately due to various reasons the latter has been considered by
many as not fit for the job of IPoIB encapsulation.

Aside from the architectural correctness, I only see two valid concerns
against not including EtherType:

1. Compatibility issue - how many existing stack implementations expect
EtherType, or part of framing header to be present in the datastream?
Although the driver can always fabricate a pseudo header if one is
needed but is missing, it may incur some performance penalty since it
won't be in-line.

2. Not including Ethertype forbids one from sharing the same QP with
other protocols outside of IPv4/ARP/IPv6. Is this a legitimate concern?
Note that this concerns only applies unicast. For multicast, by
embedding a special IPoIB signature in MGIDs for IPv4/ARP/IPv6 use only,
demuxing can be based on MGID so sharing QPs won't be an issue.

Before the two questions are fully understood, I'm leaning slightly
toward including the EtherType, just to be on the safe side.

Jerry

<snip>


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Sun Jan 13 15:19:53 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA14835
	for <ipoverib-archive@odin.ietf.org>; Sun, 13 Jan 2002 15:19:53 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id PAA04137
	for ipoverib-archive@odin.ietf.org; Sun, 13 Jan 2002 15:19:57 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA04123;
	Sun, 13 Jan 2002 15:19:11 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA04091
	for <ipoverib@optimus.ietf.org>; Sun, 13 Jan 2002 15:19:09 -0500 (EST)
Received: from pdxpo.dsl-only.net (pdxpo.dsl-only.net [63.105.16.3])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA14828
	for <ipoverib@ietf.org>; Sun, 13 Jan 2002 15:19:04 -0500 (EST)
Received: from bonfire (unverified [63.105.18.201]) by pdxpo.dsl-only.net
 (Rockliffe SMTPRA 4.5.4) with SMTP id <B0009127213@pdxpo.dsl-only.net>;
 Sun, 13 Jan 2002 12:18:17 -0800
From: "Bill Strahm" <bill@strahm.net>
To: "Arvind Murching" <arvindm@windows.microsoft.com>,
        "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>, <ipoverib@ietf.org>
Subject: RE: [Ipoverib] IPoIB architecture/encapsulation (EtherType or not)
Date: Sun, 13 Jan 2002 12:27:13 -0800
Message-ID: <HHEDIJGOEFLPCPNILMPAIEMPCEAA.bill@strahm.net>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0)
Importance: Normal
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
In-Reply-To: <2E33960095B58E40A4D3345AB9F65EC102261843@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

I am very much talking about a Bill Strahm personal opinion here, and not as
WG Chair...

You can consider the "ethertype" to be something akin to a port, saying
please deliver this to the protocol above with this ethertype.  The
equivalent to "ethertype" for IB is QPN in my eyes.  I can put as many
"Layer 3" protocols on top of IB by having each Layer 3 protocol bind to a
QPN, then have packets to that QPN show up to the bottom of the Layer 3
protocol...

I understand that QPN are a scarce resource, but attaching multiple
protocols on top of a single QPN and having an upper layer demux does not
seem like the answer, if I do this, then I do not know what memory address
to deliver the packets to, so I might incur a copy... At a minimum I incur a
page flip.

WG Chair hat on ...
My understanding is that we have 2 major points to consider before we can
get to WG last call on the Encapsulation draft they are
1) Use of Ethertype, or not in the encapsulation
2) What to do about IP subnets crossing IB subnets, allow, architect for,
don't consider

I know that there are some link issues that Jerry has brought up, hopefully
his draft on the topic will come up early next week and people can tell me
what there opinions are.

The goal is to have a draft that is VERY close to WG Last call for the
Minnesota IETF, so we can tweak it and ask for Last Call after the meeting.
Please comment on these topics and I will try to direct the conversation,
and see if we have an answer.

Bill

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Arvind Murching
Sent: Sunday, January 13, 2002 11:12 AM
To: H.K. Jerry Chu; ipoverib@ietf.org
Subject: RE: [Ipoverib] IPoIB architecture/encapsulation (EtherType or
not)


FWIW, other IP over FOO RFCs do take care to allow protocols other than
IP to share the link layer connection, typically using an EtherType.

-----Original Message-----
From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
Sent: Saturday, January 12, 2002 4:52 PM
To: ipoverib@ietf.org
Subject: Re: [Ipoverib] IPoIB architecture/encapsulation


Sorry I am late to the party (was stuck on something else most of Dec.),
but like to make some comments regarding EtherType.

While I don't buy the performance arguement, I sympathize with Vivek on
his feeling toward EtherType. Architecturally it is ugly to force one
into the payload. The right approach would have the framing layer
carrying a field identifying the type of data being encapsulated. But
BTH wasn't designed for encapusulation purpose. That is supposed to be
the job and the only purpose of raw datagrams.
Unfortunately due to various reasons the latter has been considered by
many as not fit for the job of IPoIB encapsulation.

Aside from the architectural correctness, I only see two valid concerns
against not including EtherType:

1. Compatibility issue - how many existing stack implementations expect
EtherType, or part of framing header to be present in the datastream?
Although the driver can always fabricate a pseudo header if one is
needed but is missing, it may incur some performance penalty since it
won't be in-line.

2. Not including Ethertype forbids one from sharing the same QP with
other protocols outside of IPv4/ARP/IPv6. Is this a legitimate concern?
Note that this concerns only applies unicast. For multicast, by
embedding a special IPoIB signature in MGIDs for IPv4/ARP/IPv6 use only,
demuxing can be based on MGID so sharing QPs won't be an issue.

Before the two questions are fully understood, I'm leaning slightly
toward including the EtherType, just to be on the safe side.

Jerry

<snip>


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 14 01:33:01 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA20862
	for <ipoverib-archive@odin.ietf.org>; Mon, 14 Jan 2002 01:33:00 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id BAA25575
	for ipoverib-archive@odin.ietf.org; Mon, 14 Jan 2002 01:33:01 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id BAA25532;
	Mon, 14 Jan 2002 01:30:56 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id BAA25501
	for <ipoverib@optimus.ietf.org>; Mon, 14 Jan 2002 01:30:53 -0500 (EST)
Received: from patan.sun.com (patan.Sun.COM [192.18.98.43])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA20838
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 01:30:51 -0500 (EST)
Received: from sunmail1.Sun.COM ([129.145.1.2])
	by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id XAA04844;
	Sun, 13 Jan 2002 23:30:51 -0700 (MST)
Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.85.31])
	by sunmail1.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1-Sun.COM.mod.2) with ESMTP id WAA08900;
	Sun, 13 Jan 2002 22:32:00 -0800 (PST)
Received: from sweethome (dsl-194-138.Eng.Sun.COM [129.146.194.138])
	by jurassic.eng.sun.com (8.12.2+Sun/8.12.2) with SMTP id g0E6UkqT891932;
	Sun, 13 Jan 2002 22:30:49 -0800 (PST)
Message-Id: <200201140630.g0E6UkqT891932@jurassic.eng.sun.com>
Date: Sun, 13 Jan 2002 22:31:09 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Subject: RE: [Ipoverib] IPoIB architecture/encapsulation (EtherType or not)
To: bill@strahm.net
Cc: ipoverib@ietf.org
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: zJkNgIT9tqBoOE2TcgOxpQ==
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.2 SunOS 5.8 sun4u sparc 
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

>From: "Bill Strahm" <bill@strahm.net>
...
<snip>

>I understand that QPN are a scarce resource, but attaching multiple
>protocols on top of a single QPN and having an upper layer demux does not
>seem like the answer, if I do this, then I do not know what memory address
>to deliver the packets to, so I might incur a copy... At a minimum I incur a
>page flip.

Even dedicating one QP per layer-3 protocol won't give you zero-copy.
To target per-session memory, you'll need one QP per session, e.g. TCP
connection.

Almost a year ago in my first IPoIB proposal to IBTA/LWG (predating
IETF/WG), I had a "skip EtherType" flag in the IPoIB MAC addr.
Each node can decide whether it wants EtherType to be included in
all the unicast pkts destined for it. E.g. if a node has abundant
QPs, it can dedicate one for each EtherType and tells everyone else
not to embed one in payload. But later I felt it was such a frivilous 
optimization to be worth carried forward...

>
>WG Chair hat on ...
>My understanding is that we have 2 major points to consider before we can
>get to WG last call on the Encapsulation draft they are
>1) Use of Ethertype, or not in the encapsulation
>2) What to do about IP subnets crossing IB subnets, allow, architect for,
>don't consider

Actually I think the number one question to be answered is what
consistutes an IPoIB link layer address.

Jerry

>
>I know that there are some link issues that Jerry has brought up, hopefully
>his draft on the topic will come up early next week and people can tell me
>what there opinions are.
>
>The goal is to have a draft that is VERY close to WG Last call for the
>Minnesota IETF, so we can tweak it and ask for Last Call after the meeting.
>Please comment on these topics and I will try to direct the conversation,
>and see if we have an answer.
>
>Bill
>
>-----Original Message-----
>From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
>Of Arvind Murching
>Sent: Sunday, January 13, 2002 11:12 AM
>To: H.K. Jerry Chu; ipoverib@ietf.org
>Subject: RE: [Ipoverib] IPoIB architecture/encapsulation (EtherType or
>not)
>
>
>FWIW, other IP over FOO RFCs do take care to allow protocols other than
>IP to share the link layer connection, typically using an EtherType.
>
>-----Original Message-----
>From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
>Sent: Saturday, January 12, 2002 4:52 PM
>To: ipoverib@ietf.org
>Subject: Re: [Ipoverib] IPoIB architecture/encapsulation
>
>
>Sorry I am late to the party (was stuck on something else most of Dec.),
>but like to make some comments regarding EtherType.
>
>While I don't buy the performance arguement, I sympathize with Vivek on
>his feeling toward EtherType. Architecturally it is ugly to force one
>into the payload. The right approach would have the framing layer
>carrying a field identifying the type of data being encapsulated. But
>BTH wasn't designed for encapusulation purpose. That is supposed to be
>the job and the only purpose of raw datagrams.
>Unfortunately due to various reasons the latter has been considered by
>many as not fit for the job of IPoIB encapsulation.
>
>Aside from the architectural correctness, I only see two valid concerns
>against not including EtherType:
>
>1. Compatibility issue - how many existing stack implementations expect
>EtherType, or part of framing header to be present in the datastream?
>Although the driver can always fabricate a pseudo header if one is
>needed but is missing, it may incur some performance penalty since it
>won't be in-line.
>
>2. Not including Ethertype forbids one from sharing the same QP with
>other protocols outside of IPv4/ARP/IPv6. Is this a legitimate concern?
>Note that this concerns only applies unicast. For multicast, by
>embedding a special IPoIB signature in MGIDs for IPv4/ARP/IPv6 use only,
>demuxing can be based on MGID so sharing QPs won't be an issue.
>
>Before the two questions are fully understood, I'm leaning slightly
>toward including the EtherType, just to be on the safe side.
>
>Jerry
>
><snip>
>
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>http://www.ietf.org/mailman/listinfo/ipoverib
>


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 14 12:48:48 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA09249
	for <ipoverib-archive@odin.ietf.org>; Mon, 14 Jan 2002 12:48:48 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id MAA16667
	for ipoverib-archive@odin.ietf.org; Mon, 14 Jan 2002 12:48:49 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA16424;
	Mon, 14 Jan 2002 12:44:37 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA16393
	for <ipoverib@optimus.ietf.org>; Mon, 14 Jan 2002 12:44:36 -0500 (EST)
Received: from omegaband.com (faust.seagullsemi.com [206.196.68.34])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA09053
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 12:44:31 -0500 (EST)
Received: from seahorsepc [207.8.52.226] by omegaband.com with ESMTP
  (SMTPD32-7.04) id A736FBBA0134; Mon, 14 Jan 2002 11:36:54 -0600
From: "Ashwin Kamath" <kamath@omegaband.com>
To: <ipoverib@ietf.org>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
Date: Mon, 14 Jan 2002 11:42:46 -0600
Message-ID: <IFEKJBDLPFOEGGPIIGMFCECECAAA.kamath@omegaband.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0)
X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400
In-Reply-To: <200201130528.g0D5S9R8812864@jurassic.eng.sun.com>
Importance: Normal
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit


I agree with using GID both for longevity and scope. What about using
SIDR(Private Data) for communicating all the capability bits and any other
extras? If it is not too late for this, I have some new feedback.

For ARP on an IB subet one would require that
a. IB Switches support multicast. (This is true in most cases today.)
b. IP addresses used in an infiniband subnet are unique for use by IPoIB.
(This is an important constraint which gets rid of the ethertype field)
c. Some issues related to Multicast over IB need to be resolved (like
multiple consumers using multicast resources.)

Requirement (b) was addressed by defining a multicast group and the IP
address format. One would still have to deal with the GID, LID and the IP
address. Including the LID in the ARP Response will be difficult to deal
with because
 a. LID Assignments change.
 b. ARP might have to traverse a router, for IP Subnets spanning IB Subnets
(a) might not really be an issue because ARP entries are aged. In ethernet,
the ethernet topology might change and MAC Addresses can change on server
reboots (by way of using a different MAC or by assigning a MAC Address.) The
current ARP implementations do take these scenarios into account. Moreover
Path record lookups alone would not help resolve the issue of changing LIDs.
The locally cached path record will not be valid if the LID changes. For a
true solution, there has to be some way of using IB Traps to get information
about topology changes as and when it happens. An end node can subscribe for
a trap, but the end node will have to do something like,
 * Tell me when all the switches on the path to this node report a state
change on such and such a port.
 This will not work very well right now because one will have to subscribe
for traps on all switches leading to the end node, which implies that an end
node interested in the existence of another end node will need to have
knowledge of the complete IB topology. This is probably something worth
considering in the Infiniband WGs.

On the same token, if you use the globally unique GID, information about the
GID will not be maintained by the local subnet's subnet administration and
hence a path record lookup based on the GID would fail. So if the GID lookup
fails, one would send the ipoib packet to the designated IB router (as is
done with ethernet today, albiet based on netmasks). But what if you have an
IP subnet spanning IB Subnets. The designated router exists for an IP Subnet
and the local subnet IB administration will not have information about all
the IP nodes, because they could exist on another IB Subnet.

 The following IB transactions are required before sending any IPoIB data
over the subnet (assuming the multicast join is already done and we use GIDs
as link layer addresses)
 1. IpoIB ARP.
 2. Path record lookup.
 3. SIDR

In all this, the existence of the ARP is questionable. The same
functionality can be achieved using 2 and 3 above.
If you use GIDs as the IP addresses ,you can use a Path record lookup and
then use SIDR to get all the information to use IPoIB. The advantages are
evident, for eg,
 a. Traversing a router. Assuming that the path record lookup fails, one
will redirect all the related ipoib packets to the Gateway/router. Since ARP
does not exist, one does not have to worry about the issues involved by ARP
traversing a router, or the router proxying for end nodes on the same IP
Subnet but different IB Subnets. IP and IB Subnets will be one and the same.
 b. Debate on using the ether-type field vanishes. The Service ID will be
unique to IPoIB and will be assigned by some WG. You could potentially use
two Service IDs (might be an overkill), one for IPv6 packets and another for
IPv4.
 c. No special format for IP addresses is required (I have not thought this
through very well.)
 d. Switches are not required to support Multicast.
 e. There are no issues with multiple consumers using the same multicast
resources.
 f. Multicast could be used by an IB Gateways only to avoid multi-unicast.

In this scenario the IP subnet is the same as an IB Subnet, since the same
network addresses are used.

-Ashwin



-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of H.K. Jerry Chu
Sent: None
To: ipoverib@ietf.org
Subject: [Ipoverib] What's inside an IPoIB link layer address?


While proposals beyond the basic IPoIB over UD started showing up,
I still don't think we've reached a rough consensus on some basic
questions such as what the IPoIB link layer address should contain.
Again I'd like to apologize for not being there at the SLC meeting
last month. Perhaps a consensus was reached there but i've missed
the boat. In that case please let me know what it is and I'll shut
up.

What I'd like to see included in the IPoIB link layer address is

GID
QPN

and I'd like to allow QPN field to be 0.

What I don't like to see included is LID.

Discussions:

There has been a consensus of using GID, not LID as the basic
IB address identifying the port hosting an IP address. GID is
sufficiently stable, not subject to change across a node reboot
or subnet merge (IB level) like LID is. Also people don't want
to exclude the possibility of IPoIB link spanning more than
one IB subnet. GID supports scope beyond local link so it's
suitable for that purpose.

Multiple GIDs with different scopes may have been assigned to a CA.
It is required that a CA returns the GID with a scope no less than
the one for the IPoIB link. E.g. if an IPoIB link covers multiple
IB subnets, but a CA uses a link-local GID as part of its IPoIB
link layer address, it's not going to work well.

I don't see any value of including LID besides GID as proposed
by Vivek. I believes a lookup of PathRecord is always needed
in order for a node to retrieve complete path info before sending
unicast pkts to another node. The destination LID is part of
the PathRecord so including it in the link layer address returned
from ARP/ND doesn't save work. But you have all the drawbacks that
swayed people away from using LID - it adds volatility to the link
address. LID changing due to subnet merge or other reasons requires
potentially new mechanism being added to IP/driver implementation
to trigger ARP/ND flush. Lastly, what should be the LID if the IP
node is in a different IB subnet? Obviously it can't be the LID of
an intermediate IB router.

For QPN I'd like to suggest one thing. Since SIDR is a required
feature by IBA, the QPN field is allowed to be zero. What that
means is that the requester must use SIDR to retrieve the remote
QPN, given the GID of a remote IP address.

This feature allows the support of static/persistent "MAC addr"
wrt node reboot by removing the volatile part of the link layer
address - the QPN. In many implementations the QPN allocated for
a CA may change across node reboot. Support of static "MAC addr"
has been considered desirable by a number of people.

I don't like to add additional requirement such as SIDR to IPoIB,
and was trying to make the above feature "optional". Ideally nodes
can express whether they'd like QPN to be returned by embedding
some flags, such as "I don't talk SIDR so please return QPN" in
their IPoIB link layer addresses. But the broadcast nature of
IPv4/ARP forbids the use of different "MAC addrs" for the same IP
address, so my attempt failed.

Comment?

Jerry


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 14 12:58:50 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA09839
	for <ipoverib-archive@odin.ietf.org>; Mon, 14 Jan 2002 12:58:45 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id MAA17422
	for ipoverib-archive@odin.ietf.org; Mon, 14 Jan 2002 12:58:47 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA17299;
	Mon, 14 Jan 2002 12:57:28 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA17265
	for <ipoverib@optimus.ietf.org>; Mon, 14 Jan 2002 12:57:26 -0500 (EST)
Received: from omegaband.com (faust.seagullsemi.com [206.196.68.34])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA09776
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 12:57:24 -0500 (EST)
Received: from seahorsepc [207.8.52.226] by omegaband.com with ESMTP
  (SMTPD32-7.04) id AA3B281E0132; Mon, 14 Jan 2002 11:49:47 -0600
From: "Ashwin Kamath" <kamath@omegaband.com>
To: <ipoverib@ietf.org>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
Date: Mon, 14 Jan 2002 11:55:40 -0600
Message-ID: <IFEKJBDLPFOEGGPIIGMFKECECAAA.kamath@omegaband.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0)
X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400
In-Reply-To: <IFEKJBDLPFOEGGPIIGMFCECECAAA.kamath@omegaband.com>
Importance: Normal
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

A small correction,
 IB and IP Subnets need not be the same. IB routers will be used to traverse
IB Subnets and the SIDR is transparent to existence of such routers.
 Also this is only an alternative, ARP by itself could completely solve the
problem but there are issues it has within the IB subnet. This is an attempt
to work around the issues by suggesting an alternative.

-Ashwin.

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Ashwin Kamath
Sent: Monday, January 14, 2002 11:43 AM
To: ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?



I agree with using GID both for longevity and scope. What about using
SIDR(Private Data) for communicating all the capability bits and any other
extras? If it is not too late for this, I have some new feedback.

For ARP on an IB subet one would require that
a. IB Switches support multicast. (This is true in most cases today.)
b. IP addresses used in an infiniband subnet are unique for use by IPoIB.
(This is an important constraint which gets rid of the ethertype field)
c. Some issues related to Multicast over IB need to be resolved (like
multiple consumers using multicast resources.)

Requirement (b) was addressed by defining a multicast group and the IP
address format. One would still have to deal with the GID, LID and the IP
address. Including the LID in the ARP Response will be difficult to deal
with because
 a. LID Assignments change.
 b. ARP might have to traverse a router, for IP Subnets spanning IB Subnets
(a) might not really be an issue because ARP entries are aged. In ethernet,
the ethernet topology might change and MAC Addresses can change on server
reboots (by way of using a different MAC or by assigning a MAC Address.) The
current ARP implementations do take these scenarios into account. Moreover
Path record lookups alone would not help resolve the issue of changing LIDs.
The locally cached path record will not be valid if the LID changes. For a
true solution, there has to be some way of using IB Traps to get information
about topology changes as and when it happens. An end node can subscribe for
a trap, but the end node will have to do something like,
 * Tell me when all the switches on the path to this node report a state
change on such and such a port.
 This will not work very well right now because one will have to subscribe
for traps on all switches leading to the end node, which implies that an end
node interested in the existence of another end node will need to have
knowledge of the complete IB topology. This is probably something worth
considering in the Infiniband WGs.

On the same token, if you use the globally unique GID, information about the
GID will not be maintained by the local subnet's subnet administration and
hence a path record lookup based on the GID would fail. So if the GID lookup
fails, one would send the ipoib packet to the designated IB router (as is
done with ethernet today, albiet based on netmasks). But what if you have an
IP subnet spanning IB Subnets. The designated router exists for an IP Subnet
and the local subnet IB administration will not have information about all
the IP nodes, because they could exist on another IB Subnet.

 The following IB transactions are required before sending any IPoIB data
over the subnet (assuming the multicast join is already done and we use GIDs
as link layer addresses)
 1. IpoIB ARP.
 2. Path record lookup.
 3. SIDR

In all this, the existence of the ARP is questionable. The same
functionality can be achieved using 2 and 3 above.
If you use GIDs as the IP addresses ,you can use a Path record lookup and
then use SIDR to get all the information to use IPoIB. The advantages are
evident, for eg,
 a. Traversing a router. Assuming that the path record lookup fails, one
will redirect all the related ipoib packets to the Gateway/router. Since ARP
does not exist, one does not have to worry about the issues involved by ARP
traversing a router, or the router proxying for end nodes on the same IP
Subnet but different IB Subnets. IP and IB Subnets will be one and the same.
 b. Debate on using the ether-type field vanishes. The Service ID will be
unique to IPoIB and will be assigned by some WG. You could potentially use
two Service IDs (might be an overkill), one for IPv6 packets and another for
IPv4.
 c. No special format for IP addresses is required (I have not thought this
through very well.)
 d. Switches are not required to support Multicast.
 e. There are no issues with multiple consumers using the same multicast
resources.
 f. Multicast could be used by an IB Gateways only to avoid multi-unicast.

In this scenario the IP subnet is the same as an IB Subnet, since the same
network addresses are used.

-Ashwin



-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of H.K. Jerry Chu
Sent: None
To: ipoverib@ietf.org
Subject: [Ipoverib] What's inside an IPoIB link layer address?


While proposals beyond the basic IPoIB over UD started showing up,
I still don't think we've reached a rough consensus on some basic
questions such as what the IPoIB link layer address should contain.
Again I'd like to apologize for not being there at the SLC meeting
last month. Perhaps a consensus was reached there but i've missed
the boat. In that case please let me know what it is and I'll shut
up.

What I'd like to see included in the IPoIB link layer address is

GID
QPN

and I'd like to allow QPN field to be 0.

What I don't like to see included is LID.

Discussions:

There has been a consensus of using GID, not LID as the basic
IB address identifying the port hosting an IP address. GID is
sufficiently stable, not subject to change across a node reboot
or subnet merge (IB level) like LID is. Also people don't want
to exclude the possibility of IPoIB link spanning more than
one IB subnet. GID supports scope beyond local link so it's
suitable for that purpose.

Multiple GIDs with different scopes may have been assigned to a CA.
It is required that a CA returns the GID with a scope no less than
the one for the IPoIB link. E.g. if an IPoIB link covers multiple
IB subnets, but a CA uses a link-local GID as part of its IPoIB
link layer address, it's not going to work well.

I don't see any value of including LID besides GID as proposed
by Vivek. I believes a lookup of PathRecord is always needed
in order for a node to retrieve complete path info before sending
unicast pkts to another node. The destination LID is part of
the PathRecord so including it in the link layer address returned
from ARP/ND doesn't save work. But you have all the drawbacks that
swayed people away from using LID - it adds volatility to the link
address. LID changing due to subnet merge or other reasons requires
potentially new mechanism being added to IP/driver implementation
to trigger ARP/ND flush. Lastly, what should be the LID if the IP
node is in a different IB subnet? Obviously it can't be the LID of
an intermediate IB router.

For QPN I'd like to suggest one thing. Since SIDR is a required
feature by IBA, the QPN field is allowed to be zero. What that
means is that the requester must use SIDR to retrieve the remote
QPN, given the GID of a remote IP address.

This feature allows the support of static/persistent "MAC addr"
wrt node reboot by removing the volatile part of the link layer
address - the QPN. In many implementations the QPN allocated for
a CA may change across node reboot. Support of static "MAC addr"
has been considered desirable by a number of people.

I don't like to add additional requirement such as SIDR to IPoIB,
and was trying to make the above feature "optional". Ideally nodes
can express whether they'd like QPN to be returned by embedding
some flags, such as "I don't talk SIDR so please return QPN" in
their IPoIB link layer addresses. But the broadcast nature of
IPv4/ARP forbids the use of different "MAC addrs" for the same IP
address, so my attempt failed.

Comment?

Jerry


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 14 15:05:14 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA16425
	for <ipoverib-archive@odin.ietf.org>; Mon, 14 Jan 2002 15:05:14 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id PAA22529
	for ipoverib-archive@odin.ietf.org; Mon, 14 Jan 2002 15:05:17 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA21885;
	Mon, 14 Jan 2002 14:55:19 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA21834
	for <ipoverib@optimus.ietf.org>; Mon, 14 Jan 2002 14:55:11 -0500 (EST)
Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA15841
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 14:55:07 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e31.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAA268772;
	Mon, 14 Jan 2002 14:52:06 -0500
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0EJt5783118;
	Mon, 14 Jan 2002 12:55:05 -0700
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0EJt3V07611;
	Mon, 14 Jan 2002 11:55:03 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0EJt2k28103;
	Mon, 14 Jan 2002 11:55:02 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201141955.g0EJt2k28103@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] IPoIB architecture/encapsulation
To: Jerry.Chu@eng.sun.com
Date: Mon, 14 Jan 2002 11:55:01 -0800 (PST)
Cc: ipoverib@ietf.org
In-Reply-To: <200201130051.g0D0p8R8802157@jurassic.eng.sun.com> from "H.K. Jerry Chu" at Jan 12, 2002 03:51:31 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

 I couldn't have put it better wrt why Ethertype is not needed: it is
 ugly. It really buys us nothing:


	* It does not buy us differentiation

                a) If one has a QP, by definition one knows what to
                   expect. There is no need for any type field.

                b) Ethertype does not help with differentiating with
                   other protocols. They are under no obligation to
                   follow any header defined by IPoIB. Are we
                   considering any headers that might be proposed by
                   SDP/SRP or any other encapsulation in IB ? No.
                   Similarly there nothing binding the rest of the IB
                   world to follow the rules set by IPoIB.

        We expect a small set of packets at a QP. It is easy to
        differentiate between these based on the signatures of the
        draft. The only packet we expect on an IPoIB QP are : IPv4/v6
        and ARP. These in themselves are easily distinguishable.

> 
> While I don't buy the performance arguement, I sympathize with
> Vivek on his feeling toward EtherType. Architecturally it is ugly
> to force one into the payload. The right approach would have the
> framing layer carrying a field identifying the type of data being
> encapsulated. But BTH wasn't designed for encapusulation purpose.
> That is supposed to be the job and the only purpose of raw datagrams. 
> Unfortunately due to various reasons the latter has been considered
> by many as not fit for the job of IPoIB encapsulation.
> 
> Aside from the architectural correctness, I only see two valid concerns
> against not including EtherType:
> 
> 1. Compatibility issue - how many existing stack implementations
> expect EtherType, or part of framing header to be present in the
> datastream? Although the driver can always fabricate a pseudo header
> if one is needed but is missing, it may incur some performance
> penalty since it won't be in-line.

[ I thought you said you don't buy any performance argument :). ]

        Does an 'classical IP over ATM' or 'ITX' return ethertype ?
        Secondly, if an implementation needs a type the driver can
        return it to the upper layers in any form one desires - as an
        argument to a function, as a type in the buffer header,
        another appended header. It can surely be done very
        efficiently. 

        Anyway, that is an internal issue of the implementation. Why
        impose it on implemenations that don't need/want it ? The ones
        that wnat it can add the header internally. Why put it on the
        wire where it is not needed ?

> 
> 2. Not including Ethertype forbids one from sharing the same QP with
> other protocols outside of IPv4/ARP/IPv6. Is this a legitimate concern?
> Note that this concerns only applies unicast. For multicast, by
> embedding a special IPoIB signature in MGIDs for IPv4/ARP/IPv6 use
> only, demuxing can be based on MGID so sharing QPs won't be an issue.
> 

        As noted before one will need everybody to follow the same
        rules as IPoIB to work this way. That is unlikely to happen.
        Only IBTA can dictate universal rules for all of IB.

> Before the two questions are fully understood, I'm leaning slightly
> toward including the EtherType, just to be on the safe side.

	Another method out of this logjam is allow both - hold on :).

        In IP over ethernet one sets up the interface with SNAP (like
        Ethertype) and NOSNAP(no ethertype). Let us follow that. If
        set SNAP/Ethertype the interface will send and expect
        Ethertype header. If not it will work without it. This allows
        an endpoint to use whatever option they prefer.

        In another year or two we will know which is preferred and
        will live with that. 

Vivek

> 
> Jerry
> 
> >From: Vivek Kashyap <kashyapv@us.ibm.com>
> >Subject: Re: [Ipoverib] IPoIB architecture/encapsulation
> >To: yaronh@voltaire.com (Yaron Haviv)
> >Date: Thu, 13 Dec 2001 16:23:24 -0800 (PST)
> >Cc: ipoverib@ietf.org
> >> Just trying to finalize my arguments 
> >> 
> >> As for having 4 bits or 4 byes ethertype:
> >> Comparing 4 bits takes exactly the same resources as 4 bytes
> >> If the header size is constant the header processing is not complicated,
> >> when the next header offset is variable (in case of IP options, LLC, etc')
> >> you need to parse the packet
> >
> >        Is it less complicated than not having the header at all ?
> >        Sure one can work with the header but why do something when
> >        there is no need for it? 
> >
> >        In my previous mail I explained the advantages of protocol
> >        signature over the 'ethertype'. Those advantages are not
> >        overcome by your argument. All you are saying is that one can
> >        make the receive processing in the presence of 4 byte
> >        'ethertype' as good as the 'protocol signature' no-header
> >        processing.  
> >
> >> Its not the case here you just start the IP header processing in offset 4,
> >> after you checked the first 4 bytes
> >> Putting the 4 bits in pieces of the IB header is less natural
> >
> >        I am afraid I have lost you. There is no putting of 4 bits of
> >        pieces in IB header. The 4 bits I refer to are already part of
> >        the packet.
> >
> >        The IP packet starts with the IP version (a 4 bit field) and
> >        the ARP encapsulation starts with hardware type (a 16 bit
> >        value). These values are different in the first 4 bits as I
> >        explained in an earlier mail about the values. Hence the
> >        information is already present in the packet and there is no
> >        need to add a header. As noted earlier we are disambiguating
> >        between 2 protocols only.
> >
> >
> >> We have done several NIC drivers in different OS's, including an
> >IPoIB > driver > So I could tell you based on our experience it is not
> >a problem
> >
> >        Anything can be made to work. I only object to have to make
> >        something work that has not been shown to bring anything to
> >        the table.
> >
> >> 
> >> As for the demuxing 
> >> 1. The network driver doesn't process the ARP's, the OS above it does
> >
> >        Yes, but the driver or some layer just above it demultiplexes
> >        the streams.
> >
> >> 2. You separate to different queues when A) you want to save context switch,
> >> copies or b) when you want to avoid going over the queue and look for your
> >> specific information, this is not the case here we are in the same context,
> >> the ARP messages are sporadic and could wait in the queue for a while until
> >> I will get to them 
> >	
> >        Sure one can do as you say. Put them all in one queue and then
> >        process them on the type. Why would one need to invent a
> >        header for that? All one has to do is to look at the protocol
> >        signature. 
> >
> >> 
> >> As I mentioned before you are looking for optimizations in the wrong place,
> >> there are many other places where we could optimize in future work 
> >
> >        I am pointing out that if there is no advantage from more
> >        work then less work is always better than more work.
> >
> >Vivek
> >
> >
> >> -----Original Message-----
> >> From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
> >> Vivek Kashyap
> >> Sent: Thursday, December 13, 2001 1:41 AM
> >> To: yaronh-voltaire.com@mail.barak.net.il
> >> Cc: ipoverib@ietf.org
> >> Subject: Re: Re: [Ipoverib] IPoIB architecture/encapsulation
> >> 
> >> Yaron,
> >>         I am afraid we have sidetracked. The comment about performance
> >>         was different. The point I was making was that generally you
> >>         would get better performance if you had less to do. It doesn't
> >>         mean that one cannot by specialisation mitigate the negatives
> >>         of a more complicated task.
> >> 
> >>           delete/add header vs. no such work. I like no work.
> >> 
> >>         Here is the problem and the reasoning:
> >> 
> >>         The network stack doesn't change you merely add a network
> >>         interface driver. This driver has the task of determining the
> >>         upper layer to call. It can call it directly or it can slot
> >>         the packet in some type based queue. The interface driver will
> >>         determine the type from the packet stream. It can do so in
> >>         many ways. One of the ways is by using the protocol signature.
> >>         Another is by way of 'ethertype' and a third is by mandating a
> >>         QP per protocol.
> >> 
> >>         The use of 'ethertype' is not universal. A choice by IPoIB to
> >>         use ethertype does not mean 'protocol X' over IB will use the
> >>         same encapsulation and therefore you cannot rely on
> >>         'ethertype' differentiating your packets.
> >> 
> >>         So the 'ethertype' is of use in differentiating IPoIB
> >>         protocols only (IPv4, IPv6 and ARP). For this task we have two
> >>         other methods that require no change in the on-wire
> >>         encapsulation: use the protocol signatures (4 bits in the
> >>         begining) and use separate QPs per protocol. Of the two : use
> >>         protocol signatures doesn't require any on-wire change,
> >>         doesn't require the driver to strip headers, doesn't require
> >>         the TCP layer to keep space for the header. That is why it is
> >>         preferable. 
> >> 
> >>         In essence the protocol signature solution gives us the same
> >>         returns with the additional advantage of :
> >> 
> >>                 - no need to reflect it on the wire
> >>                 - no removal/addition of headers
> >>        
> >> 
> >> Vivek
> >> 
> >> 
> >> >
> >> > Vivek,
> >> >
> >> > I generally agrees with Mike, comparing 4 bit vs. 4 bytes has no impact on
> >> mem access as well as CPU cycles
> >> > My assumption is that OSV's will not change the way their TCP/IP stack is
> >> built and separate ARP's from IP's, they will just add IPoIB as another net
> >> interface driver
> >> > The demultiplexing doesn't add much value since it is done in the same
> >> kernel context, there is no urgency in dealing with the ARP's, there aren't
> >> many ARP's, it also relates to the stack work (ARP response might release
> >> pending IP packets)
> >> >
> >> > it looks to me like the 4 bit option is not the straightforward approach
> >> >
> >> > If you want to improve performance significantly, without affecting the
> >> stack you should do things similar to what the new NIC's do:
> >> > -  Offload checksum (avoid cache trashing)
> >> > -  Increase MTU, MSS (avoid UDP fragmentation, TCP segmentation)
> >> > -  Suppress Interrupts
> >> > -  Improve memory management
> >> > These things could be done with RC/RD and using some IB tricks
> >> >
> >> > Yaron
> >> >
> >> > Original Message:
> >> > -----------------
> >> > From: Vivek Kashyap kashyapv@us.ibm.com
> >> > Date: Wed, 12 Dec 2001 12:27:20 -0800 (PST)
> >> > To: yaronh@voltaire.com, ipoverib@ietf.org
> >> > Subject: Re: [Ipoverib] IPoIB architecture/encapsulation
> >> >
> >> >
> >> > Yaron,
> >> >
> >> > >
> >> > > Few questions & suggestions
> >> > >
> >> > > I'm trying to understand way not to take the straightforward approach
> >> for
> >> > > the ethrtype field
> >> > > And just add that before the IP header in the payload area, if alignment
> >> is
> >> > > needed so align it
> >> >
> >> >         Since we already filter the recepients by way of QP to one
> >> >         (IPv6) or two (IPv4/ARP) why go to this trouble? The type of
> >> >         work is the same whether you see 4 bits or 32 bits. But it is a
> >> >         lot less with 4 bits.
> >> >
> >> >         The interface driver will distinguish the types and call the
> >> >         upper layer accordingly.
> >> >
> >> >                 It is easier with 4 bits than with 4 bytes. Some people
> >> >                 have mentioned performance - I would say that if you
> >> >                 have less to do you are more performant. With 4 bits
> >> >                 you have less to do. Secondly, if you are in
> >> >                 'performance' mode switching in hardware then it is
> >> >                 even better since you don't have to strip the header
> >> >                 but merely have to pass the buffer to the relevant
> >> >                 upper layer directly.
> >> >
> >> >         The interface driver does not have to do the followig when
> >> >         using protocol signatures:
> >> >
> >> >     - It doen't have to add a header when sending
> >> >     - It doesn't have to receive header when receiving
> >> >
> >> >   These tasks are required for if one invents an 'ethertype'.
> >> >
> >> >         The upper layer doesn't have to keep space for the header, the
> >> >         ethertype, when allocating data for sending if the protocol
> >> >         signatures are to be used.
> >> >
> >> >         I don't see any gain if an artificial header is included to
> >> >         distinguish protocols in IPoIB. I see more work. We already
> >> >         know, by definition, it is IP or ARP. Just test which one it
> >> >         is by looking at the first 4 bits.
> >> >
> >> >
> >> > >
> >> > > I might be missing something here so forgive me if I do
> >> > > It looks to me like TCP stacks are used to get a packet with everything
> >> > > including Ethernet header, it is not like we are doing full header/data
> >> > > separation that could help the stack
> >> > > On the worst case if the TCP implementation has problems with it we
> >> could
> >> > > have 2 data segments in the receiving side, one for the first word,
> >> second
> >> > > for the rest
> >> >
> >> > Things are generally done in simpler ways. The IP stack gets the IP
> >> > packet. The lower headers are stripped away. If you don't have the
> >> > any header to strip in your interface driver all the better.
> >> >
> >> > >
> >> > > Another thing that was mentioned in the recent emails was that IPoIB is
> >> > > defined just for UD
> >> >
> >> >         If you look at the beginning of this thread (the mail that
> >> >         started it all) I have discussed capability flags and how
> >> >         RC/RD/UC connections can be setup and work interoperably with
> >> >         UD only implementations. IPoIB over UD is the default and
> >> >         madatory.
> >> >
> >> >
> >> > Vivek
> >> >
> >> > _______________________________________________
> >> > IPoverIB mailing list
> >> > IPoverIB@ietf.org
> >> > http://www.ietf.org/mailman/listinfo/ipoverib
> >> >
> >> > --------------------------------------------------------------------
> >> > mail2web - Check your email from the web at
> >> > http://mail2web.com/ .
> >> >
> >> >
> >> 
> >> 
> >> --
> >> Vivek Kashyap
> >> IBM
> >> kashyapv@us.ibm.com
> >> vivk@us.ibm.com
> >> 503 578 3422 (o)
> >> 
> >> _______________________________________________
> >> IPoverIB mailing list
> >> IPoverIB@ietf.org
> >> http://www.ietf.org/mailman/listinfo/ipoverib
> >> 
> >
> >
> >-- 
> >Vivek Kashyap
> >Linux Technology Center, IBM
> >kashyapv@us.ibm.com 
> >vivk@us.ibm.com 
> >503 578 3422 (o)
> >
> >_______________________________________________
> >IPoverIB mailing list
> >IPoverIB@ietf.org
> >http://www.ietf.org/mailman/listinfo/ipoverib
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 14 15:25:57 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA17430
	for <ipoverib-archive@odin.ietf.org>; Mon, 14 Jan 2002 15:25:56 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id PAA22906
	for ipoverib-archive@odin.ietf.org; Mon, 14 Jan 2002 15:25:59 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA22846;
	Mon, 14 Jan 2002 15:22:57 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA22797
	for <ipoverib@optimus.ietf.org>; Mon, 14 Jan 2002 15:22:53 -0500 (EST)
Received: from patan.sun.com (patan.Sun.COM [192.18.98.43])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA17340
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 15:22:49 -0500 (EST)
Received: from sunmail1.Sun.COM ([129.145.1.2])
	by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id NAA16942
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 13:22:51 -0700 (MST)
Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.88.130])
	by sunmail1.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1-Sun.COM.mod.2) with ESMTP id MAA25551
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 12:24:02 -0800 (PST)
Received: from taipei (taipei.Eng.Sun.COM [129.146.85.178])
	by jurassic.eng.sun.com (8.12.2+Sun/8.12.2) with SMTP id g0EKMoqT104840
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 12:22:51 -0800 (PST)
Message-Id: <200201142022.g0EKMoqT104840@jurassic.eng.sun.com>
Date: Mon, 14 Jan 2002 12:20:23 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
To: ipoverib@ietf.org
MIME-Version: 1.0
Content-Type: MULTIPART/mixed; BOUNDARY=Bed_of_Mussels_472_000
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.6_06 SunOS 5.8 sun4u sparc 
Subject: [Ipoverib] IPoIB link and multicast draft
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

--Bed_of_Mussels_472_000
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: shSNljhSErDFfTkwxd0xvQ==

I still don't see the link and multicast draft announced by the
document editor. So I'm enclosing the draft below.

A few high points:

1. P_Key/IB partition defines an IPoIB link

2. MTU, Q_Key are link attributes

3. A fixed, algorithmic mapping that embeds an IPoIB signature
is used to map IP multicast addrs to MGIDs.

4. IPv6 uses a different IPoIB signature than IPv4.

5. An IPoIB link can span more than one IB link (kind of implied by
   1 above already.)
   
6. An all-node IB multicast/broadcast group on top of an IPoIB link
lays the foundation for all further IPoIB operations.

There are still some weak areas in the draft. But I expect them to
be addressed through a few rounds of revisions in the next couple of
months.

Please comment!

Jerry

--Bed_of_Mussels_472_000
Content-Type: TEXT/plain; name="draft-ietf-ipoib-link-multicast-00.txt"; charset=us-ascii; x-unix-mode=0664
Content-Description: draft-ietf-ipoib-link-multicast-00.txt
Content-MD5: qhkl1w58b6vtPJCLYKjssg==







INTERNET-DRAFT                                            H.K. Jerry Chu
<draft-ietf-ipoib-link-multicast-00.txt>                Sun Microsystems
                                                           Vivek Kashyap
                                                                     IBM
Expires: July, 2002                                        January, 2002


             IP link and multicast over InfiniBand networks


Status of this Memo

   This document is an Internet-Draft and is in full conformance with
   all provisions of Section 10 of RFC2026.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups. Note that other
   groups may also distribute working documents as Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time. It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

   Copyright (C) The Internet Society (date).  All Rights Reserved.


Abstract

   This document specifies a method for setting up IP subnets and
   multicast services over InfiniBand(TM) networks. Discussions in this
   document are applicable to both IPv4 and IPv6, unless explicitly
   specified. A separate document will cover unicast and encapsulation
   of IP datagrams over InfiniBand networks.


1. Introduction

   InfiniBand Architecture (IBA) defines four layers of network services
   corresponding to layer one through layer four of the OSI reference
   model.  For the purpose of running IP over an InfiniBand (IB)
   network, the IB network and all its link, network, and transport



Chu & Kashyap                                                   [Page 1]

draft-ietf-ipoib-link-multicast-00.txt                      January 2002


   layers collectively constitute the data link layer to the IP stack.

   An IP network is often divided into many subnets connected by IP
   routers.  Subnetting allows the containment of broadcast traffic
   within a single subnet. It also provides certain degree of isolation
   between nodes on different subnets. The latter may be an important
   consideration for administration purpose.

   This document will focus on all the steps required to lay out an IP
   network on top of an IB network. It will describe all the elements an
   IP over InfiniBand (IPoIB) link consists of, how to configure its
   associated link attributes, and how to set up basic broadcast and
   multicast services on an IPoIB link.

2. Terminology

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119].

3. Basic IPoIB Transport - Unreliable Datagram

   InfiniBand defines four types of transport services [IBTA]. They are
   reliable connection, unreliable connection, reliable datagram,
   unreliable datagram. IBA also defines a special raw datagram service
   for encapsulation purpose. Both unreliable datagram and raw datagram
   define support for multicast. They provide the basic transport
   mechanism that best matches the IP datagram paradigm.

   IB unreliable datagram provides many additional transport features
   such as the partition key (P_Key) protection, multiple queue pairs
   (QPs), and Q_Key protection. Moreover, it requires a 32-bit invariant
   CRC checksum, which provides a much stronger protection against data
   corruption, compared with the 16-bit CRC that a raw datagram carries.

   For these reasons, IB unreliable datagram is considered to be a much
   better choice as the basic IPoIB transport than the raw datagram, and
   is chosen as the default IPoIB transport mechanism for the rest of
   discussions in this document.

   An IB unreliable datagram contains the following headers:

   o  Local Route Header (LRH) - provides IB link-layer addressing
      information. An IB link layer address is based on a 16-bit
      identifier called Local Identifier (LID), and is used by IB
      switches to relay packets within an IB subnet.

   o  Global Route Header (GRH) - provides routing information for IB



Chu & Kashyap                                                   [Page 2]

draft-ietf-ipoib-link-multicast-00.txt                      January 2002


      routers to relay packets between IB subnets inside an IB fabric.
      GRH is only required for all multicast packets and any unicast
      packet that is destined to a node in a different IB subnet. GRH
      carries an IB network-layer address, which is an 128-bit
      identifier called Global Identifier (GID) that closely mimics IPv6
      addressing architecture [AARCH].

   o  Base Transport Header (BTH) - provides various information,
      including P_Key, destination queue pair number (QPN) for IB
      transport services.

   o  Datagram Extended Header (DETH) - provides additional IB
      information such as Q_Key, source queue pair number for datagram
      services.

   From the perspective of IP over IB encapsulation, all the above IB
   headers are considered as link layer encapsulation for IP datagrams.

4. IB Multicast Architecture

   IBA defines two layers of multicast services. Its link layer uses
   multicast LIDs (MLIDs), which are allocated by the Subnet Manager
   (SM) and fall in the range between 0xC0000 to 0xFFFE (approximately
   16k). MLIDs are used by IB switches to program their multicast
   forwarding tables. An IB switch implementation may support much fewer
   MLIDs in its forwarding table though.

   IB network layer uses multicast GIDs (MGIDs), which closely resemble
   IPv6 multicast addresses [AARCH] as shown in the following.

   |   8    |  4 |  4 |                  112 bits                   |
   +------ -+----+----+---------------------------------------------+
   |11111111|flgs|scop|                  group ID                   |
   +--------+----+----+---------------------------------------------+

      11111111 at the start of the address identifies the address as
      being a multicast address.

                                    +-+-+-+-+
      flgs is a set of 4 flags:     |0|0|0|T|
                                    +-+-+-+-+

         The high-order 3 flags are reserved, and must be initialized to
         0.

         T = 0 indicates a permanently-assigned ("well-known") multicast
         address, assigned by the global internet numbering authority.




Chu & Kashyap                                                   [Page 3]

draft-ietf-ipoib-link-multicast-00.txt                      January 2002


         T = 1 indicates a non-permanently-assigned ("transient")
         multicast address.

      scop is a 4-bit multicast scope value used to limit the scope of
      the multicast group.  The values are:

         0  reserved
         1  node-local scope
         2  link-local scope
         3  (unassigned)
         4  (unassigned)
         5  site-local scope
         6  (unassigned)
         7  (unassigned)
         8  organization-local scope
         9  (unassigned)
         A  (unassigned)
         B  (unassigned)
         C  (unassigned)
         D  (unassigned)
         E  global scope
         F  reserved

      group ID identifies the multicast group, either permanent or
      transient, within the given scope.

   MGIDs are mainly used by IB routers when forwarding multicast packets
   to remote IB subnets that are part of a multicast forwarding tree.
   Since every IB multicast packet is required to carry both LRH and
   GRH, a valid MGID and a valid MLID are both needed before a valid IB
   multicast packet can be constructed.

   An IB multicast group is uniquely identified by a valid MGID. Before
   a MGID can be used within an IB subnet, either as a destination
   address of a multicast packet, or representing a multicast group that
   an IB node can join, a "MCGroupRecord" corresponding to the MGID must
   be created through the Subnet Administrator (SA). Besides the MGID,
   the creator must supply values of the path MTU, Q_Key, TClass,
   FlowLabel, HopLimit that are appropriate for all the potential
   clients of the multicast group to use. In return, SA will allocate a
   MLID to be used by switches in the local IB subnet.

   Note that MLIDs are allocated and managed by SM when new MGIDs are
   created though the creation of MCGroupRecords. The number of valid
   MLIDs that are available in a given IB subnet is limited by the
   implementation-dependent size of multicast forwarding table of IB
   switches.  Since the number can be small, reuses of MLIDs for MGIDs
   may be inevitable. Implementation should nevertheless avoid sharing



Chu & Kashyap                                                   [Page 4]

draft-ietf-ipoib-link-multicast-00.txt                      January 2002


   the same MLID among high volume multicast groups in order to reduce
   software filtering overhead and attain higher efficiency.

   Unreliable multicast is defined by IBA as an optional functionality
   for channel adaptors (CAs) and switches. In today's IP technology,
   link multicast has become an indispensable function for better
   supporting a modern IP network. For this reason, it is required that
   an IPoIB fabric supports multicast. This includes all the CAs and
   switches that make up an IP network.

5. IB Links vs IPoIB Links

   A link segment on top of which an IP subnet can be configured is
   defined in [IPV6] as a communication facility or medium over which
   nodes can communicate at the "link" layer.  For most types of
   communication media, the boundary between different data link
   segments follows the physical topology of the network connectivity,
   and is pretty obvious. E.g. an Ethernet network connected by
   switches, hubs, or bridges usually forms a single link segment and
   broadcast/multicast domain.  Different Ethernet segments can be
   connected by IP routers at the network layer.

   InfiniBand defines its own link-layer and subnets consisting of nodes
   connected by IB switches. However, the IPoIB link boundary needs not
   follow the IB link boundary. Nodes residing on different IB subnets
   can still communicate directly with one another through IB routers at
   the InfiniBand network layer. The same applies to multicast as well.
   I.e.  nodes on the same IB subnet can exchange multicast packets with
   one another all within the same subnet through the IB link multicast
   facility. But even nodes on different IB subnets can still exchange
   multicast packets with one another using IB network-layer multicast.

   The ultimate requirement for two nodes in the same IB fabric to
   communicate at the IB level, besides the physical connectivity, is a
   common P_Key.

   Partitioning in IB provides an isolation mechanism among nodes in an
   IB fabric, much like VLANs in an Ethernet network.  Each port of an
   endnode contains a P_Key table of all the valid P_Keys the port is
   allowed to use.  The P_Key table is set up by the SM of the local IB
   subnet.  Each QP is programmed with a P_Key from the local P_Key
   table.  This P_Key is carried in the BTH of all the outgoing packets
   from the QP, and is used to compare against the P_Key in the BTH of
   all the incoming packets to the QP. Reception of an invalid P_Key
   causes the packet to be discarded. IB switches may optionally enforce
   partition checking too.

   Therefore P_Key and IB partition are the most natural choice for



Chu & Kashyap                                                   [Page 5]

draft-ietf-ipoib-link-multicast-00.txt                      January 2002


   defining IPoIB link boundary. It also affords much flexibility to the
   network administrators when different links are set up in a large
   network. This is very similar to VLANs in Ethernet.

6. Setting up an IPoIB Link

   A network administrator defines an IPoIB link by setting up an IB
   partition and assigning it a unique P_Key. An IB partition may or may
   not span multiple IB subnets. But whether it does or not is mostly
   irrelevant to IPoIB.

   Each node attached to the IB partition MUST have one of its CA
   assigned the P_Key to use.

   Once an IB partition is established for IPoIB use, the link MTU and
   Q_Key are two other important attributes that must be chosen before
   the IPoIB link can be configured.

6.1 Maximum Transmission Unit

   IB defines five permissible maximum payload sizes. They are 256, 512,
   1024, 2048 and 4096 bytes.  [IPV6] requires a link MTU of 1280 bytes
   or greater. This leaves only 2048 and 4096 bytes as two acceptable
   choices for IPv6. Channel adaptors supporting a maximum payload size
   less than the minimal MTU requirement can still expose an acceptable
   link MTU to IP through an adaptation layer that fragments larger
   messages into smaller IB packets, and reassembles them on the
   receiving end. But this must be done in a way that is completely
   transparent to the IP stack.

   It is up to the network administrator to select a link MTU to use
   when configuring an IPoIB link. The link MTU SHALL not be greater
   than the maximum payload size of any CA or switch connected to the
   IPoIB link.

   In general a larger link MTU can potentially offer a better
   throughput performance. The caveat is that once a link MTU is chosen
   for a given IPoIB link, nodes connected by CAs of a smaller maximum
   payload size won't be able to join the link unless the whole link and
   all the nodes attached to it are reconfigured to use a smaller MTU.

   Note that the above discussion assumes that IP datagrams are fully
   encapsulated in the payload of IB unreliable datagrams. The actual
   MTU size, i.e., the payload size available for IP datagrams to use,
   may be slightly smaller. This will depend on the actual IPoIB
   encapsulation scheme, which will be covered in a separate document.

   Note also that in case an IPoIB link spans more than one IB subnet,



Chu & Kashyap                                                   [Page 6]

draft-ietf-ipoib-link-multicast-00.txt                      January 2002


   the IPoIB link MTU MUST not be set to greater than the path MTU of
   any path connecting two nodes in the same IB partition. It is up to
   the network administrator to determine the appropriate path MTU value
   that will work for any node in the same IPoIB link.

6.2. IPoIB Link Q_Key

   A Q_Key is programmed by the source QP in every IB datagram, and is
   verified by the destination QP against the Q_Key it has been
   assigned. A Q_Key violation will cause the offending datagram to be
   dropped, and a Q_Key violation trap to be raised.

   A Q_Key must be selected to be used by all the QPs attached to an
   IPoIB link. It is recommended that a controlled Q_Key be used with
   the high order bit set. This is to prevent non-privileged software
   from fabricating and sending out bogus IP datagrams. All QPs
   configured to use on a given IPoIB link SHALL be assigned the same
   per-link Q_Key.

6.3 Other Link Attributes

   TClass, FlowLabel, and HopLimit are three other attributes that are
   required for an IPoIB link covering more than a single IB subnet.
   The selection of these values are implementation dependent. But it
   must take into account the topology of IB subnets comprising the
   IPoIB link to ensure successful communication between any two nodes
   in the same IPoIB link.

7. The IPoIB All-Node Multicast and Broadcast Group

   Once an IB partition is created with link attributes identified for
   an IPoIB link, the network administrator must create a special IB
   multicast group for every node on the IPoIB link to join. This is
   achieved through the creation of "MCGroupRecord" in each IB subnet
   that the IB partition encompasses, as described in section 4 above.

   The MGID will have the P_Key of the IB partition that defines the
   IPoIB link embedded in it. A special signature is also embedded to
   identify the MGID for IPoIB use only. For IPv4 over IB, the signature
   will be "0x401B". For IPv6 over IB, the signature will be "0x601B".

   For an IPv4 subnet, the MGID for this special IB multicast group
   SHALL have the following format:

   |   8    |  4 |  4 |     16 bits     | 16 bits | 48 bits  | 32 bits |
   +--------+----+----+-----------------+---------+----------+---------+
   |11111111|0001|scop|<IPoIB signature>|< P_Key >|00.......0|<all 1's>|
   +--------+----+----+-----------------+---------+----------+---------+



Chu & Kashyap                                                   [Page 7]

draft-ietf-ipoib-link-multicast-00.txt                      January 2002


   For an IPv6 subnet, the format of the MGID SHALL look like this:

   |   8    |  4 |  4 |     16 bits     | 16 bits |       80 bits      |
   +--------+----+----+-----------------+---------+--------------------+
   |11111111|0001|scop|<IPoIB signature>|< P_Key >|000.............0001|
   +--------+----+----+-----------------+---------+--------------------+

   As for the scop bits, if the IPoIB link is fully contained within a
   single IB subnet, the scop bits SHALL be set to 2 (link-local).
   Otherwise the scope will be set higher.

   A MCGroupRecord will be created with all the IPoIB link attributes
   described before, including the link MTU, Q_Key, TClass, FlowLabel,
   and HopLimit. When a node is attached to an IPoIB link identified by
   a P_Key, it must look for a special, all-node multicast/broadcast
   group to join. This is done by constructing the MGID with the link
   P_Key and the IPoIB signature. The node SHOULD always look for a MGID
   of a link-local scope first before attempting one with a greater
   scope.

   Once the right MGID and MCGroupRecord are identified, the local node
   SHOULD use the link MTU recorded in the MCGroupRecord. It MUST accept
   a smaller MTU if one is advertised through the link MTU option of a
   router advertisement [DISC].

   In case the link MTU is greater than the maximum payload size that
   the local HCA can support, the node can not join the IPoIB link and
   operate as an IP node.

   After the right MTU is determined, the local node must join the
   special all-node multicast/broadcast group by calling the SA to
   create a MCMemberRecord corresponding to the MGID. The SA will return
   all the link attributes for the local node to use. The node MUST use
   these attributes in all future multicast operations to the local
   IPoIB link.  The broadcast group for IPv4 will serve to provide a
   broadcast service for protocol like ARP to use.

   In addition to the all-node multicast/broadcast group, an all-router
   multicast group SHOULD be created at link configuration time if an IP
   router will be attached to the link. This is to facilitate IP
   multicast operations described later. A MCGroupRecord for the all-
   router MGID must be created in every IB subnet that the IPoIB link
   encompasses. The format of the all-router MGID will be covered in
   next section.

8. Mapping for other Multicast Groups

   The support of general IP multicast [IPMULT] over IB is similar to



Chu & Kashyap                                                   [Page 8]

draft-ietf-ipoib-link-multicast-00.txt                      January 2002


   the case of the special all-node multicast/broadcast group discussed
   above. An algorithmic mapping is used so that given an IP multicast
   address, individual host can compute the corresponding IB multicast
   address (MGID) all by itself without having to consult an external
   entity. This also removes the need for an externally maintained IP to
   IB multicast mapping table.

   The IPoIB multicast mapping is defined as follows. The same mapping
   function is used for both IPv4 and IPv6 except the IPoIB signature
   field.

   |   8    |  4 |  4 |     16 bits     | 16 bits |      80 bits       |
   +------ -+----+----+-----------------+---------+--------------------+
   |11111111|0001|scop|<IPoIB signature>|< P_Key >|      group ID      |
   +--------+----+----+-----------------+---------+--------------------+

   Since a MGID allocated for transporting IP multicast datagrams is
   considered only a transient link-layer multicast address, all IB
   MGIDs allocated for IPoIB purpose SHOULD have T = 1. The scope bits
   SHALL be the same as that of the all-node MGID for the same IPoIB
   link.

   An IP multicast address is used together with a given IPoIB link
   P_Key to form the MGID of the IB multicast group. For IPv6 the lower
   80-bit of the group ID is used directly in the lower 80-bit of the
   MGID. For IPv4, the group ID is only 28-bit long and the rest of the
   80 bits are filled with 0.

   The rest of the bits are the same as those of the all-node MGID.
   E.g. on an IPoIB link that is fully contained within a single IB
   subnet with a P_Key of 8, the MGIDs for the all-router multicast
   group with group ID 2 [AARCH, IGMP2] are:

   FF12:401B:8:0:0:0:0:2

   or

   FF12:401B:8::2

   for IPv4 in a compressed format, and

   FF12:601B:8:0:0:0:0:2

   or

   FF12:601B:8::2

   for IPv6 in a compressed format.



Chu & Kashyap                                                   [Page 9]

draft-ietf-ipoib-link-multicast-00.txt                      January 2002


   A special case exists for the IPv4 limited broadcast address
   "255.255.255.255" [HOSTS]. The address SHALL be mapped to the
   broadcast MGID for IPv4 networks as described in section 7 above.
   Also the IPv6 all-node multicast address "FF0X::1" [AARCH] will be
   mapped to the the special all-node MGID for IPv6 networks.

   When a node wishes to join an IP multicast group on a local link, it
   first needs to construct the corresponding MGID, using the rule
   described above. Note that it must remember the scope bits from the
   all-node MGID, and use the same scope in all later MGIDs it
   constructs.

   The local node then checks with SA to see if a MCGroupRecord
   corresponding to the MGID already exists. If not, one must be created
   first. The MCGroupRecord MUST be created with the IPoIB link MTU. For
   the rest of the attributes, it is recommended that it uses the same
   values from the all-node multicast/broadcast group corresponding to
   the link.

   Note that for an IPoIB link that spans more than one IB subnet
   connected by IB routers, adequate multicast forwarding support at the
   IB level is required for multicast packets to be forwarded properly
   to members in remote IB subnets. The specific mechanism for this will
   be covered in [IBTA], and is out of scope of this document.

   Once the IB multicast group is identified, the node must join the
   group, unless it is a member already, by calling the SA to create a
   MCMemberRecord corresponding to the MGID.  The join call enables SM
   to program local IB switches and routers with the new multicast
   information.  Specifically it causes an IB switch to add the LID of
   the caller to its forwarding table entry corresponding to the MLID
   allocated for the group.  It also causes an IB router to attach
   itself to the IB multicast tree corresponding to the MGID.

   When a node leaves an IP multicast group, it SHOULD delete the
   MCMemberRecord from the SA. This allows the SA to free up related
   resources. SM should delete MCGroupRecords that are no longer in use,
   and free up the MLIDs allocated for them. The specific algorithm is
   implementation-dependent, and therefore is out of scope of this
   document.

   In order to send a packet destined for an IP multicast address, a
   node must first check if a MCGroupRecord for the corresponding MGID
   of the outbound link exists or not. If one already exists, the MLID
   allocated by the SM for the MCGroupRecord is used as the DLID for the
   packet. Otherwise, it means no member exists on the local link.  The
   packet should be forwarded to the all-router multicast group
   described before. If one doesn't already exist, it implies no router



Chu & Kashyap                                                  [Page 10]

draft-ietf-ipoib-link-multicast-00.txt                      January 2002


   presence on the local subnet. The packet can then be silently
   dropped.

   Note that the local node MUST be notified when an IB multicast group
   corresponding to the MGID ever comes into existence later. This
   signifies that an interested party just showed up on the local link
   and therefore must be copied.

9. Support for IP Multicast Routing

   IP multicast routing requires a router to receive a copy of every
   link multicast packet on a locally connected link [IPMULT, IP6MLD].
   For Ethernet this is usually done by turning on promiscuous multicast
   mode on a locally connected Ethernet interface.

   Unfortunately IBA does not support promiscuous multicast mode.
   Therefore the IPoIB driver should forward a copy of every outbound
   multicast datagram to the MGID corresponding to the all-router
   multicast group. This is to ensure multicast packets be properly
   forwarded to remote IP networks.

10. Security Considerations

   All the operations for creating and configuring an IPoIB link
   described in this document, including assigning P_Keys to CAs,
   creating MCGroupRecords and MCMemberRecords in SA, creating and
   attaching QPs to IB multicast groups,... etc are privileged
   operations, and MUST be protected by the underlying operating system.
   This is to prevent malicious, non- privileged software from hijacking
   important resources and configurations.  E.g. A bogus all-node IPoIB
   multicast group may prevent a proper one from being created when the
   network administrator tries to set up a link.

   Controlled Q_Keys SHOULD be used in IB multicast groups in order to
   prevent non-privileged software from fabricating IP datagrams to
   send, as mentioned in section 6.2.

11. Acknowledgments

   The authors would like to thank Bruce Beukema, David Brean, Dan
   Cassiday, Thomas Narten, Erik Nordmark, Greg Pfister, Renato Recio,
   David L. Stevens, and Madhu Talluri for their suggestions and many
   clarifications on the IBA specification.

12. References

   [AARCH]   Hinden, R. and S. Deering "IP Version 6 Addressing
             Architecture", RFC 2373, July 1998.



Chu & Kashyap                                                  [Page 11]

draft-ietf-ipoib-link-multicast-00.txt                      January 2002


   [DISC]    Narten, T., Nordmark, E. and W. Simpson, "Neighbor
             Discovery for IP Version 6 (IPv6)", RFC 2461, December
             1998.

   [HOSTS]   Braden R., "Requirements for Internet Hosts --
             Communication Layers", RFC 1122, October 1989

   [IBTA]    InfiniBand Architecture Specification, Release 1.0.a by
             InfiniBand Trade Association at www.infinibandta.org

   [IGMP2]   Fenner W., "Internet Group Management Protocol, Version 2",
             RFC 2236, November 1997.

   [IPMULT]  Deering S., "Host Extensions for IP Multicasting", RFC
             1112, August 1989.

   [IPV6]    Deering, S. and R. Hinden, "Internet Protocol, Version 6
             (IPv6) Specification", RFC 2460, December 1998.

   [IP6MLD]  Deering S., Fenner W., Haberman B., "Multicast Listener
             Discovery (MLD) for IPv6", RFC 2710, October 1999.

   [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
             Requirement Levels", BCP 14, RFC 2119, March 1997.


13. Author's Address

   H.K. Jerry Chu
   901 San Antonio Road, UMPK17-201
   Palo Alto, CA 94303-4900
   USA

   Phone: +1 650 786-5146
   EMail: jerry.chu@sun.com


   Vivek Kashyap
   IBM
   15450, SW Koll Parkway
   Beaverton, OR 97006

   Phone: 503 578 3422
   EMail: vivk@us.ibm.com



14. Full Copyright Statement



Chu & Kashyap                                                  [Page 12]

draft-ietf-ipoib-link-multicast-00.txt                      January 2002


   Copyright (C) The Internet Society (2001>.  All Rights Reserved.

   This document and translations of it may be copied and furnished to
   others, and derivative works that comment on or otherwise explain it
   or assist in its implementation may be prepared, copied, published
   and distributed, in whole or in part, without restriction of any
   kind, provided that the above copyright notice and this paragraph are
   included on all such copies and derivative works.  However, this
   document itself may not be modified in any way, such as by removing
   the copyright notice or references to the Internet Society or other
   Internet organizations, except as needed for the purpose of
   developing Internet standards in which case the procedures for
   copyrights defined in the Internet Standards process must be
   followed, or as required to translate it into languages other than
   English.

   The limited permissions granted above are perpetual and will not be
   revoked by the Internet Society or its successors or assigns.

   This document and the information contained herein is provided on an
   "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
   TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
   BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
   HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
   MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.


























Chu & Kashyap                                                  [Page 13]


--Bed_of_Mussels_472_000--

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 14 15:55:12 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA18651
	for <ipoverib-archive@odin.ietf.org>; Mon, 14 Jan 2002 15:55:12 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id PAA24324
	for ipoverib-archive@odin.ietf.org; Mon, 14 Jan 2002 15:55:15 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA24141;
	Mon, 14 Jan 2002 15:51:48 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA24095
	for <ipoverib@optimus.ietf.org>; Mon, 14 Jan 2002 15:51:45 -0500 (EST)
Received: from e32.co.us.ibm.com (e32.co.us.ibm.com [32.97.110.130])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA18524;
	Mon, 14 Jan 2002 15:51:41 -0500 (EST)
Received: from westrelay01.boulder.ibm.com (westrelay01.boulder.ibm.com [9.99.140.22])
	by e32.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id PAA123618;
	Mon, 14 Jan 2002 15:48:41 -0500
Received: from d03nm036.boulder.ibm.com (avpilot.boulder.ibm.com [9.17.188.135])
	by westrelay01.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0EKpek69896;
	Mon, 14 Jan 2002 13:51:40 -0700
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Cc: ipoverib@ietf.org, ipoverib-admin@ietf.org
X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000
Message-ID: <OF94027AB1.85EBE9D1-ON88256B41.0028FA1D@boulder.ibm.com>
From: "Vivek Kashyap" <vivk@us.ibm.com>
Date: Mon, 14 Jan 2002 12:50:06 -0800
X-MIMETrack: Serialize by Router on D03NM036/03/M/IBM(Release 5.0.9 |November 16, 2001) at
 01/14/2002 01:51:39 PM
MIME-Version: 1.0
Content-type: text/plain; charset=us-ascii
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org


Folks,
      It is close to the last call. In this regard I would like to state
that we have been discussing this issue for a long time now. The ARP
outline, as discussed in the thread .... has been presented to the WG since
July last. It was restated in december on the mailing list and at the WG
meeting. I feel that it provides the best compromise we are likely to get
and we should not step back in time and restart the whole process. I really
do have a complaint - it has been a few months that we

      The Identifier:

      LID:
            People said they want the 'link-address' to be 48 bits. ( LID +
QPN + cap_flags is 48 bits).

            People said they don't want to go to the SM if possible (LID
allows for that)

            People said they want the process to end in small number of
messages (LID allows a query/response to settle the issue. As noted one
doesn't necessarily have to go to the SM to get the SL).

            LID is always needed.

      GID:
            People want to cross subnet boundaries. Mind you, we are
stretching here since such implementations are not possible yet.

      GUID:
            People want to keep a permanent address. (GUID can be made part
of GID and also got from the SM via the LID for those interested).


      Given the above reasons LID+GID is a good compromise. It allows for
all the conditions and doesn't shortchange anyone. Let us not rehash the
whole process again.

      The ARP process:

      We want it to be like RFC826. A broadcast is sent out and a reply
received.

            * People want it to be a single step process.
            * People want to avoid the SM
            * Peopel want to consult the SM

            All the above conditions are easily achieved as has been
discussed for some time now.

            Receive the LID+cap_flags+QPN + GID of the peer.

            If you like one step/avoiding the SM things are set. If you
don't you are free to consult the SM using the LID or the GID.  You always
get the IP QPN - that is not kept with the SM. The ones that want to go
back to the peer go into the advanced capabilities mode.

      If the remote sets the QPN flag then it supports SIDR. The SIDR and
you can resolve the packet that way. If the QPN flag is not set then we are
at the basic set and we c


   --
   Vivek Kashyap
   Linux Technology Center, IBM
   vivk@us.ibm.com
   kashyapv@us.ibm.com
   Ph: 503 578 3422 T/L: 775 3422



                                                                                                                                  
                      "H.K. Jerry Chu"                                                                                            
                      <Jerry.Chu@eng.su        To:       ipoverib@ietf.org                                                        
                      n.com>                   cc:                                                                                
                      Sent by:                 Subject:  [Ipoverib] What's inside an IPoIB link layer address?                    
                      ipoverib-admin@ie                                                                                           
                      tf.org                                                                                                      
                                                                                                                                  
                                                                                                                                  
                      01/12/2002 09:28                                                                                            
                      PM                                                                                                          
                      Please respond to                                                                                           
                      "H.K. Jerry Chu"                                                                                            
                                                                                                                                  
                                                                                                                                  



While proposals beyond the basic IPoIB over UD started showing up,
I still don't think we've reached a rough consensus on some basic
questions such as what the IPoIB link layer address should contain.
Again I'd like to apologize for not being there at the SLC meeting
last month. Perhaps a consensus was reached there but i've missed
the boat. In that case please let me know what it is and I'll shut
up.

What I'd like to see included in the IPoIB link layer address is

GID
QPN

and I'd like to allow QPN field to be 0.

What I don't like to see included is LID.

Discussions:

There has been a consensus of using GID, not LID as the basic
IB address identifying the port hosting an IP address. GID is
sufficiently stable, not subject to change across a node reboot
or subnet merge (IB level) like LID is. Also people don't want
to exclude the possibility of IPoIB link spanning more than
one IB subnet. GID supports scope beyond local link so it's
suitable for that purpose.

Multiple GIDs with different scopes may have been assigned to a CA.
It is required that a CA returns the GID with a scope no less than
the one for the IPoIB link. E.g. if an IPoIB link covers multiple
IB subnets, but a CA uses a link-local GID as part of its IPoIB
link layer address, it's not going to work well.

I don't see any value of including LID besides GID as proposed
by Vivek. I believes a lookup of PathRecord is always needed
in order for a node to retrieve complete path info before sending
unicast pkts to another node. The destination LID is part of
the PathRecord so including it in the link layer address returned
from ARP/ND doesn't save work. But you have all the drawbacks that
swayed people away from using LID - it adds volatility to the link
address. LID changing due to subnet merge or other reasons requires
potentially new mechanism being added to IP/driver implementation
to trigger ARP/ND flush. Lastly, what should be the LID if the IP
node is in a different IB subnet? Obviously it can't be the LID of
an intermediate IB router.

For QPN I'd like to suggest one thing. Since SIDR is a required
feature by IBA, the QPN field is allowed to be zero. What that
means is that the requester must use SIDR to retrieve the remote
QPN, given the GID of a remote IP address.

This feature allows the support of static/persistent "MAC addr"
wrt node reboot by removing the volatile part of the link layer
address - the QPN. In many implementations the QPN allocated for
a CA may change across node reboot. Support of static "MAC addr"
has been considered desirable by a number of people.

I don't like to add additional requirement such as SIDR to IPoIB,
and was trying to make the above feature "optional". Ideally nodes
can express whether they'd like QPN to be returned by embedding
some flags, such as "I don't talk SIDR so please return QPN" in
their IPoIB link layer addresses. But the broadcast nature of
IPv4/ARP forbids the use of different "MAC addrs" for the same IP
address, so my attempt failed.

Comment?

Jerry


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib





_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 14 16:18:16 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA19649
	for <ipoverib-archive@odin.ietf.org>; Mon, 14 Jan 2002 16:18:16 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id QAA25666
	for ipoverib-archive@odin.ietf.org; Mon, 14 Jan 2002 16:18:19 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA25008;
	Mon, 14 Jan 2002 16:04:09 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA24970
	for <ipoverib@optimus.ietf.org>; Mon, 14 Jan 2002 16:04:07 -0500 (EST)
Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA19043
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 16:04:03 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e31.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id QAA249930
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 16:01:04 -0500
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0EL42715540
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 14:04:02 -0700
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id g0EL40g18888;
	Mon, 14 Jan 2002 13:04:00 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0EL3xW02674;
	Mon, 14 Jan 2002 13:03:59 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201142103.g0EL3xW02674@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
To: vivk@us.ibm.com (Vivek Kashyap)
Date: Mon, 14 Jan 2002 13:03:59 -0800 (PST)
Cc: ipoverib@ietf.org
In-Reply-To: <OF94027AB1.85EBE9D1-ON88256B41.0028FA1D@boulder.ibm.com> from "Vivek Kashyap" at Jan 14, 2002 11:50:06 AM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Sorry, for the incomplete mail.

My mailer died I thought and the message went out while I was on the
phone talking to Arvind Murching before I could complete, review and
send it out. Will send the mail out in a while. I was trying to update
the points based on things Arvind and I were discussing. Remind me not
to do two things at once.

Vivek


> 
> 
> Folks,
>       It is close to the last call. In this regard I would like to state
> that we have been discussing this issue for a long time now. The ARP
> outline, as discussed in the thread .... has been presented to the WG since
> July last. It was restated in december on the mailing list and at the WG
> meeting. I feel that it provides the best compromise we are likely to get
> and we should not step back in time and restart the whole process. I really
> do have a complaint - it has been a few months that we
> 
>       The Identifier:
> 
>       LID:
>             People said they want the 'link-address' to be 48 bits. ( LID +
> QPN + cap_flags is 48 bits).
> 
>             People said they don't want to go to the SM if possible (LID
> allows for that)
> 
>             People said they want the process to end in small number of
> messages (LID allows a query/response to settle the issue. As noted one
> doesn't necessarily have to go to the SM to get the SL).
> 
>             LID is always needed.
> 
>       GID:
>             People want to cross subnet boundaries. Mind you, we are
> stretching here since such implementations are not possible yet.
> 
>       GUID:
>             People want to keep a permanent address. (GUID can be made part
> of GID and also got from the SM via the LID for those interested).
> 
> 
>       Given the above reasons LID+GID is a good compromise. It allows for
> all the conditions and doesn't shortchange anyone. Let us not rehash the
> whole process again.
> 
>       The ARP process:
> 
>       We want it to be like RFC826. A broadcast is sent out and a reply
> received.
> 
>             * People want it to be a single step process.
>             * People want to avoid the SM
>             * Peopel want to consult the SM
> 
>             All the above conditions are easily achieved as has been
> discussed for some time now.
> 
>             Receive the LID+cap_flags+QPN + GID of the peer.
> 
>             If you like one step/avoiding the SM things are set. If you
> don't you are free to consult the SM using the LID or the GID.  You always
> get the IP QPN - that is not kept with the SM. The ones that want to go
> back to the peer go into the advanced capabilities mode.
> 
>       If the remote sets the QPN flag then it supports SIDR. The SIDR and
> you can resolve the packet that way. If the QPN flag is not set then we are
> at the basic set and we c
> 
> 
>    --
>    Vivek Kashyap
>    Linux Technology Center, IBM
>    vivk@us.ibm.com
>    kashyapv@us.ibm.com
>    Ph: 503 578 3422 T/L: 775 3422
> 
> 
> 
>                                                                                                                                   
>                       "H.K. Jerry Chu"                                                                                            
>                       <Jerry.Chu@eng.su        To:       ipoverib@ietf.org                                                        
>                       n.com>                   cc:                                                                                
>                       Sent by:                 Subject:  [Ipoverib] What's inside an IPoIB link layer address?                    
>                       ipoverib-admin@ie                                                                                           
>                       tf.org                                                                                                      
>                                                                                                                                   
>                                                                                                                                   
>                       01/12/2002 09:28                                                                                            
>                       PM                                                                                                          
>                       Please respond to                                                                                           
>                       "H.K. Jerry Chu"                                                                                            
>                                                                                                                                   
>                                                                                                                                   
> 
> 
> 
> While proposals beyond the basic IPoIB over UD started showing up,
> I still don't think we've reached a rough consensus on some basic
> questions such as what the IPoIB link layer address should contain.
> Again I'd like to apologize for not being there at the SLC meeting
> last month. Perhaps a consensus was reached there but i've missed
> the boat. In that case please let me know what it is and I'll shut
> up.
> 
> What I'd like to see included in the IPoIB link layer address is
> 
> GID
> QPN
> 
> and I'd like to allow QPN field to be 0.
> 
> What I don't like to see included is LID.
> 
> Discussions:
> 
> There has been a consensus of using GID, not LID as the basic
> IB address identifying the port hosting an IP address. GID is
> sufficiently stable, not subject to change across a node reboot
> or subnet merge (IB level) like LID is. Also people don't want
> to exclude the possibility of IPoIB link spanning more than
> one IB subnet. GID supports scope beyond local link so it's
> suitable for that purpose.
> 
> Multiple GIDs with different scopes may have been assigned to a CA.
> It is required that a CA returns the GID with a scope no less than
> the one for the IPoIB link. E.g. if an IPoIB link covers multiple
> IB subnets, but a CA uses a link-local GID as part of its IPoIB
> link layer address, it's not going to work well.
> 
> I don't see any value of including LID besides GID as proposed
> by Vivek. I believes a lookup of PathRecord is always needed
> in order for a node to retrieve complete path info before sending
> unicast pkts to another node. The destination LID is part of
> the PathRecord so including it in the link layer address returned
> from ARP/ND doesn't save work. But you have all the drawbacks that
> swayed people away from using LID - it adds volatility to the link
> address. LID changing due to subnet merge or other reasons requires
> potentially new mechanism being added to IP/driver implementation
> to trigger ARP/ND flush. Lastly, what should be the LID if the IP
> node is in a different IB subnet? Obviously it can't be the LID of
> an intermediate IB router.
> 
> For QPN I'd like to suggest one thing. Since SIDR is a required
> feature by IBA, the QPN field is allowed to be zero. What that
> means is that the requester must use SIDR to retrieve the remote
> QPN, given the GID of a remote IP address.
> 
> This feature allows the support of static/persistent "MAC addr"
> wrt node reboot by removing the volatile part of the link layer
> address - the QPN. In many implementations the QPN allocated for
> a CA may change across node reboot. Support of static "MAC addr"
> has been considered desirable by a number of people.
> 
> I don't like to add additional requirement such as SIDR to IPoIB,
> and was trying to make the above feature "optional". Ideally nodes
> can express whether they'd like QPN to be returned by embedding
> some flags, such as "I don't talk SIDR so please return QPN" in
> their IPoIB link layer addresses. But the broadcast nature of
> IPv4/ARP forbids the use of different "MAC addrs" for the same IP
> address, so my attempt failed.
> 
> Comment?
> 
> Jerry
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 
> 
> 
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 14 17:05:21 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA21568
	for <ipoverib-archive@odin.ietf.org>; Mon, 14 Jan 2002 17:05:21 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id RAA28025
	for ipoverib-archive@odin.ietf.org; Mon, 14 Jan 2002 17:05:23 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA27985;
	Mon, 14 Jan 2002 17:03:07 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA27955
	for <ipoverib@optimus.ietf.org>; Mon, 14 Jan 2002 17:03:05 -0500 (EST)
Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA21494
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 17:03:02 -0500 (EST)
Received: from sunmail1.Sun.COM ([129.145.1.2])
	by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id OAA08770;
	Mon, 14 Jan 2002 14:03:04 -0800 (PST)
Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.89.50])
	by sunmail1.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1-Sun.COM.mod.2) with ESMTP id OAA05450;
	Mon, 14 Jan 2002 14:04:15 -0800 (PST)
Received: from taipei (taipei.Eng.Sun.COM [129.146.85.178])
	by jurassic.eng.sun.com (8.12.2+Sun/8.12.2) with SMTP id g0EM33qT126365;
	Mon, 14 Jan 2002 14:03:03 -0800 (PST)
Message-Id: <200201142203.g0EM33qT126365@jurassic.eng.sun.com>
Date: Mon, 14 Jan 2002 14:00:35 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
To: kamath@omegaband.com
Cc: ipoverib@ietf.org
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: ia+2SjSw4Xaj3yIH3fVqkw==
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.6_06 SunOS 5.8 sun4u sparc 
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

My comments are embeded in the following.

>I agree with using GID both for longevity and scope. What about using
>SIDR(Private Data) for communicating all the capability bits and any other
>extras? If it is not too late for this, I have some new feedback.
>
>For ARP on an IB subet one would require that
>a. IB Switches support multicast. (This is true in most cases today.)
>b. IP addresses used in an infiniband subnet are unique for use by IPoIB.
>(This is an important constraint which gets rid of the ethertype field)
>c. Some issues related to Multicast over IB need to be resolved (like
>multiple consumers using multicast resources.)

This one is really simple. A MGID contains 128 bits. Even IPv6 recognizes
that it only needs the lower 32bits for the forseeable future. So the
probability of MGID collisions is negligible. If one really wants
something bullet-proof, we can ask IBTA to reserve a chunk (or endorse
the IPoIB signature covered in the link and multicast draft) for IPoIB.

>
>Requirement (b) was addressed by defining a multicast group and the IP
>address format. One would still have to deal with the GID, LID and the IP
>address. Including the LID in the ARP Response will be difficult to deal
>with because
> a. LID Assignments change.
> b. ARP might have to traverse a router, for IP Subnets spanning IB Subnets
>(a) might not really be an issue because ARP entries are aged. In ethernet,
>the ethernet topology might change and MAC Addresses can change on server
>reboots (by way of using a different MAC or by assigning a MAC Address.) The
>current ARP implementations do take these scenarios into account. Moreover
>Path record lookups alone would not help resolve the issue of changing LIDs.
>The locally cached path record will not be valid if the LID changes. For a
>true solution, there has to be some way of using IB Traps to get information
>about topology changes as and when it happens. An end node can subscribe for

This is a generic, not IPoIB specific issue. So the work belongs to IBTA
to ensure some mechanism exists for flushing stale path records.

>a trap, but the end node will have to do something like,
> * Tell me when all the switches on the path to this node report a state
>change on such and such a port.
> This will not work very well right now because one will have to subscribe
>for traps on all switches leading to the end node, which implies that an end
>node interested in the existence of another end node will need to have
>knowledge of the complete IB topology. This is probably something worth
>considering in the Infiniband WGs.

Ok, we're saying the same thing.

>
>On the same token, if you use the globally unique GID, information about the
>GID will not be maintained by the local subnet's subnet administration and
>hence a path record lookup based on the GID would fail.

In this case you follow the IBA rule. The rule says that you don't even
attempt to lookup a pathrecord for a remote GID in a different IB subnet.
You need to find an IB route to it first. This is similar to IP.

> So if the GID lookup
>fails, one would send the ipoib packet to the designated IB router (as is
>done with ethernet today, albiet based on netmasks). But what if you have an
>IP subnet spanning IB Subnets. The designated router exists for an IP Subnet
>and the local subnet IB administration will not have information about all
>the IP nodes, because they could exist on another IB Subnet.

Hmmm, you seem to confuse routing at the IB level with that at the IP
level. (Or am I the one who is confused?)

>
> The following IB transactions are required before sending any IPoIB data
>over the subnet (assuming the multicast join is already done and we use GIDs
>as link layer addresses)
> 1. IpoIB ARP.
> 2. Path record lookup.
> 3. SIDR
>
>In all this, the existence of the ARP is questionable. The same
>functionality can be achieved using 2 and 3 above.
>If you use GIDs as the IP addresses ,you can use a Path record lookup and
>then use SIDR to get all the information to use IPoIB. The advantages are
>evident, for eg,

I think similar idea has been brought up before. One major obstacle here
is that a given interface may host many (e.g. a few thousand) IP addresses
simultaneously (e.g. virtual hosting). You just don't have space for that
many GIDs on a HCA.

Jerry

> a. Traversing a router. Assuming that the path record lookup fails, one
>will redirect all the related ipoib packets to the Gateway/router. Since ARP
>does not exist, one does not have to worry about the issues involved by ARP
>traversing a router, or the router proxying for end nodes on the same IP
>Subnet but different IB Subnets. IP and IB Subnets will be one and the same.
> b. Debate on using the ether-type field vanishes. The Service ID will be
>unique to IPoIB and will be assigned by some WG. You could potentially use
>two Service IDs (might be an overkill), one for IPv6 packets and another for
>IPv4.
> c. No special format for IP addresses is required (I have not thought this
>through very well.)
> d. Switches are not required to support Multicast.
> e. There are no issues with multiple consumers using the same multicast
>resources.
> f. Multicast could be used by an IB Gateways only to avoid multi-unicast.
>
>In this scenario the IP subnet is the same as an IB Subnet, since the same
>network addresses are used.
>
>-Ashwin
>
>
>
>-----Original Message-----
>From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
>Of H.K. Jerry Chu
>Sent: None
>To: ipoverib@ietf.org
>Subject: [Ipoverib] What's inside an IPoIB link layer address?
>
>
>While proposals beyond the basic IPoIB over UD started showing up,
>I still don't think we've reached a rough consensus on some basic
>questions such as what the IPoIB link layer address should contain.
>Again I'd like to apologize for not being there at the SLC meeting
>last month. Perhaps a consensus was reached there but i've missed
>the boat. In that case please let me know what it is and I'll shut
>up.
>
>What I'd like to see included in the IPoIB link layer address is
>
>GID
>QPN
>
>and I'd like to allow QPN field to be 0.
>
>What I don't like to see included is LID.
>
>Discussions:
>
>There has been a consensus of using GID, not LID as the basic
>IB address identifying the port hosting an IP address. GID is
>sufficiently stable, not subject to change across a node reboot
>or subnet merge (IB level) like LID is. Also people don't want
>to exclude the possibility of IPoIB link spanning more than
>one IB subnet. GID supports scope beyond local link so it's
>suitable for that purpose.
>
>Multiple GIDs with different scopes may have been assigned to a CA.
>It is required that a CA returns the GID with a scope no less than
>the one for the IPoIB link. E.g. if an IPoIB link covers multiple
>IB subnets, but a CA uses a link-local GID as part of its IPoIB
>link layer address, it's not going to work well.
>
>I don't see any value of including LID besides GID as proposed
>by Vivek. I believes a lookup of PathRecord is always needed
>in order for a node to retrieve complete path info before sending
>unicast pkts to another node. The destination LID is part of
>the PathRecord so including it in the link layer address returned
>from ARP/ND doesn't save work. But you have all the drawbacks that
>swayed people away from using LID - it adds volatility to the link
>address. LID changing due to subnet merge or other reasons requires
>potentially new mechanism being added to IP/driver implementation
>to trigger ARP/ND flush. Lastly, what should be the LID if the IP
>node is in a different IB subnet? Obviously it can't be the LID of
>an intermediate IB router.
>
>For QPN I'd like to suggest one thing. Since SIDR is a required
>feature by IBA, the QPN field is allowed to be zero. What that
>means is that the requester must use SIDR to retrieve the remote
>QPN, given the GID of a remote IP address.
>
>This feature allows the support of static/persistent "MAC addr"
>wrt node reboot by removing the volatile part of the link layer
>address - the QPN. In many implementations the QPN allocated for
>a CA may change across node reboot. Support of static "MAC addr"
>has been considered desirable by a number of people.
>
>I don't like to add additional requirement such as SIDR to IPoIB,
>and was trying to make the above feature "optional". Ideally nodes
>can express whether they'd like QPN to be returned by embedding
>some flags, such as "I don't talk SIDR so please return QPN" in
>their IPoIB link layer addresses. But the broadcast nature of
>IPv4/ARP forbids the use of different "MAC addrs" for the same IP
>address, so my attempt failed.
>
>Comment?
>
>Jerry
>
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>http://www.ietf.org/mailman/listinfo/ipoverib
>
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 14 17:19:19 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA22099
	for <ipoverib-archive@odin.ietf.org>; Mon, 14 Jan 2002 17:19:19 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id RAA28308
	for ipoverib-archive@odin.ietf.org; Mon, 14 Jan 2002 17:19:21 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA28280;
	Mon, 14 Jan 2002 17:18:19 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA28235
	for <ipoverib@optimus.ietf.org>; Mon, 14 Jan 2002 17:18:16 -0500 (EST)
Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA22056
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 17:18:10 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e31.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id RAA48254;
	Mon, 14 Jan 2002 17:15:02 -0500
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0EMHo7105738;
	Mon, 14 Jan 2002 15:17:51 -0700
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id g0EMHng20787;
	Mon, 14 Jan 2002 14:17:49 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0EMHnD07082;
	Mon, 14 Jan 2002 14:17:49 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201142217.g0EMHnD07082@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] IPoIB advanced capabilities
To: yaronh@voltaire.com
Date: Mon, 14 Jan 2002 14:17:47 -0800 (PST)
Cc: arvindm@windows.microsoft.com ('Arvind Murching'),
        kashyapv@us.ibm.com ('Vivek Kashyap'), ipoverib@ietf.org,
        jpink@microsoft.com ('Jim Pinkerton')
In-Reply-To: <00fd01c19b84$396986d0$0100007f@vaio505he> from "Yaron Haviv" at Jan 12, 2002 05:14:32 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

yaron,	
        I have posted the draft and it got announced today-
        draft-kashyap-ipoib-advanced-capabilities-00.txt 

        The RC flag certainly doesn't imply SDP. I leave it to the WG
        to decide and I can accordingly update the flag.

        In general I agree with you and prefer avoiding the SM and
        multiple-resolution messages between the nodes. Thus an
        address resolution packet that gives us the necessary
        information is better.

        Include the following in the packet:

	LID - you don't have to go the SM.
	QPN - you don't have to back and forth between the two endpoints
		Secondly if SIDR is not supported the solution breaks down.
	GID - allows proxy-ARP of the type RogerR discussed for bridging
	      allows future expansion across IB subnets
        cap-flags- provides for the kind of issues you mention and
                   also allows interoperability among all
                   implementations 
                   In general I would like to restric the capabilitY
                   flags to RC and QPN. If SDP is agreed to by WG then
                   that could the third flag. I am not sure if there
                   are many takers for UC/RD.

Vivek 

> 
> This is a multi-part message in MIME format.
> 
> --Boundary_(ID_MH0w1LW5oe3PQ6xdgF91BA)
> Content-type: text/plain; charset=us-ascii
> Content-Transfer-Encoding: 7BIT
> 
> Arvind,
>  
> In our case the capability (RC/UD)  is part of the link layer attributes
> The host could always use UD if they want to, however if both of them
> support RC and for some reason (MTU ..) want to work RC they can
> As far as I know Vivek has a proposal for how to set an RC connection, I'm
> not sure if he posted it yet
>  
> Another capability bit I would like to add that would help is and "SDP"
> capability bit
> Since with the new proposal there is no need to go to the SM for ARP, in
> order to know if a node supports SDP I must add an SM lookup to the process
> That could slow the process significantly
> If we add this bit we could avoid that and quite easily know if we could use
> SDP for the connection
> Since protocol offloading is an integral (non-optional) part of InfiniBand
> it also makes sense to put it as part of the IB ARP
> (not to mention that it is specified in SDP that it uses the IP over IB
> address resolution)
>  
> Yaron
>  
> -----Original Message-----
> From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
> Arvind Murching
> Sent: Saturday, January 12, 2002 3:07 AM
> To: Vivek Kashyap; ipoverib@ietf.org
> Cc: Jerry.Chu@eng.sun.com
> Subject: RE: [Ipoverib] IPoIB advanced capabilities
>  
> Vivek, two issues regarding the capabilities flags:
> (a) From a purist point of view, the link layer address is not the right
> place to specify end-node capabilities.
> (b) Section 3.2 says:
>           The method by which a host decides it wants to use
>           the IPoIB-RC/UC/RD services is beyond the scope
>         of this specification.
> However, this means that two nodes that conform to this specification, and
> decide to use the capability bits, can easily fail to interoperate. This is
> because this document does not specify how they, for example, set up RC/UC
> connections (what service ID is used etc). We should either fully specify
> the encapsulation and connection mechanisms for these other transport
> methods (if time permits), or simply label the bits used for this purpose as
> "Reserved" in this spec, and require all conformant implementations to set
> this to 0 on transmit and ignore on receive. Later versions of this spec can
> define these bits, along with details of how they are used.
> - Arvind
> -----Original Message-----
> From: Vivek Kashyap [ mailto:kashyapv@us.ibm.com]
> Sent: Friday, January 11, 2002 4:34 PM
> To: ipoverib@ietf.org
> Cc: kashyapv@us.ibm.com
> Subject: [Ipoverib] IPoIB advanced capabilities
> Folks,
>         We have discussed IP over non-UD links a few times on this
>         mailing list.The capability-flags in the address-resolution
>         packets provide the necessary hook to implementing such
>         solutions in an interoperable way. I presented a solution at
>         the IPoIB WG meeting at SLC earlier this month.
>         Attached is a draft that I submitted a few days back. It
>         hasn't been announced yet but I thought that we need to move
>         forward. The draft and the solution is completely disjoint
>         from the encapsulation draft except for the capability-bits
>         being included in the address resolution packet.
>         I look forward to your comments.
> Vivek
> <snip>
> 
> --Boundary_(ID_MH0w1LW5oe3PQ6xdgF91BA)
> Content-type: text/html; charset=us-ascii
> Content-Transfer-Encoding: 7BIT
> 
> <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
> <html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns="http://www.w3.org/TR/REC-html40">
> 
> <head>
> <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
> 
> 
> <meta name=ProgId content=Word.Document>
> <meta name=Generator content="Microsoft Word 9">
> <meta name=Originator content="Microsoft Word 9">
> <link rel=File-List href="cid:filelist.xml@01C19B94.FB947480">
> <title>RE: [Ipoverib] IPoIB advanced capabilities</title>
> <!--[if gte mso 9]><xml>
>  <o:OfficeDocumentSettings>
>   <o:DoNotRelyOnCSS/>
>  </o:OfficeDocumentSettings>
> </xml><![endif]--><!--[if gte mso 9]><xml>
>  <w:WordDocument>
>   <w:Zoom>0</w:Zoom>
>   <w:DocumentKind>DocumentEmail</w:DocumentKind>
>   <w:EnvelopeVis/>
>  </w:WordDocument>
> </xml><![endif]-->
> <style>
> <!--
>  /* Font Definitions */
> @font-face
> 	{font-family:Tahoma;
> 	panose-1:2 11 6 4 3 5 4 4 2 4;
> 	mso-font-charset:0;
> 	mso-generic-font-family:swiss;
> 	mso-font-pitch:variable;
> 	mso-font-signature:553679495 -2147483648 8 0 66047 0;}
>  /* Style Definitions */
> p.MsoNormal, li.MsoNormal, div.MsoNormal
> 	{mso-style-parent:"";
> 	margin:0in;
> 	margin-bottom:.0001pt;
> 	mso-pagination:widow-orphan;
> 	font-size:12.0pt;
> 	font-family:"Times New Roman";
> 	mso-fareast-font-family:"Times New Roman";}
> a:link, span.MsoHyperlink
> 	{color:blue;
> 	text-decoration:underline;
> 	text-underline:single;}
> a:visited, span.MsoHyperlinkFollowed
> 	{color:blue;
> 	text-decoration:underline;
> 	text-underline:single;}
> p.MsoAutoSig, li.MsoAutoSig, div.MsoAutoSig
> 	{margin:0in;
> 	margin-bottom:.0001pt;
> 	mso-pagination:widow-orphan;
> 	font-size:12.0pt;
> 	font-family:"Times New Roman";
> 	mso-fareast-font-family:"Times New Roman";}
> p
> 	{margin-right:0in;
> 	mso-margin-top-alt:auto;
> 	mso-margin-bottom-alt:auto;
> 	margin-left:0in;
> 	mso-pagination:widow-orphan;
> 	font-size:12.0pt;
> 	font-family:"Times New Roman";
> 	mso-fareast-font-family:"Times New Roman";}
> span.EmailStyle18
> 	{mso-style-type:personal-reply;
> 	mso-ansi-font-size:10.0pt;
> 	mso-ascii-font-family:Arial;
> 	mso-hansi-font-family:Arial;
> 	mso-bidi-font-family:Arial;
> 	color:navy;}
> @page Section1
> 	{size:8.5in 11.0in;
> 	margin:1.0in 1.25in 1.0in 1.25in;
> 	mso-header-margin:.5in;
> 	mso-footer-margin:.5in;
> 	mso-paper-source:0;
> 	mso-gutter-direction:rtl;}
> div.Section1
> 	{page:Section1;}
> -->
> </style>
> </head>
> 
> <body lang=EN-US link=blue vlink=blue style='tab-interval:.5in'>
> <!-- Converted from text/rtf format -->
> 
> <div class=Section1 dir=RTL>
> 
> <p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
> face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
> Arial'>Arvind,<o:p></o:p></span></font></span></p>
> 
> <p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
> face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
> Arial'><![if !supportEmptyParas]>&nbsp;<![endif]><o:p></o:p></span></font></span></p>
> 
> <p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
> face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
> Arial'>In our case the capability (RC/UD) <span style="mso-spacerun:
> yes">&nbsp;</span>is part of the link layer attributes <o:p></o:p></span></font></span></p>
> 
> <p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
> face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
> Arial'>The host could always use UD if they want to, however if both of them
> support RC and for some reason (MTU ..) want to work RC they can<o:p></o:p></span></font></span></p>
> 
> <p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
> face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
> Arial'>As far as I know Vivek has a proposal for how to set an RC connection, I&#8217;m
> not sure if he posted it yet<o:p></o:p></span></font></span></p>
> 
> <p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
> face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
> Arial'><![if !supportEmptyParas]>&nbsp;<![endif]><o:p></o:p></span></font></span></p>
> 
> <p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
> face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
> Arial'>Another capability bit I would like to add that would help is and &#8220;SDP&#8221;
> capability bit<o:p></o:p></span></font></span></p>
> 
> <p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
> face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
> Arial'>Since with the new proposal there is no need to go to the SM for ARP, in
> order to know if a node supports SDP I must add an SM lookup to the process<o:p></o:p></span></font></span></p>
> 
> <p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
> face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
> Arial'>That could slow the process significantly<o:p></o:p></span></font></span></p>
> 
> <p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
> face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
> Arial'>If we add this bit we could avoid that and quite easily know if we could
> use SDP for the connection<o:p></o:p></span></font></span></p>
> 
> <p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
> face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
> Arial'>Since protocol offloading is an integral (non-optional) part of InfiniBand
> it also makes sense to put it as part of the IB ARP<o:p></o:p></span></font></span></p>
> 
> <p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
> face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
> Arial'>(not to mention that it is specified in SDP that it uses the IP over IB
> address resolution) <o:p></o:p></span></font></span></p>
> 
> <p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
> face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
> Arial'><![if !supportEmptyParas]>&nbsp;<![endif]><o:p></o:p></span></font></span></p>
> 
> <p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
> face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
> Arial'>Yaron<o:p></o:p></span></font></span></p>
> 
> <p class=MsoNormal dir=LTR><span class=EmailStyle18><font size=2 color=navy
> face=Arial><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt;font-family:
> Arial'><![if !supportEmptyParas]>&nbsp;<![endif]><o:p></o:p></span></font></span></p>
> 
> <p class=MsoNormal dir=LTR style='margin-left:.5in'><font size=2 color=black
> face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma;color:black'>-----Original
> Message-----<br>
> <b><span style='font-weight:bold'>From:</span></b> ipoverib-admin@ietf.org
> [mailto:ipoverib-admin@ietf.org]<b><span style='font-weight:bold'>On Behalf Of </span></b>Arvind
> Murching<br>
> <b><span style='font-weight:bold'>Sent:</span></b> Saturday, January 12, 2002
> 3:07 AM<br>
> <b><span style='font-weight:bold'>To:</span></b> Vivek Kashyap;
> ipoverib@ietf.org<br>
> <b><span style='font-weight:bold'>Cc:</span></b> Jerry.Chu@eng.sun.com<br>
> <b><span style='font-weight:bold'>Subject:</span></b> RE: [Ipoverib] IPoIB advanced
> capabilities</span></font></p>
> 
> <p class=MsoNormal dir=LTR style='margin-left:.5in'><font size=3
> face="Times New Roman"><span style='font-size:12.0pt'><![if !supportEmptyParas]>&nbsp;<![endif]><o:p></o:p></span></font></p>
> 
> <p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>Vivek,</span></font><font
> color=black><span style='color:black'> </span></font><font size=2 color=black
> face="Courier New"><span style='font-size:10.0pt;font-family:"Courier New";
> color:black'>two issues regarding the capabilities flags:</span></font><font
> color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>
> 
> <p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>(a) From a
> purist point of view, the link layer address is not the right place to specify
> end-node capabilities.</span></font><font color=black><span style='color:black;
> mso-color-alt:windowtext'><o:p></o:p></span></font></p>
> 
> <p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>(b)</span></font><font
> color=black><span style='color:black'> </span></font><font size=2 color=black
> face="Courier New"><span style='font-size:10.0pt;font-family:"Courier New";
> color:black'>Section 3.2 says:</span></font><font color=black><span
> style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>
> 
> <p dir=LTR style='margin-left:.5in'><font size=3 color=black
> face="Times New Roman"><span style='font-size:12.0pt;color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> </span></font><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;</span></font><font
> color=black><span style='color:black'> </span></font><font size=2 color=black
> face="Courier New"><span style='font-size:10.0pt;font-family:"Courier New";
> color:black'>The method by which a host decides it</span></font><font
> color=black><span style='color:black'> </span></font><font size=2 color=black
> face="Courier New"><span style='font-size:10.0pt;font-family:"Courier New";
> color:black'>wants to use</span></font><font color=black><span
> style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>
> 
> <p dir=LTR style='margin-left:.5in'><font size=3 color=black
> face="Times New Roman"><span style='font-size:12.0pt;color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> </span></font><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;</span></font><font
> color=black><span style='color:black'> </span></font><font size=2 color=black
> face="Courier New"><span style='font-size:10.0pt;font-family:"Courier New";
> color:black'>the IPoIB-RC/UC/RD services is beyond the scope</span></font><font
> color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>
> 
> <p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> of this specification.</span></font><font color=black><span style='color:black;
> mso-color-alt:windowtext'><o:p></o:p></span></font></p>
> 
> <p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>However, this
> means that two</span></font><font color=black><span style='color:black'> </span></font><font
> size=2 color=black face="Courier New"><span style='font-size:10.0pt;font-family:
> "Courier New";color:black'>nodes that conform to this specification,</span></font><font
> color=black><span style='color:black'> </span></font><font size=2 color=black
> face="Courier New"><span style='font-size:10.0pt;font-family:"Courier New";
> color:black'>and</span></font><font color=black><span style='color:black'> </span></font><font
> size=2 color=black face="Courier New"><span style='font-size:10.0pt;font-family:
> "Courier New";color:black'>decide to use the capability bits,</span></font><font
> color=black><span style='color:black'> </span></font><font size=2 color=black
> face="Courier New"><span style='font-size:10.0pt;font-family:"Courier New";
> color:black'>can easily fail to interoperate. This is because this document
> does not specify how</span></font><font color=black><span style='color:black'> </span></font><font
> size=2 color=black face="Courier New"><span style='font-size:10.0pt;font-family:
> "Courier New";color:black'>they, for example, set up RC/UC connections (what
> service ID is used etc).</span></font><font color=black><span style='color:
> black'> </span></font><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>We should either
> fully specify the encapsulation and</span></font><font color=black><span
> style='color:black'> </span></font><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>connection
> mechanisms for these other transport methods (if time permits), or</span></font><font
> color=black><span style='color:black'> </span></font><font size=2 color=black
> face="Courier New"><span style='font-size:10.0pt;font-family:"Courier New";
> color:black'>simply label the bits used for this purpose as
> &quot;Reserved&quot; in this spec, and require all conformant implementations
> to set this to 0 on transmit and ignore on receive.</span></font><font
> color=black><span style='color:black'> </span></font><font size=2 color=black
> face="Courier New"><span style='font-size:10.0pt;font-family:"Courier New";
> color:black'>Later versions of this spec can define these bits, along with
> details of how they are used.</span></font><font color=black><span
> style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>
> 
> <p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>- Arvind</span></font><font
> color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>
> 
> <p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>-----Original
> Message-----<br>
> From: Vivek Kashyap [<a href="mailto:kashyapv@us.ibm.com">mailto:kashyapv@us.ibm.com</a>]<br>
> Sent: Friday, January 11, 2002 4:34 PM<br>
> To: ipoverib@ietf.org<br>
> Cc: kashyapv@us.ibm.com<br>
> Subject: [Ipoverib] IPoIB advanced capabilities</span></font><font color=black><span
> style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>
> 
> <p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>Folks,</span></font><font
> color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>
> 
> <p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> We have discussed IP over non-UD links a few times on this</span></font><font
> color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>
> 
> <p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> mailing list.The capability-flags in the address-resolution</span></font><font
> color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>
> 
> <p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> packets provide the necessary hook to implementing such</span></font><font
> color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>
> 
> <p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> solutions in an interoperable way. I presented a solution at</span></font><font
> color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>
> 
> <p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> the IPoIB WG meeting at SLC earlier this month.</span></font><font color=black><span
> style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>
> 
> <p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> Attached is a draft that I submitted a few days back. It</span></font><font
> color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>
> 
> <p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> hasn't been announced yet but I thought that we need to move</span></font><font
> color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>
> 
> <p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> forward. The draft and the solution is completely disjoint</span></font><font
> color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>
> 
> <p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> from the encapsulation draft except for the capability-bits</span></font><font
> color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>
> 
> <p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> being included in the address resolution packet.</span></font><font
> color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>
> 
> <p dir=LTR style='margin-left:.5in'><font size=3 color=black
> face="Times New Roman"><span style='font-size:12.0pt;color:black'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> </span></font><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>I look forward
> to your comments. </span></font><font color=black><span style='color:black;
> mso-color-alt:windowtext'><o:p></o:p></span></font></p>
> 
> <p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>Vivek</span></font><font
> color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>
> 
> <p dir=LTR style='margin-left:.5in'><font size=2 color=black face="Courier New"><span
> style='font-size:10.0pt;font-family:"Courier New";color:black'>&lt;snip&gt;</span></font><font
> color=black><span style='color:black;mso-color-alt:windowtext'><o:p></o:p></span></font></p>
> 
> </div>
> 
> </body>
> 
> </html>
> 
> --Boundary_(ID_MH0w1LW5oe3PQ6xdgF91BA)--
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)
-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 14 17:46:24 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA23223
	for <ipoverib-archive@odin.ietf.org>; Mon, 14 Jan 2002 17:46:24 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id RAA29239
	for ipoverib-archive@odin.ietf.org; Mon, 14 Jan 2002 17:46:26 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA29228;
	Mon, 14 Jan 2002 17:45:51 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA29190
	for <ipoverib@optimus.ietf.org>; Mon, 14 Jan 2002 17:45:49 -0500 (EST)
Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA23166
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 17:45:46 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id QAA77302;
	Mon, 14 Jan 2002 16:41:38 -0600
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0EMjlf163588;
	Mon, 14 Jan 2002 17:45:47 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0EMjjV13616;
	Mon, 14 Jan 2002 14:45:45 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0EMjiV08688;
	Mon, 14 Jan 2002 14:45:44 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201142245.g0EMjiV08688@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
To: Jerry.Chu@eng.sun.com
Date: Mon, 14 Jan 2002 14:45:43 -0800 (PST)
Cc: ipoverib@ietf.org
In-Reply-To: <200201130528.g0D5S9R8812864@jurassic.eng.sun.com> from "H.K. Jerry Chu" at Jan 12, 2002 08:28:29 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Folks,

       It is close to the last call. In this regard I would like to
       state that we have been discussing this issue for a long time
       now. The most recent discussion is what I presented at the SLC
       last december. (I certainly felt that the room was in agreement
       - no queries, objection etc.) 

       I believe the following link-description provides the best
       assimilation of all the choices that have been put forth. If we
       step back and restart the whole discussion again we are likely
       to come back to the following - so here is a brief compilation
       of issues.


	Suggested Link description to be used in address resolution:

		LID + capability flags + GID + QPN


 
       Why the LID?

             People said they don't want to go to the SM if possible
             (LID allows for that. As noted here and LWG the SL can be
             determined based on QoS or other parameters. The SL may
             also be determined from the SA/SM)

             People said they want the process to end in small number
             of messages (LID allows a query/response to settle the
             issue. As noted one doesn't necessarily have to go to the
             SM to get the SL).

             LID is always needed. (you can't escape it).

             BTW, people said they want the 'link-address' to be 48 bits.
             (LID + QPN + cap_flags is 48 bits).


       Why the GID ?
       
                People want IP to span across IB subnets. We are
                stretching stretching here since such implementations
                are not possible yet.
                However, we can define it such that it will work in
                the future. Therefore include the GID.

	GUID

		Can be made part of the GID.


	Why the QPN: 

                1) There is only one other way to get the QPN. That is
                the SIDR protocol. However, this has a drawback.

		SIDR is NOT REQUIRED by IBTA.

                I had raised this point last August 'Feasibility of
                SIDR': 

                ***** Section 12.11 essentially states that SIDR is
                optional. As the specification states:

               << GSAs shall support this protocol if non-management
               services are pro-vided on the Channel Adapter at other
               than fixed QPNs. If this protocol is *not supported*,
               the status "The method/attribute combination is not
               sup-ported" shall be returned, as described in section
               Table 101 MAD Common Status Field Bit Values >>

		*****

                That still holds. SIDR is NOT required by IBTA. A
                device can always choose to provide all non-management
                services at fixed QPNs. It may just choose to provide
                IP and nothing else at a particular QP and hence won't
                do SIDR.

                That is why I propose that the QPN be always included
                for the default case. SIDR usage is gated through the
                QPN flag and is in the realm of the advanced
                capabilities draft as has been suggested by many on
                this mailing list. And I agree.


                2) Address resolution that returns the QPN is done in
                one request/reply exchange. Any other method, while
                talking to the same endpoint, requires 6 messages. 2
                for ARP/ND and 2 each for each of the lower layer
                request/reply. IT is wasteful, more complicated,
                more time consuming wrt fast failover etc.


	Capability flags :

                Multiple people have opined that they would prefer in
                certain cases to map the protocol/ULP endpoints to
                specific QPs. People also want to use IP over RC mode
                (and others maybe) due to the advantages of larger MTU
                and Automatic path failover etc.

                The capability flags allow for interoperation between
                the default/dumb devices and the more sophisticated
                devices. IT also allows the endpoints to avoid
                unnecessary trial and error/to and fro when trying to
                utilise these features. The flag indicates the
                advanced capabilities supported and the receiver can
                choose to utilies them.


Result:

        We cannot depend on SIDR. We need to include the default QPN.
        We should not force implementations to go to the SA/SM (IBTA
        does not) and so allow LID/GID to be determined from the
        remote host. The default case works with :

	LID+Capability flags(all set to 0)+QPN+GID


Vivek




_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 14 18:13:13 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA24350
	for <ipoverib-archive@odin.ietf.org>; Mon, 14 Jan 2002 18:13:00 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id SAA00130
	for ipoverib-archive@odin.ietf.org; Mon, 14 Jan 2002 18:13:02 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA00102;
	Mon, 14 Jan 2002 18:11:59 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA00068
	for <ipoverib@optimus.ietf.org>; Mon, 14 Jan 2002 18:11:57 -0500 (EST)
Received: from omegaband.com (faust.seagullsemi.com [206.196.68.34])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA24296
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 18:11:54 -0500 (EST)
Received: from seahorsepc [207.8.52.226] by omegaband.com with ESMTP
  (SMTPD32-7.04) id A3F3BC10148; Mon, 14 Jan 2002 17:04:19 -0600
From: "Ashwin Kamath" <kamath@omegaband.com>
To: <ipoverib@ietf.org>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
Date: Mon, 14 Jan 2002 17:10:11 -0600
Message-ID: <IFEKJBDLPFOEGGPIIGMFIECJCAAA.kamath@omegaband.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0)
Importance: Normal
In-Reply-To: <200201142203.g0EM33qT126365@jurassic.eng.sun.com>
X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

There is a general thought that ARP is a one step solution for obtaining the
parameters required for establishing IPoIB over UD. But I think that even
for using ARP, one will require to get the Path record information. I do not
think that ARP is a one stop solution. What happens if both nodes determine
that their MTU is 2K without considering that a switch component in between
might restrict the MTU to 256 bytes? Such information can be obtained only
from the central SM via a path record. Am I missing something here? If
address resolution is all that ARP is required to do, without trying to
establish parameters for communication one may not want to know the MTU, but
from the discussions I see that the Address resolution is being used for
establishing communication parameters (like the QPN)

So the only real problem with having the LID in the ARP response, that I see
is when the ARP response needs to cross IB Routers. Can I assume that
changes in LID assignments are not a big problem because such changes are
not intended to be frequent and current ARP cache implementations already
factor in the possibility of a link level address change. Changing LIDs
would have a bigger impact on shared devices using RC than on IPoIB using
UD. Is it ok if I consider a change in LID assignment to be equivalent of a
server reboot?

If I understood what Jerry, one will have a problem if the ARP responder
lies in another IB Subnet, in which case the LID in the response will not be
relavant to the originator of the ARP request, in-fact the LID will have to
be the LID of the router. Was there any resolution reached for this problem?

In short you do need an ARP + Path record diagnosis to determine all the
parameters for communication. What would happen if we bring an IB router in
between? One cannot obtain a path record to a node in another subnet. One
can obtain the path record to the IB Router. Should we be talking about the
behaviour of the IB router in case of mismatch of MTUs or should we say that
we will always use the smallest mtu for communicating across an IB router.
Or should we say that this could fall on the plate of the Infiniband WGs.

-Ashwin.

-----Original Message-----
From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
Sent: None
To: kamath@omegaband.com
Cc: ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?


My comments are embeded in the following.

>I agree with using GID both for longevity and scope. What about using
>SIDR(Private Data) for communicating all the capability bits and any other
>extras? If it is not too late for this, I have some new feedback.
>
>For ARP on an IB subet one would require that
>a. IB Switches support multicast. (This is true in most cases today.)
>b. IP addresses used in an infiniband subnet are unique for use by IPoIB.
>(This is an important constraint which gets rid of the ethertype field)
>c. Some issues related to Multicast over IB need to be resolved (like
>multiple consumers using multicast resources.)

This one is really simple. A MGID contains 128 bits. Even IPv6 recognizes
that it only needs the lower 32bits for the forseeable future. So the
probability of MGID collisions is negligible. If one really wants
something bullet-proof, we can ask IBTA to reserve a chunk (or endorse
the IPoIB signature covered in the link and multicast draft) for IPoIB.

>
>Requirement (b) was addressed by defining a multicast group and the IP
>address format. One would still have to deal with the GID, LID and the IP
>address. Including the LID in the ARP Response will be difficult to deal
>with because
> a. LID Assignments change.
> b. ARP might have to traverse a router, for IP Subnets spanning IB Subnets
>(a) might not really be an issue because ARP entries are aged. In ethernet,
>the ethernet topology might change and MAC Addresses can change on server
>reboots (by way of using a different MAC or by assigning a MAC Address.)
The
>current ARP implementations do take these scenarios into account. Moreover
>Path record lookups alone would not help resolve the issue of changing
LIDs.
>The locally cached path record will not be valid if the LID changes. For a
>true solution, there has to be some way of using IB Traps to get
information
>about topology changes as and when it happens. An end node can subscribe
for

This is a generic, not IPoIB specific issue. So the work belongs to IBTA
to ensure some mechanism exists for flushing stale path records.

>a trap, but the end node will have to do something like,
> * Tell me when all the switches on the path to this node report a state
>change on such and such a port.
> This will not work very well right now because one will have to subscribe
>for traps on all switches leading to the end node, which implies that an
end
>node interested in the existence of another end node will need to have
>knowledge of the complete IB topology. This is probably something worth
>considering in the Infiniband WGs.

Ok, we're saying the same thing.

>
>On the same token, if you use the globally unique GID, information about
the
>GID will not be maintained by the local subnet's subnet administration and
>hence a path record lookup based on the GID would fail.

In this case you follow the IBA rule. The rule says that you don't even
attempt to lookup a pathrecord for a remote GID in a different IB subnet.
You need to find an IB route to it first. This is similar to IP.

> So if the GID lookup
>fails, one would send the ipoib packet to the designated IB router (as is
>done with ethernet today, albiet based on netmasks). But what if you have
an
>IP subnet spanning IB Subnets. The designated router exists for an IP
Subnet
>and the local subnet IB administration will not have information about all
>the IP nodes, because they could exist on another IB Subnet.

Hmmm, you seem to confuse routing at the IB level with that at the IP
level. (Or am I the one who is confused?)

>
> The following IB transactions are required before sending any IPoIB data
>over the subnet (assuming the multicast join is already done and we use
GIDs
>as link layer addresses)
> 1. IpoIB ARP.
> 2. Path record lookup.
> 3. SIDR
>
>In all this, the existence of the ARP is questionable. The same
>functionality can be achieved using 2 and 3 above.
>If you use GIDs as the IP addresses ,you can use a Path record lookup and
>then use SIDR to get all the information to use IPoIB. The advantages are
>evident, for eg,

I think similar idea has been brought up before. One major obstacle here
is that a given interface may host many (e.g. a few thousand) IP addresses
simultaneously (e.g. virtual hosting). You just don't have space for that
many GIDs on a HCA.

Jerry

> a. Traversing a router. Assuming that the path record lookup fails, one
>will redirect all the related ipoib packets to the Gateway/router. Since
ARP
>does not exist, one does not have to worry about the issues involved by ARP
>traversing a router, or the router proxying for end nodes on the same IP
>Subnet but different IB Subnets. IP and IB Subnets will be one and the
same.
> b. Debate on using the ether-type field vanishes. The Service ID will be
>unique to IPoIB and will be assigned by some WG. You could potentially use
>two Service IDs (might be an overkill), one for IPv6 packets and another
for
>IPv4.
> c. No special format for IP addresses is required (I have not thought this
>through very well.)
> d. Switches are not required to support Multicast.
> e. There are no issues with multiple consumers using the same multicast
>resources.
> f. Multicast could be used by an IB Gateways only to avoid multi-unicast.
>
>In this scenario the IP subnet is the same as an IB Subnet, since the same
>network addresses are used.
>
>-Ashwin
>
>
>
>-----Original Message-----
>From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
>Of H.K. Jerry Chu
>Sent: None
>To: ipoverib@ietf.org
>Subject: [Ipoverib] What's inside an IPoIB link layer address?
>
>
>While proposals beyond the basic IPoIB over UD started showing up,
>I still don't think we've reached a rough consensus on some basic
>questions such as what the IPoIB link layer address should contain.
>Again I'd like to apologize for not being there at the SLC meeting
>last month. Perhaps a consensus was reached there but i've missed
>the boat. In that case please let me know what it is and I'll shut
>up.
>
>What I'd like to see included in the IPoIB link layer address is
>
>GID
>QPN
>
>and I'd like to allow QPN field to be 0.
>
>What I don't like to see included is LID.
>
>Discussions:
>
>There has been a consensus of using GID, not LID as the basic
>IB address identifying the port hosting an IP address. GID is
>sufficiently stable, not subject to change across a node reboot
>or subnet merge (IB level) like LID is. Also people don't want
>to exclude the possibility of IPoIB link spanning more than
>one IB subnet. GID supports scope beyond local link so it's
>suitable for that purpose.
>
>Multiple GIDs with different scopes may have been assigned to a CA.
>It is required that a CA returns the GID with a scope no less than
>the one for the IPoIB link. E.g. if an IPoIB link covers multiple
>IB subnets, but a CA uses a link-local GID as part of its IPoIB
>link layer address, it's not going to work well.
>
>I don't see any value of including LID besides GID as proposed
>by Vivek. I believes a lookup of PathRecord is always needed
>in order for a node to retrieve complete path info before sending
>unicast pkts to another node. The destination LID is part of
>the PathRecord so including it in the link layer address returned
>from ARP/ND doesn't save work. But you have all the drawbacks that
>swayed people away from using LID - it adds volatility to the link
>address. LID changing due to subnet merge or other reasons requires
>potentially new mechanism being added to IP/driver implementation
>to trigger ARP/ND flush. Lastly, what should be the LID if the IP
>node is in a different IB subnet? Obviously it can't be the LID of
>an intermediate IB router.
>
>For QPN I'd like to suggest one thing. Since SIDR is a required
>feature by IBA, the QPN field is allowed to be zero. What that
>means is that the requester must use SIDR to retrieve the remote
>QPN, given the GID of a remote IP address.
>
>This feature allows the support of static/persistent "MAC addr"
>wrt node reboot by removing the volatile part of the link layer
>address - the QPN. In many implementations the QPN allocated for
>a CA may change across node reboot. Support of static "MAC addr"
>has been considered desirable by a number of people.
>
>I don't like to add additional requirement such as SIDR to IPoIB,
>and was trying to make the above feature "optional". Ideally nodes
>can express whether they'd like QPN to be returned by embedding
>some flags, such as "I don't talk SIDR so please return QPN" in
>their IPoIB link layer addresses. But the broadcast nature of
>IPv4/ARP forbids the use of different "MAC addrs" for the same IP
>address, so my attempt failed.
>
>Comment?
>
>Jerry
>
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>http://www.ietf.org/mailman/listinfo/ipoverib
>
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>http://www.ietf.org/mailman/listinfo/ipoverib



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 14 18:48:44 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA26077
	for <ipoverib-archive@odin.ietf.org>; Mon, 14 Jan 2002 18:48:44 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id SAA01156
	for ipoverib-archive@odin.ietf.org; Mon, 14 Jan 2002 18:48:47 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA01139;
	Mon, 14 Jan 2002 18:48:07 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA01108
	for <ipoverib@optimus.ietf.org>; Mon, 14 Jan 2002 18:48:04 -0500 (EST)
Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA26022
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 18:48:00 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e31.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id SAA237212;
	Mon, 14 Jan 2002 18:45:02 -0500
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0ENm27110664;
	Mon, 14 Jan 2002 16:48:02 -0700
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id g0ENm1g22951;
	Mon, 14 Jan 2002 15:48:01 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0ENm0F11674;
	Mon, 14 Jan 2002 15:48:00 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201142348.g0ENm0F11674@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
To: kamath@omegaband.com (Ashwin Kamath)
Date: Mon, 14 Jan 2002 15:47:59 -0800 (PST)
Cc: ipoverib@ietf.org
In-Reply-To: <IFEKJBDLPFOEGGPIIGMFIECJCAAA.kamath@omegaband.com> from "Ashwin Kamath" at Jan 14, 2002 04:10:11 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

> 
> There is a general thought that ARP is a one step solution for obtaining the
> parameters required for establishing IPoIB over UD. But I think that even
> for using ARP, one will require to get the Path record information. I do not
> think that ARP is a one stop solution. What happens if both nodes determine
> that their MTU is 2K without considering that a switch component in between
> might restrict the MTU to 256 bytes? Such information can be obtained only
> from the central SM via a path record. Am I missing something here? If

        The all-nodes multicast(IPv6)/broadcast(IPv4) addresses are
        mapped to a multicast GID. This multicast GID is setup with
        the MTU for the subnet/link. All nodes always IB_join this
        MGID. Thus the MTU is always determined from the multicast
        GID. So is the Q_Key to be used. See the ipoib architecture
        and multicast drafts that have been posted. You can get them
        from the ipoverib pages at IETF site.

        Other parameters LID/GID/QPN are received via ARP. SL is
        determined based on the implementation. So, ARP is and
        certainly can be a one-step solution like you mention in the
        begining of your note.


Vivek

> address resolution is all that ARP is required to do, without trying to
> establish parameters for communication one may not want to know the MTU, but
> from the discussions I see that the Address resolution is being used for
> establishing communication parameters (like the QPN)
> 
> So the only real problem with having the LID in the ARP response, that I see
> is when the ARP response needs to cross IB Routers. Can I assume that
> changes in LID assignments are not a big problem because such changes are
> not intended to be frequent and current ARP cache implementations already
> factor in the possibility of a link level address change. Changing LIDs
> would have a bigger impact on shared devices using RC than on IPoIB using
> UD. Is it ok if I consider a change in LID assignment to be equivalent of a
> server reboot?
> 
> If I understood what Jerry, one will have a problem if the ARP responder
> lies in another IB Subnet, in which case the LID in the response will not be
> relavant to the originator of the ARP request, in-fact the LID will have to
> be the LID of the router. Was there any resolution reached for this problem?
> 
> In short you do need an ARP + Path record diagnosis to determine all the
> parameters for communication. What would happen if we bring an IB router in
> between? One cannot obtain a path record to a node in another subnet. One
> can obtain the path record to the IB Router. Should we be talking about the
> behaviour of the IB router in case of mismatch of MTUs or should we say that
> we will always use the smallest mtu for communicating across an IB router.
> Or should we say that this could fall on the plate of the Infiniband WGs.
> 
> -Ashwin.
> 
> -----Original Message-----
> From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
> Sent: None
> To: kamath@omegaband.com
> Cc: ipoverib@ietf.org
> Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
> 
> 
> My comments are embeded in the following.
> 
> >I agree with using GID both for longevity and scope. What about using
> >SIDR(Private Data) for communicating all the capability bits and any other
> >extras? If it is not too late for this, I have some new feedback.
> >
> >For ARP on an IB subet one would require that
> >a. IB Switches support multicast. (This is true in most cases today.)
> >b. IP addresses used in an infiniband subnet are unique for use by IPoIB.
> >(This is an important constraint which gets rid of the ethertype field)
> >c. Some issues related to Multicast over IB need to be resolved (like
> >multiple consumers using multicast resources.)
> 
> This one is really simple. A MGID contains 128 bits. Even IPv6 recognizes
> that it only needs the lower 32bits for the forseeable future. So the
> probability of MGID collisions is negligible. If one really wants
> something bullet-proof, we can ask IBTA to reserve a chunk (or endorse
> the IPoIB signature covered in the link and multicast draft) for IPoIB.
> 
> >
> >Requirement (b) was addressed by defining a multicast group and the IP
> >address format. One would still have to deal with the GID, LID and the IP
> >address. Including the LID in the ARP Response will be difficult to deal
> >with because
> > a. LID Assignments change.
> > b. ARP might have to traverse a router, for IP Subnets spanning IB Subnets
> >(a) might not really be an issue because ARP entries are aged. In ethernet,
> >the ethernet topology might change and MAC Addresses can change on server
> >reboots (by way of using a different MAC or by assigning a MAC Address.)
> The
> >current ARP implementations do take these scenarios into account. Moreover
> >Path record lookups alone would not help resolve the issue of changing
> LIDs.
> >The locally cached path record will not be valid if the LID changes. For a
> >true solution, there has to be some way of using IB Traps to get
> information
> >about topology changes as and when it happens. An end node can subscribe
> for
> 
> This is a generic, not IPoIB specific issue. So the work belongs to IBTA
> to ensure some mechanism exists for flushing stale path records.
> 
> >a trap, but the end node will have to do something like,
> > * Tell me when all the switches on the path to this node report a state
> >change on such and such a port.
> > This will not work very well right now because one will have to subscribe
> >for traps on all switches leading to the end node, which implies that an
> end
> >node interested in the existence of another end node will need to have
> >knowledge of the complete IB topology. This is probably something worth
> >considering in the Infiniband WGs.
> 
> Ok, we're saying the same thing.
> 
> >
> >On the same token, if you use the globally unique GID, information about
> the
> >GID will not be maintained by the local subnet's subnet administration and
> >hence a path record lookup based on the GID would fail.
> 
> In this case you follow the IBA rule. The rule says that you don't even
> attempt to lookup a pathrecord for a remote GID in a different IB subnet.
> You need to find an IB route to it first. This is similar to IP.
> 
> > So if the GID lookup
> >fails, one would send the ipoib packet to the designated IB router (as is
> >done with ethernet today, albiet based on netmasks). But what if you have
> an
> >IP subnet spanning IB Subnets. The designated router exists for an IP
> Subnet
> >and the local subnet IB administration will not have information about all
> >the IP nodes, because they could exist on another IB Subnet.
> 
> Hmmm, you seem to confuse routing at the IB level with that at the IP
> level. (Or am I the one who is confused?)
> 
> >
> > The following IB transactions are required before sending any IPoIB data
> >over the subnet (assuming the multicast join is already done and we use
> GIDs
> >as link layer addresses)
> > 1. IpoIB ARP.
> > 2. Path record lookup.
> > 3. SIDR
> >
> >In all this, the existence of the ARP is questionable. The same
> >functionality can be achieved using 2 and 3 above.
> >If you use GIDs as the IP addresses ,you can use a Path record lookup and
> >then use SIDR to get all the information to use IPoIB. The advantages are
> >evident, for eg,
> 
> I think similar idea has been brought up before. One major obstacle here
> is that a given interface may host many (e.g. a few thousand) IP addresses
> simultaneously (e.g. virtual hosting). You just don't have space for that
> many GIDs on a HCA.
> 
> Jerry
> 
> > a. Traversing a router. Assuming that the path record lookup fails, one
> >will redirect all the related ipoib packets to the Gateway/router. Since
> ARP
> >does not exist, one does not have to worry about the issues involved by ARP
> >traversing a router, or the router proxying for end nodes on the same IP
> >Subnet but different IB Subnets. IP and IB Subnets will be one and the
> same.
> > b. Debate on using the ether-type field vanishes. The Service ID will be
> >unique to IPoIB and will be assigned by some WG. You could potentially use
> >two Service IDs (might be an overkill), one for IPv6 packets and another
> for
> >IPv4.
> > c. No special format for IP addresses is required (I have not thought this
> >through very well.)
> > d. Switches are not required to support Multicast.
> > e. There are no issues with multiple consumers using the same multicast
> >resources.
> > f. Multicast could be used by an IB Gateways only to avoid multi-unicast.
> >
> >In this scenario the IP subnet is the same as an IB Subnet, since the same
> >network addresses are used.
> >
> >-Ashwin
> >
> >
> >
> >-----Original Message-----
> >From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
> >Of H.K. Jerry Chu
> >Sent: None
> >To: ipoverib@ietf.org
> >Subject: [Ipoverib] What's inside an IPoIB link layer address?
> >
> >
> >While proposals beyond the basic IPoIB over UD started showing up,
> >I still don't think we've reached a rough consensus on some basic
> >questions such as what the IPoIB link layer address should contain.
> >Again I'd like to apologize for not being there at the SLC meeting
> >last month. Perhaps a consensus was reached there but i've missed
> >the boat. In that case please let me know what it is and I'll shut
> >up.
> >
> >What I'd like to see included in the IPoIB link layer address is
> >
> >GID
> >QPN
> >
> >and I'd like to allow QPN field to be 0.
> >
> >What I don't like to see included is LID.
> >
> >Discussions:
> >
> >There has been a consensus of using GID, not LID as the basic
> >IB address identifying the port hosting an IP address. GID is
> >sufficiently stable, not subject to change across a node reboot
> >or subnet merge (IB level) like LID is. Also people don't want
> >to exclude the possibility of IPoIB link spanning more than
> >one IB subnet. GID supports scope beyond local link so it's
> >suitable for that purpose.
> >
> >Multiple GIDs with different scopes may have been assigned to a CA.
> >It is required that a CA returns the GID with a scope no less than
> >the one for the IPoIB link. E.g. if an IPoIB link covers multiple
> >IB subnets, but a CA uses a link-local GID as part of its IPoIB
> >link layer address, it's not going to work well.
> >
> >I don't see any value of including LID besides GID as proposed
> >by Vivek. I believes a lookup of PathRecord is always needed
> >in order for a node to retrieve complete path info before sending
> >unicast pkts to another node. The destination LID is part of
> >the PathRecord so including it in the link layer address returned
> >from ARP/ND doesn't save work. But you have all the drawbacks that
> >swayed people away from using LID - it adds volatility to the link
> >address. LID changing due to subnet merge or other reasons requires
> >potentially new mechanism being added to IP/driver implementation
> >to trigger ARP/ND flush. Lastly, what should be the LID if the IP
> >node is in a different IB subnet? Obviously it can't be the LID of
> >an intermediate IB router.
> >
> >For QPN I'd like to suggest one thing. Since SIDR is a required
> >feature by IBA, the QPN field is allowed to be zero. What that
> >means is that the requester must use SIDR to retrieve the remote
> >QPN, given the GID of a remote IP address.
> >
> >This feature allows the support of static/persistent "MAC addr"
> >wrt node reboot by removing the volatile part of the link layer
> >address - the QPN. In many implementations the QPN allocated for
> >a CA may change across node reboot. Support of static "MAC addr"
> >has been considered desirable by a number of people.
> >
> >I don't like to add additional requirement such as SIDR to IPoIB,
> >and was trying to make the above feature "optional". Ideally nodes
> >can express whether they'd like QPN to be returned by embedding
> >some flags, such as "I don't talk SIDR so please return QPN" in
> >their IPoIB link layer addresses. But the broadcast nature of
> >IPv4/ARP forbids the use of different "MAC addrs" for the same IP
> >address, so my attempt failed.
> >
> >Comment?
> >
> >Jerry
> >
> >
> >_______________________________________________
> >IPoverIB mailing list
> >IPoverIB@ietf.org
> >http://www.ietf.org/mailman/listinfo/ipoverib
> >
> >
> >_______________________________________________
> >IPoverIB mailing list
> >IPoverIB@ietf.org
> >http://www.ietf.org/mailman/listinfo/ipoverib
> 
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 14 19:38:32 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA27389
	for <ipoverib-archive@odin.ietf.org>; Mon, 14 Jan 2002 19:38:27 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id TAA02799
	for ipoverib-archive@odin.ietf.org; Mon, 14 Jan 2002 19:38:29 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA02782;
	Mon, 14 Jan 2002 19:37:50 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA02731
	for <ipoverib@optimus.ietf.org>; Mon, 14 Jan 2002 19:37:47 -0500 (EST)
Received: from palrel12.hp.com (palrel12.hp.com [156.153.255.237])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA27352
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 19:37:45 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel12.hp.com (Postfix) with ESMTP id 21D9AE004A6
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 16:37:13 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id QAA13914
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 16:37:12 -0800 (PST)
Message-Id: <5.1.0.14.2.20020114142357.029b58f0@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Mon, 14 Jan 2002 16:37:08 -0800
To: ipoverib@ietf.org
From: Michael Krause <krause@cup.hp.com>
Subject: RE: [Ipoverib] IPoIB architecture/encapsulation (EtherType or
  not)
In-Reply-To: <HHEDIJGOEFLPCPNILMPAIEMPCEAA.bill@strahm.net>
References: <2E33960095B58E40A4D3345AB9F65EC102261843@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org


Two cents on a variety of subjects concerning what fields to exchange 
between endnodes.

Mike

Link address fields:

P_Key - For each partition that an endnode participates and requires IP 
over IB services, it will send an ARP request to identify other nodes on 
the IP subnet.  Only those endnodes that share a common P_Key will respond 
and thus be discovered.  Hence, all discussion of P_Key is irrelevant from 
the IP over IB perspective and should not be included within the IP over IB 
specification.

Q_Key - Q_Key provides no value to IP over IB solutions and should be 
treated as a well-defined value by the specification.  Recommend a value of 
0xFFFFFFFE be used.

GUID - The value of a GUID is primarily being ascertain how things are 
co-located on a given CA.  At first glance this seems to have some value 
but most people no longer think it is required to establish a communication 
channel between endnodes, in other words, use the GID to define the port 
and let software figure out any co-location using local means if 
required.  Hence, this field should not be included for IP over IB operations.

GID - In reality there are two ways of creating GIDs within IB (the 
original IP over IB architecture discussion isn't quite accurate). The 
first (called GID 0) and is created by concatenating the port's EUI64 with 
the GID prefix to create a 128-bit value.  The second by concatenating a 
SM-assigned 64-bit value with the GID prefix.  This second method provides 
a way for endnodes to have a variety of link addresses (e.g. one per OS 
instance, per set of IP addresses per endnode, etc.).  In general, given an 
IP subnet may span multiple IB subnets, this is the mandatory value to 
exchange to uniquely identify an endnode for address resolution purposes.

QP - There is some benefit to exchanging the QP as part of address 
resolution.  For example, an endnode may have different policies in terms 
of how a QP is mapped per IP subnet.  One might have one QP per IP address 
or may have one QP per multiple IP addresses (potentially for all IP 
communications).  As such, the ARP process should exchange the QP where it 
is known and should use QP 0xFFFFFF when it is unknown, e.g. an IP router 
at the edge of an IB fabric instance.

LID - There has been general discussion that IP subnets should be able to 
span multiple IB subnets.  As such, the LID is invisible to the address 
resolution process since this is an IB subnet only issue (whether it refers 
to an IB router or an IB endnode is irrelevant).  The counter arguments 
that one could optimize the address resolution process slightly by avoiding 
a trip to the SA / SM by including the LID as a visible exchanged item (it 
is always available since this can be derived from the IB packet in the 
first place) really don't provide much substance.  The resolution process 
should be able to resolve an address independent of the underlying IB 
fabric using a single algorithm.  Implementations can take appropriate 
steps to optimize their communications through smart caching and 
examination of the provided IB headers.

GRH - There has been some discussion of whether the GRH should be included 
for all ARP operations or not.  For an ARP request, one injects a multicast 
packet per partition.  The receiving endnode will receive the packet and 
can determine whether the source is on the same subnet or not and respond 
with either a GRH or a LRH-only packet.   As such, it seems GRH usage is 
implicit and the current IB specifications on usage do not require any 
additional work within the IP over IB specification.

Capability bits - If an IP router is allowed to exist between two endnodes, 
then the specification must include a method to exchange capabilities (e.g. 
additional IB transport services that may be used to for IP over IB 
communications).   Since this topology is allowed, the specification must 
provide a method to exchange capabilities through some form of standalone 
operation, i.e. an endnode must be able to resolve an address to an IP 
router. The endnode will then inject a capabilities enumeration message to 
the destination endnode.   Given the endnode must be able to perform this 
operation to guarantee interoperability separate from the address 
resolution process, it would be preferred if the capability discovery (only 
required if an endnode provides additional services beyond IP over IB which 
is not necessarily universal) be kept a separate operation.  Once the 
capabilities are determined, these can be cached for a fairly substantial 
amount of time such that there is no performance loss for the majority of 
the operations that may occur between endnodes.  In addition, the 
capabilities exchange may be used to provide a target QP to be used for 
each capability thus allowing one to optimize the communications to 
initiate / implementations of these capabilities.

Ethertype - Quite a bit of discussion has occurred but it may be fair to 
state that the address resolution provides a first level method to obtain 
the associated QP for a given IP address and that the capabilities exchange 
may be used to provide additional QP for a given capability (e.g. perform a 
RC establishment using QP X), it seems the Ethertype is no longer necessary 
and can be removed from the specification.  


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 14 19:51:35 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA27694
	for <ipoverib-archive@odin.ietf.org>; Mon, 14 Jan 2002 19:51:35 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id TAA02998
	for ipoverib-archive@odin.ietf.org; Mon, 14 Jan 2002 19:51:36 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA02963;
	Mon, 14 Jan 2002 19:47:40 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA02930
	for <ipoverib@optimus.ietf.org>; Mon, 14 Jan 2002 19:47:39 -0500 (EST)
Received: from omegaband.com (faust.seagullsemi.com [206.196.68.34])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA27602
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 19:47:35 -0500 (EST)
Received: from seahorsepc [207.8.52.226] by omegaband.com with ESMTP
  (SMTPD32-7.04) id AA907D0066; Mon, 14 Jan 2002 18:40:48 -0600
From: "Ashwin Kamath" <kamath@omegaband.com>
To: <ipoverib@ietf.org>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
Date: Mon, 14 Jan 2002 18:45:51 -0600
Message-ID: <IFEKJBDLPFOEGGPIIGMFOECKCAAA.kamath@omegaband.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0)
In-Reply-To: <200201142348.g0ENm0F11674@eng4.beaverton.ibm.com>
Importance: Normal
X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit


Vivek,

Is the assumption true that the MTUs established for switching multicast
packets will be the same as that used for Unicast switching? ie will
multicast packets from one end node to another traverse the same path as a
unicast packet? I personally would not try to configure a path which uses a
higher MTU for a multicast group vs a lower MTU for other path records, but
in general the SM is not restricted by this requirement. ( It is possible
that the SM chooses 4x 1K MTU links for normal data and 1x 4K mtu links for
Multicast data, if by some chance, the topology forces it to do so. If some
switches do not support multicast and atleast one path per node exists such
that multicast can be supported for the fabric as a whole, the end-to-end
paths for the unicast packets may differ from that of multicast packets for
any given pair of nodes and hence the mtus can be different, more
specifically, the mtus can be larger for multicast than for unicast. ) If it
is true that multicast and unicast packets traverse the same paths
end-to-end or if it is reasonable to assume the MTU to be that of the
Multicast group because that is the best possible configuration, I do not
see any issue, except that such a statement of restriction be made on the
subnet management for use with ipoib. I think such a restriction is
reasonable for operation of IPoIB though I do not know if it is essential,
neither can I offer an alternative at this point. Do you think that such a
statement of restriction is required?

 What about the maximum static rate of injection which is part of the path
record? This parameter is not part of the Multicast Group record and is a
function of the DLID for any node injecting packets into the fabric. The
compliance statments for static rate injection are called optional. But from
the compliance statements I could not determine whether they were optional
contingent of the CA not supporting rates higher than 2.5 GBPs or they are
truly optional.

 PS: You can ask me to shut up if it is kind of late for such questions? :)

-thanks,
Ashwin

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Vivek Kashyap
Sent: None
To: Ashwin Kamath
Cc: ipoverib@ietf.org
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?


>
> There is a general thought that ARP is a one step solution for obtaining
the
> parameters required for establishing IPoIB over UD. But I think that even
> for using ARP, one will require to get the Path record information. I do
not
> think that ARP is a one stop solution. What happens if both nodes
determine
> that their MTU is 2K without considering that a switch component in
between
> might restrict the MTU to 256 bytes? Such information can be obtained only
> from the central SM via a path record. Am I missing something here? If

        The all-nodes multicast(IPv6)/broadcast(IPv4) addresses are
        mapped to a multicast GID. This multicast GID is setup with
        the MTU for the subnet/link. All nodes always IB_join this
        MGID. Thus the MTU is always determined from the multicast
        GID. So is the Q_Key to be used. See the ipoib architecture
        and multicast drafts that have been posted. You can get them
        from the ipoverib pages at IETF site.

        Other parameters LID/GID/QPN are received via ARP. SL is
        determined based on the implementation. So, ARP is and
        certainly can be a one-step solution like you mention in the
        begining of your note.


Vivek

> address resolution is all that ARP is required to do, without trying to
> establish parameters for communication one may not want to know the MTU,
but
> from the discussions I see that the Address resolution is being used for
> establishing communication parameters (like the QPN)
>
> So the only real problem with having the LID in the ARP response, that I
see
> is when the ARP response needs to cross IB Routers. Can I assume that
> changes in LID assignments are not a big problem because such changes are
> not intended to be frequent and current ARP cache implementations already
> factor in the possibility of a link level address change. Changing LIDs
> would have a bigger impact on shared devices using RC than on IPoIB using
> UD. Is it ok if I consider a change in LID assignment to be equivalent of
a
> server reboot?
>
> If I understood what Jerry, one will have a problem if the ARP responder
> lies in another IB Subnet, in which case the LID in the response will not
be
> relavant to the originator of the ARP request, in-fact the LID will have
to
> be the LID of the router. Was there any resolution reached for this
problem?
>
> In short you do need an ARP + Path record diagnosis to determine all the
> parameters for communication. What would happen if we bring an IB router
in
> between? One cannot obtain a path record to a node in another subnet. One
> can obtain the path record to the IB Router. Should we be talking about
the
> behaviour of the IB router in case of mismatch of MTUs or should we say
that
> we will always use the smallest mtu for communicating across an IB router.
> Or should we say that this could fall on the plate of the Infiniband WGs.
>
> -Ashwin.
>
> -----Original Message-----
> From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
> Sent: None
> To: kamath@omegaband.com
> Cc: ipoverib@ietf.org
> Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>
>
> My comments are embeded in the following.
>
> >I agree with using GID both for longevity and scope. What about using
> >SIDR(Private Data) for communicating all the capability bits and any
other
> >extras? If it is not too late for this, I have some new feedback.
> >
> >For ARP on an IB subet one would require that
> >a. IB Switches support multicast. (This is true in most cases today.)
> >b. IP addresses used in an infiniband subnet are unique for use by IPoIB.
> >(This is an important constraint which gets rid of the ethertype field)
> >c. Some issues related to Multicast over IB need to be resolved (like
> >multiple consumers using multicast resources.)
>
> This one is really simple. A MGID contains 128 bits. Even IPv6 recognizes
> that it only needs the lower 32bits for the forseeable future. So the
> probability of MGID collisions is negligible. If one really wants
> something bullet-proof, we can ask IBTA to reserve a chunk (or endorse
> the IPoIB signature covered in the link and multicast draft) for IPoIB.
>
> >
> >Requirement (b) was addressed by defining a multicast group and the IP
> >address format. One would still have to deal with the GID, LID and the IP
> >address. Including the LID in the ARP Response will be difficult to deal
> >with because
> > a. LID Assignments change.
> > b. ARP might have to traverse a router, for IP Subnets spanning IB
Subnets
> >(a) might not really be an issue because ARP entries are aged. In
ethernet,
> >the ethernet topology might change and MAC Addresses can change on server
> >reboots (by way of using a different MAC or by assigning a MAC Address.)
> The
> >current ARP implementations do take these scenarios into account.
Moreover
> >Path record lookups alone would not help resolve the issue of changing
> LIDs.
> >The locally cached path record will not be valid if the LID changes. For
a
> >true solution, there has to be some way of using IB Traps to get
> information
> >about topology changes as and when it happens. An end node can subscribe
> for
>
> This is a generic, not IPoIB specific issue. So the work belongs to IBTA
> to ensure some mechanism exists for flushing stale path records.
>
> >a trap, but the end node will have to do something like,
> > * Tell me when all the switches on the path to this node report a state
> >change on such and such a port.
> > This will not work very well right now because one will have to
subscribe
> >for traps on all switches leading to the end node, which implies that an
> end
> >node interested in the existence of another end node will need to have
> >knowledge of the complete IB topology. This is probably something worth
> >considering in the Infiniband WGs.
>
> Ok, we're saying the same thing.
>
> >
> >On the same token, if you use the globally unique GID, information about
> the
> >GID will not be maintained by the local subnet's subnet administration
and
> >hence a path record lookup based on the GID would fail.
>
> In this case you follow the IBA rule. The rule says that you don't even
> attempt to lookup a pathrecord for a remote GID in a different IB subnet.
> You need to find an IB route to it first. This is similar to IP.
>
> > So if the GID lookup
> >fails, one would send the ipoib packet to the designated IB router (as is
> >done with ethernet today, albiet based on netmasks). But what if you have
> an
> >IP subnet spanning IB Subnets. The designated router exists for an IP
> Subnet
> >and the local subnet IB administration will not have information about
all
> >the IP nodes, because they could exist on another IB Subnet.
>
> Hmmm, you seem to confuse routing at the IB level with that at the IP
> level. (Or am I the one who is confused?)
>
> >
> > The following IB transactions are required before sending any IPoIB data
> >over the subnet (assuming the multicast join is already done and we use
> GIDs
> >as link layer addresses)
> > 1. IpoIB ARP.
> > 2. Path record lookup.
> > 3. SIDR
> >
> >In all this, the existence of the ARP is questionable. The same
> >functionality can be achieved using 2 and 3 above.
> >If you use GIDs as the IP addresses ,you can use a Path record lookup and
> >then use SIDR to get all the information to use IPoIB. The advantages are
> >evident, for eg,
>
> I think similar idea has been brought up before. One major obstacle here
> is that a given interface may host many (e.g. a few thousand) IP addresses
> simultaneously (e.g. virtual hosting). You just don't have space for that
> many GIDs on a HCA.
>
> Jerry
>
> > a. Traversing a router. Assuming that the path record lookup fails, one
> >will redirect all the related ipoib packets to the Gateway/router. Since
> ARP
> >does not exist, one does not have to worry about the issues involved by
ARP
> >traversing a router, or the router proxying for end nodes on the same IP
> >Subnet but different IB Subnets. IP and IB Subnets will be one and the
> same.
> > b. Debate on using the ether-type field vanishes. The Service ID will be
> >unique to IPoIB and will be assigned by some WG. You could potentially
use
> >two Service IDs (might be an overkill), one for IPv6 packets and another
> for
> >IPv4.
> > c. No special format for IP addresses is required (I have not thought
this
> >through very well.)
> > d. Switches are not required to support Multicast.
> > e. There are no issues with multiple consumers using the same multicast
> >resources.
> > f. Multicast could be used by an IB Gateways only to avoid
multi-unicast.
> >
> >In this scenario the IP subnet is the same as an IB Subnet, since the
same
> >network addresses are used.
> >
> >-Ashwin
> >
> >
> >
> >-----Original Message-----
> >From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
> >Of H.K. Jerry Chu
> >Sent: None
> >To: ipoverib@ietf.org
> >Subject: [Ipoverib] What's inside an IPoIB link layer address?
> >
> >
> >While proposals beyond the basic IPoIB over UD started showing up,
> >I still don't think we've reached a rough consensus on some basic
> >questions such as what the IPoIB link layer address should contain.
> >Again I'd like to apologize for not being there at the SLC meeting
> >last month. Perhaps a consensus was reached there but i've missed
> >the boat. In that case please let me know what it is and I'll shut
> >up.
> >
> >What I'd like to see included in the IPoIB link layer address is
> >
> >GID
> >QPN
> >
> >and I'd like to allow QPN field to be 0.
> >
> >What I don't like to see included is LID.
> >
> >Discussions:
> >
> >There has been a consensus of using GID, not LID as the basic
> >IB address identifying the port hosting an IP address. GID is
> >sufficiently stable, not subject to change across a node reboot
> >or subnet merge (IB level) like LID is. Also people don't want
> >to exclude the possibility of IPoIB link spanning more than
> >one IB subnet. GID supports scope beyond local link so it's
> >suitable for that purpose.
> >
> >Multiple GIDs with different scopes may have been assigned to a CA.
> >It is required that a CA returns the GID with a scope no less than
> >the one for the IPoIB link. E.g. if an IPoIB link covers multiple
> >IB subnets, but a CA uses a link-local GID as part of its IPoIB
> >link layer address, it's not going to work well.
> >
> >I don't see any value of including LID besides GID as proposed
> >by Vivek. I believes a lookup of PathRecord is always needed
> >in order for a node to retrieve complete path info before sending
> >unicast pkts to another node. The destination LID is part of
> >the PathRecord so including it in the link layer address returned
> >from ARP/ND doesn't save work. But you have all the drawbacks that
> >swayed people away from using LID - it adds volatility to the link
> >address. LID changing due to subnet merge or other reasons requires
> >potentially new mechanism being added to IP/driver implementation
> >to trigger ARP/ND flush. Lastly, what should be the LID if the IP
> >node is in a different IB subnet? Obviously it can't be the LID of
> >an intermediate IB router.
> >
> >For QPN I'd like to suggest one thing. Since SIDR is a required
> >feature by IBA, the QPN field is allowed to be zero. What that
> >means is that the requester must use SIDR to retrieve the remote
> >QPN, given the GID of a remote IP address.
> >
> >This feature allows the support of static/persistent "MAC addr"
> >wrt node reboot by removing the volatile part of the link layer
> >address - the QPN. In many implementations the QPN allocated for
> >a CA may change across node reboot. Support of static "MAC addr"
> >has been considered desirable by a number of people.
> >
> >I don't like to add additional requirement such as SIDR to IPoIB,
> >and was trying to make the above feature "optional". Ideally nodes
> >can express whether they'd like QPN to be returned by embedding
> >some flags, such as "I don't talk SIDR so please return QPN" in
> >their IPoIB link layer addresses. But the broadcast nature of
> >IPv4/ARP forbids the use of different "MAC addrs" for the same IP
> >address, so my attempt failed.
> >
> >Comment?
> >
> >Jerry
> >
> >
> >_______________________________________________
> >IPoverIB mailing list
> >IPoverIB@ietf.org
> >http://www.ietf.org/mailman/listinfo/ipoverib
> >
> >
> >_______________________________________________
> >IPoverIB mailing list
> >IPoverIB@ietf.org
> >http://www.ietf.org/mailman/listinfo/ipoverib
>
>
>
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
>


--
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com
vivk@us.ibm.com
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 14 19:58:51 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA27861
	for <ipoverib-archive@odin.ietf.org>; Mon, 14 Jan 2002 19:58:51 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id TAA03289
	for ipoverib-archive@odin.ietf.org; Mon, 14 Jan 2002 19:58:52 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA03272;
	Mon, 14 Jan 2002 19:58:10 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA03232
	for <ipoverib@optimus.ietf.org>; Mon, 14 Jan 2002 19:58:07 -0500 (EST)
Received: from patan.sun.com (patan.Sun.COM [192.18.98.43])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA27849
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 19:58:05 -0500 (EST)
Received: from sunmail1.Sun.COM ([129.145.1.2])
	by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id RAA07279;
	Mon, 14 Jan 2002 17:58:06 -0700 (MST)
Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.81.144])
	by sunmail1.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1-Sun.COM.mod.2) with ESMTP id QAA10712;
	Mon, 14 Jan 2002 16:59:17 -0800 (PST)
Received: from taipei (taipei.Eng.Sun.COM [129.146.85.178])
	by jurassic.eng.sun.com (8.12.2+Sun/8.12.2) with SMTP id g0F0w5qT167763;
	Mon, 14 Jan 2002 16:58:05 -0800 (PST)
Message-Id: <200201150058.g0F0w5qT167763@jurassic.eng.sun.com>
Date: Mon, 14 Jan 2002 16:55:36 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
To: kashyapv@us.ibm.com
Cc: ipoverib@ietf.org
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: 0zseKrAOQ/VxFKEplMM1uw==
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.6_06 SunOS 5.8 sun4u sparc 
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

My comments are interspersed below:

...
<snip>

>
>
>	Suggested Link description to be used in address resolution:
>
>		LID + capability flags + GID + QPN
>
>
> 
>       Why the LID?
>
>             People said they don't want to go to the SM if possible
>             (LID allows for that. As noted here and LWG the SL can be
>             determined based on QoS or other parameters. The SL may
>             also be determined from the SA/SM)

I don't think pathrecord lookup can be avoided or should be skipped.
You're proposing something that IMHO is beyond my understanding of
how things work in IBA. Can you get an experienced IBTA member
to substantiate your statements here?

And what's the real benefit of saving just a lookup out of an already
infrequent operation like ARP due to cacheing?

>
>             People said they want the process to end in small number
>             of messages (LID allows a query/response to settle the
>             issue. As noted one doesn't necessarily have to go to the
>             SM to get the SL).
>
>             LID is always needed. (you can't escape it).

There are a number of issues of including LID as I've pointed out in
another mail.

>
>             BTW, people said they want the 'link-address' to be 48 bits.
>             (LID + QPN + cap_flags is 48 bits).

I'm confused. Aren't you proposing to include GID as well. Where does
this 48 bits come from?

>
>
>       Why the GID ?
>       
>                People want IP to span across IB subnets. We are
>                stretching stretching here since such implementations
>                are not possible yet.
>                However, we can define it such that it will work in
>                the future. Therefore include the GID.

You're ignoring all the other good reasons of using GID as I and many
have pointed out before (recently in another mail).

>
>	GUID
>
>		Can be made part of the GID.
>
>
>	Why the QPN: 
>
>                1) There is only one other way to get the QPN. That is
>                the SIDR protocol. However, this has a drawback.
>
>		SIDR is NOT REQUIRED by IBTA.
>
>                I had raised this point last August 'Feasibility of
>                SIDR': 
>
>                ***** Section 12.11 essentially states that SIDR is
>                optional. As the specification states:
>
>               << GSAs shall support this protocol if non-management
>               services are pro-vided on the Channel Adapter at other
>               than fixed QPNs. If this protocol is *not supported*,
>               the status "The method/attribute combination is not
>               sup-ported" shall be returned, as described in section
>               Table 101 MAD Common Status Field Bit Values >>
>
>		*****
>
>                That still holds. SIDR is NOT required by IBTA. A
>                device can always choose to provide all non-management
>                services at fixed QPNs. It may just choose to provide
>                IP and nothing else at a particular QP and hence won't
>                do SIDR.
>
>                That is why I propose that the QPN be always included
>                for the default case. SIDR usage is gated through the
>                QPN flag and is in the realm of the advanced
>                capabilities draft as has been suggested by many on
>                this mailing list. And I agree.

I agree in general that SIDR should not be required by IPoIB.
I'd second what Arvind said, before we can fully define what
these capability are meant to work, let's mark these bits as
simply reserved.

Jerry
 
>
>
>                2) Address resolution that returns the QPN is done in
>                one request/reply exchange. Any other method, while
>                talking to the same endpoint, requires 6 messages. 2
>                for ARP/ND and 2 each for each of the lower layer
>                request/reply. IT is wasteful, more complicated,
>                more time consuming wrt fast failover etc.
>
>
>	Capability flags :
>
>                Multiple people have opined that they would prefer in
>                certain cases to map the protocol/ULP endpoints to
>                specific QPs. People also want to use IP over RC mode
>                (and others maybe) due to the advantages of larger MTU
>                and Automatic path failover etc.
>
>                The capability flags allow for interoperation between
>                the default/dumb devices and the more sophisticated
>                devices. IT also allows the endpoints to avoid
>                unnecessary trial and error/to and fro when trying to
>                utilise these features. The flag indicates the
>                advanced capabilities supported and the receiver can
>                choose to utilies them.

I'd second what Arvind said, before we have a consensus that
including these capability flags in the link addr is a good thing
and have their meaning fully defined, let's mark these bits as
simply reserved.

Jerry

>
>
>Result:
>
>        We cannot depend on SIDR. We need to include the default QPN.
>        We should not force implementations to go to the SA/SM (IBTA
>        does not) and so allow LID/GID to be determined from the
>        remote host. The default case works with :
>
>	LID+Capability flags(all set to 0)+QPN+GID
>
>
>Vivek
>
>
>


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 14 19:58:57 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA27878
	for <ipoverib-archive@odin.ietf.org>; Mon, 14 Jan 2002 19:58:52 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id TAA03303
	for ipoverib-archive@odin.ietf.org; Mon, 14 Jan 2002 19:58:54 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA03224;
	Mon, 14 Jan 2002 19:58:04 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA03196
	for <ipoverib@optimus.ietf.org>; Mon, 14 Jan 2002 19:58:03 -0500 (EST)
Received: from palrel11.hp.com (palrel11.hp.com [156.153.255.246])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA27846
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 19:58:00 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel11.hp.com (Postfix) with ESMTP id 5EAEAE005AF
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 16:57:32 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id QAA17790
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 16:57:32 -0800 (PST)
Message-Id: <5.1.0.14.2.20020114164653.00ad7d40@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Mon, 14 Jan 2002 16:57:13 -0800
To: <ipoverib@ietf.org>
From: Michael Krause <krause@cup.hp.com>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
In-Reply-To: <IFEKJBDLPFOEGGPIIGMFIECJCAAA.kamath@omegaband.com>
References: <200201142203.g0EM33qT126365@jurassic.eng.sun.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

At 05:10 PM 1/14/2002 -0600, Ashwin Kamath wrote:
>There is a general thought that ARP is a one step solution for obtaining the
>parameters required for establishing IPoIB over UD. But I think that even
>for using ARP, one will require to get the Path record information. I do not
>think that ARP is a one stop solution. What happens if both nodes determine
>that their MTU is 2K without considering that a switch component in between
>might restrict the MTU to 256 bytes?

The minimum supported MTU for UD traffic is 2KB so a path with a 256 bytes 
MTU should not be used by the IP over IB service.  The SM working with the 
partition manager will establish a set of paths that can be used for IP 
over IB traffic which will include the setting up of an all nodes multicast 
group per partition that support IP traffic.

>Such information can be obtained only
>from the central SM via a path record. Am I missing something here? If
>address resolution is all that ARP is required to do, without trying to
>establish parameters for communication one may not want to know the MTU, but
>from the discussions I see that the Address resolution is being used for
>establishing communication parameters (like the QPN)

QP identification is an optimization that allows one to eliminate the 
Ethertype field.  It also allows an implementation to map different IP 
addresses to different QP thereby providing an implementation optimization 
through more fine grain hardware-based de-multiplexing.  An example use 
might be to assign an IP address per OS instance which shares a HCA port 
and then allow all traffic to target a corresponding QP without having to 
flow through an intermediate hypervisor that would have to de-multiplex the 
traffic.  This can provide both performance and scalability improvements 
for vendor offerings.

>So the only real problem with having the LID in the ARP response, that I see
>is when the ARP response needs to cross IB Routers. Can I assume that
>changes in LID assignments are not a big problem because such changes are
>not intended to be frequent and current ARP cache implementations already
>factor in the possibility of a link level address change. Changing LIDs
>would have a bigger impact on shared devices using RC than on IPoIB using
>UD. Is it ok if I consider a change in LID assignment to be equivalent of a
>server reboot?

No.  Applications should not be impacted by a change in LID assignment as 
there are defined methods to migrate applications from one LID or GID or 
port to another that are completely transparent to the application.  For 
UD-based services, one could simply re-ARP the endnode via the all-nodes 
multicast and provide a simple update of the LID mapping as this can be 
derived from the IB headers if desired.

>If I understood what Jerry, one will have a problem if the ARP responder
>lies in another IB Subnet, in which case the LID in the response will not be
>relavant to the originator of the ARP request, in-fact the LID will have to
>be the LID of the router. Was there any resolution reached for this problem?
>
>In short you do need an ARP + Path record diagnosis to determine all the
>parameters for communication. What would happen if we bring an IB router in
>between? One cannot obtain a path record to a node in another subnet. One
>can obtain the path record to the IB Router. Should we be talking about the
>behaviour of the IB router in case of mismatch of MTUs or should we say that
>we will always use the smallest mtu for communicating across an IB router.
>Or should we say that this could fall on the plate of the Infiniband WGs.

The IB router protocol will define how one routes across IB subnets - an 
IBTA issue.  An IP router may be implemented along side an IB router or it 
may be an edge device such as a multi-protocol router.  The fact that IP 
subnets may span multiple IB subnets isn't really that big of an issue per 
se.  By keeping the LID not part of the link address structure, the ARP 
should be able to transparently flow across multiple IB subnets.

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 14 20:21:06 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA28323
	for <ipoverib-archive@odin.ietf.org>; Mon, 14 Jan 2002 20:21:02 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id UAA04102
	for ipoverib-archive@odin.ietf.org; Mon, 14 Jan 2002 20:21:03 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA04085;
	Mon, 14 Jan 2002 20:20:28 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA04051
	for <ipoverib@optimus.ietf.org>; Mon, 14 Jan 2002 20:20:27 -0500 (EST)
Received: from palrel11.hp.com (palrel11.hp.com [156.153.255.246])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA28305
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 20:20:24 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel11.hp.com (Postfix) with ESMTP id 20E45E00409
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 17:19:56 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id RAA22756
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 17:19:55 -0800 (PST)
Message-Id: <5.1.0.14.2.20020114171136.029bf830@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Mon, 14 Jan 2002 17:19:30 -0800
To: <ipoverib@ietf.org>
From: Michael Krause <krause@cup.hp.com>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
In-Reply-To: <IFEKJBDLPFOEGGPIIGMFOECKCAAA.kamath@omegaband.com>
References: <200201142348.g0ENm0F11674@eng4.beaverton.ibm.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

At 06:45 PM 1/14/2002 -0600, Ashwin Kamath wrote:

>Vivek,
>
>Is the assumption true that the MTUs established for switching multicast
>packets will be the same as that used for Unicast switching? ie will
>multicast packets from one end node to another traverse the same path as a
>unicast packet? I personally would not try to configure a path which uses a
>higher MTU for a multicast group vs a lower MTU for other path records, but
>in general the SM is not restricted by this requirement. ( It is possible
>that the SM chooses 4x 1K MTU links for normal data and 1x 4K mtu links for
>Multicast data, if by some chance, the topology forces it to do so.

The requirement should be to use a 2KB MTU as the default for all UD-based 
communications.  If the fabric can support a 4KB MTU, then it may also be 
used between endnodes.

>If some
>switches do not support multicast and atleast one path per node exists such
>that multicast can be supported for the fabric as a whole, the end-to-end
>paths for the unicast packets may differ from that of multicast packets for
>any given pair of nodes and hence the mtus can be different, more
>specifically, the mtus can be larger for multicast than for unicast. )

All switches should be required to support multicast so this class of 
problems does not exist. Given that most implementations appear to support 
multicast in order to support IP over IB, I don't see why we need to add 
any complexity to accommodate switches that do not support multicast.

>If it is true that multicast and unicast packets traverse the same paths
>end-to-end or if it is reasonable to assume the MTU to be that of the
>Multicast group because that is the best possible configuration, I do not
>see any issue, except that such a statement of restriction be made on the
>subnet management for use with ipoib. I think such a restriction is
>reasonable for operation of IPoIB though I do not know if it is essential,
>neither can I offer an alternative at this point. Do you think that such a
>statement of restriction is required?

The only restrictions for IP over IB in terms of what the IB fabric 
provides are 2KB PMTU  between participating endnodes and switch-based 
multicast (will apply to routers) support.

>What about the maximum static rate of injection which is part of the path
>record? This parameter is not part of the Multicast Group record and is a
>function of the DLID for any node injecting packets into the fabric. The
>compliance statments for static rate injection are called optional. But from
>the compliance statements I could not determine whether they were optional
>contingent of the CA not supporting rates higher than 2.5 GBPs or they are
>truly optional.

Static rate control is not optional for any port injecting packets into the 
fabric that operates at greater than 2.5 Gbps (current 1x 
bandwidth).   This is clearly indicated in both Chapter 9 compliance 
statements as well as chapter 17 table 254 (volume 1, release 1.0.a).

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 14 20:32:22 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA28630
	for <ipoverib-archive@odin.ietf.org>; Mon, 14 Jan 2002 20:32:22 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id UAA04736
	for ipoverib-archive@odin.ietf.org; Mon, 14 Jan 2002 20:32:24 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA04723;
	Mon, 14 Jan 2002 20:31:43 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA04670
	for <ipoverib@optimus.ietf.org>; Mon, 14 Jan 2002 20:31:40 -0500 (EST)
Received: from palrel12.hp.com (palrel12.hp.com [156.153.255.237])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA28618
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 20:31:37 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel12.hp.com (Postfix) with ESMTP id C04C4E000DF
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 17:31:09 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id RAA24734
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 17:31:09 -0800 (PST)
Message-Id: <5.1.0.14.2.20020114172317.028a3d68@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Mon, 14 Jan 2002 17:28:43 -0800
To: ipoverib@ietf.org
From: Michael Krause <krause@cup.hp.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
In-Reply-To: <200201150058.g0F0w5qT167763@jurassic.eng.sun.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

At 04:55 PM 1/14/2002 -0800, H.K. Jerry Chu wrote:
>My comments are interspersed below:
>
>...
><snip>
>
> >
> >
> >       Suggested Link description to be used in address resolution:
> >
> >               LID + capability flags + GID + QPN
> >
> >
> >
> >       Why the LID?
> >
> >             People said they don't want to go to the SM if possible
> >             (LID allows for that. As noted here and LWG the SL can be
> >             determined based on QoS or other parameters. The SL may
> >             also be determined from the SA/SM)
>
>I don't think pathrecord lookup can be avoided or should be skipped.
>You're proposing something that IMHO is beyond my understanding of
>how things work in IBA. Can you get an experienced IBTA member
>to substantiate your statements here?

An endnode may acquire the path record to determine the appropriate SL to 
use when communicating to another endnode.  Given SL is set on a per IB 
subnet basis, there is no fixed method to determine the appropriate SL to 
use within a given subnet.   Work is under consideration to develop a QoS 
mapping function that will allow an endnode to use the IB TClass field to 
communicate the QoS requirements at a global level. This would then allow 
each IB subnet to transparently map the IB TClass to an SL without 
requiring the application to explicitly set this field.

>And what's the real benefit of saving just a lookup out of an already
>infrequent operation like ARP due to cacheing?

Unclear.  One has to deal with IP subnets that span multiple IB subnets so 
having a single mechanism used all of the time seems more practical.


> >
> >             People said they want the process to end in small number
> >             of messages (LID allows a query/response to settle the
> >             issue. As noted one doesn't necessarily have to go to the
> >             SM to get the SL).
> >
> >             LID is always needed. (you can't escape it).
>
>There are a number of issues of including LID as I've pointed out in
>another mail.
>
> >
> >             BTW, people said they want the 'link-address' to be 48 bits.
> >             (LID + QPN + cap_flags is 48 bits).
>
>I'm confused. Aren't you proposing to include GID as well. Where does
>this 48 bits come from?

Perhaps someone is trying to re-use an existing Ethernet MAC structure but 
that is perhaps strictly an implementation objective outside the scope of 
this specification.

> >
> >       Why the GID ?
> >
> >                People want IP to span across IB subnets. We are
> >                stretching stretching here since such implementations
> >                are not possible yet.
> >                However, we can define it such that it will work in
> >                the future. Therefore include the GID.
>
>You're ignoring all the other good reasons of using GID as I and many
>have pointed out before (recently in another mail).
>
> >
> >       GUID
> >
> >               Can be made part of the GID.

Only if using GID 0 which may not always be the case for an endnode that 
supports multiple GID per port.

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 14 20:42:23 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA28885
	for <ipoverib-archive@odin.ietf.org>; Mon, 14 Jan 2002 20:42:23 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id UAA04951
	for ipoverib-archive@odin.ietf.org; Mon, 14 Jan 2002 20:42:25 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA04927;
	Mon, 14 Jan 2002 20:41:12 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA04895
	for <ipoverib@optimus.ietf.org>; Mon, 14 Jan 2002 20:41:10 -0500 (EST)
Received: from omegaband.com (faust.seagullsemi.com [206.196.68.34])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA28864
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 20:41:08 -0500 (EST)
Received: from seahorsepc [207.8.52.226] by omegaband.com with ESMTP
  (SMTPD32-7.04) id A71D1250062; Mon, 14 Jan 2002 19:34:21 -0600
From: "Ashwin Kamath" <kamath@omegaband.com>
To: <ipoverib@ietf.org>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
Date: Mon, 14 Jan 2002 19:39:24 -0600
Message-ID: <IFEKJBDLPFOEGGPIIGMFIECMCAAA.kamath@omegaband.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0)
In-Reply-To: <5.1.0.14.2.20020114171136.029bf830@esalpha2.cup.hp.com>
Importance: Normal
X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit


Mike,
 So it would be mandatory for the end nodes to determine the static rate
before injecting packets targetted for a particular DLID. The path record is
the only mechanism for an end node to determine the static rate for a path.
Does that make the path record query mandatory for using a UD queue pair for
IPoIB? If not then the suggested ARP mechanism for ipoib is truly
self-sufficient.
 Also since 2K minimum MTU is required, should this be stated as REQUIRED of
the ipoib requirements draft, or is it outside its scope?
-Ashwin.

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Michael Krause
Sent: Monday, January 14, 2002 7:20 PM
To: ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?


At 06:45 PM 1/14/2002 -0600, Ashwin Kamath wrote:

>Vivek,
>
>Is the assumption true that the MTUs established for switching multicast
>packets will be the same as that used for Unicast switching? ie will
>multicast packets from one end node to another traverse the same path as a
>unicast packet? I personally would not try to configure a path which uses a
>higher MTU for a multicast group vs a lower MTU for other path records, but
>in general the SM is not restricted by this requirement. ( It is possible
>that the SM chooses 4x 1K MTU links for normal data and 1x 4K mtu links for
>Multicast data, if by some chance, the topology forces it to do so.

The requirement should be to use a 2KB MTU as the default for all UD-based
communications.  If the fabric can support a 4KB MTU, then it may also be
used between endnodes.

>If some
>switches do not support multicast and atleast one path per node exists such
>that multicast can be supported for the fabric as a whole, the end-to-end
>paths for the unicast packets may differ from that of multicast packets for
>any given pair of nodes and hence the mtus can be different, more
>specifically, the mtus can be larger for multicast than for unicast. )

All switches should be required to support multicast so this class of
problems does not exist. Given that most implementations appear to support
multicast in order to support IP over IB, I don't see why we need to add
any complexity to accommodate switches that do not support multicast.

>If it is true that multicast and unicast packets traverse the same paths
>end-to-end or if it is reasonable to assume the MTU to be that of the
>Multicast group because that is the best possible configuration, I do not
>see any issue, except that such a statement of restriction be made on the
>subnet management for use with ipoib. I think such a restriction is
>reasonable for operation of IPoIB though I do not know if it is essential,
>neither can I offer an alternative at this point. Do you think that such a
>statement of restriction is required?

The only restrictions for IP over IB in terms of what the IB fabric
provides are 2KB PMTU  between participating endnodes and switch-based
multicast (will apply to routers) support.

>What about the maximum static rate of injection which is part of the path
>record? This parameter is not part of the Multicast Group record and is a
>function of the DLID for any node injecting packets into the fabric. The
>compliance statments for static rate injection are called optional. But
from
>the compliance statements I could not determine whether they were optional
>contingent of the CA not supporting rates higher than 2.5 GBPs or they are
>truly optional.

Static rate control is not optional for any port injecting packets into the
fabric that operates at greater than 2.5 Gbps (current 1x
bandwidth).   This is clearly indicated in both Chapter 9 compliance
statements as well as chapter 17 table 254 (volume 1, release 1.0.a).

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 14 22:28:45 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA01810
	for <ipoverib-archive@odin.ietf.org>; Mon, 14 Jan 2002 22:28:44 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id WAA07896
	for ipoverib-archive@odin.ietf.org; Mon, 14 Jan 2002 22:28:48 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA07865;
	Mon, 14 Jan 2002 22:26:55 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA07834
	for <ipoverib@optimus.ietf.org>; Mon, 14 Jan 2002 22:26:49 -0500 (EST)
Received: from palrel13.hp.com (palrel13.hp.com [156.153.255.238])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA01796
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 22:26:45 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel13.hp.com (Postfix) with ESMTP id 4F268E001DA
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 19:26:18 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id TAA16131
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 19:26:18 -0800 (PST)
Message-Id: <5.1.0.14.2.20020114182406.028a0b48@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Mon, 14 Jan 2002 19:26:09 -0800
To: <ipoverib@ietf.org>
From: Michael Krause <krause@cup.hp.com>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
In-Reply-To: <IFEKJBDLPFOEGGPIIGMFIECMCAAA.kamath@omegaband.com>
References: <5.1.0.14.2.20020114171136.029bf830@esalpha2.cup.hp.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

At 07:39 PM 1/14/2002 -0600, Ashwin Kamath wrote:

>Mike,
>  So it would be mandatory for the end nodes to determine the static rate
>before injecting packets targetted for a particular DLID.

Yes.

>The path record is the only mechanism for an end node to determine the 
>static rate for a path.
>Does that make the path record query mandatory for using a UD queue pair 
>for IPoIB?

For the initial unicast communication, one must know what the static rate 
is for a given path within an IB subnet.  I would also contend that one 
will need to know this when crossing IB subnets as well should an IP subnet 
span multiple IB subnets.  This is currently under investigation for the IB 
router specification.  A multi-protocol IP router at the edge of an IB 
fabric (of any diameter including a single link if desired) would act as an 
injection point (CA) and would be required to support static rate control 
if operating at greater than 2.5 Gbps.

>If not then the suggested ARP mechanism for ipoib is truly self-sufficient.
>Also since 2K minimum MTU is required, should this be stated as REQUIRED of
>the ipoib requirements draft, or is it outside its scope?

Yes. Required.

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 14 23:37:40 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA04323
	for <ipoverib-archive@odin.ietf.org>; Mon, 14 Jan 2002 23:37:40 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id XAA10506
	for ipoverib-archive@odin.ietf.org; Mon, 14 Jan 2002 23:37:44 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id XAA10480;
	Mon, 14 Jan 2002 23:36:00 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id XAA10448
	for <ipoverib@optimus.ietf.org>; Mon, 14 Jan 2002 23:35:58 -0500 (EST)
Received: from sentinel.sanera.net (ns1.sanera.net [208.253.254.10])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA04309
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 23:35:53 -0500 (EST)
Received: from icarus.sanera.net (icarus [192.168.254.11])
	by sentinel.sanera.net (8.11.2/8.11.2) with ESMTP id g0F4ZQu16918
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 20:35:26 -0800
Received: from newton.sanera.net (newton.sanera.net [172.16.2.11])
	by icarus.sanera.net (8.11.6/8.11.6) with ESMTP id g0F4ZLn03487
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 20:35:21 -0800
Received: from sanera.net (dhcp-pc-100-161.sanera.net [192.168.100.161])
	by newton.sanera.net (8.11.4/8.11.4) with ESMTP id g0F4ZKM28294
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 20:35:20 -0800 (PST)
Message-ID: <3C43B187.9C2BB72C@sanera.net>
Date: Mon, 14 Jan 2002 20:35:19 -0800
From: Joe Pelissier <joep@Sanera.net>
X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U)
X-Accept-Language: en
MIME-Version: 1.0
To: ipoverib@ietf.org
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
References: <200201142203.g0EM33qT126365@jurassic.eng.sun.com> <5.1.0.14.2.20020114164653.00ad7d40@esalpha2.cup.hp.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Greetings:
I unfortunately have had to reduce my participation in this effort for a while,
but have now been studying the email threads attempting to get back up to speed.
My apologies in advance if my comments below are missing some context from
previous disucssions.

It seems that there continues to be a great deal of discussion as to whether ARP
can be accomplished without consulting the SA (or at least cached information
from a previous consultation).  There seems to be two major issues: static rate
and service level.

All nodes that operate at greater than x1 speed are required to support static
rate.  The static rate setting is a function of the path between two endpoints.
It is important to note that the path multicast uses is very likely to be quite
different than the path that unicast traffic uses.  Therefore, the static rate
may well be different.  The only way to get this information is to consult the SA
(actually, one could always set their static rate to x1 speed and avoid the
consultation, but if you got a x12 link, it seems a shame not to use it just to
avoid talking to your friendly SA ;-).

The other is SL.  It has repeatedly been argued that the spec does not require
getting the SL from the SA.  This is true.  This has been recognized by the IBTA
and steps are being taken to correct it.  In order to support multiple paths
through the IB fabric, it was a fundamental assumption that the SA controls which
SLs a node may use based on path.  The SA may provide a set of SLs from which a
node may choose, but the SA needs to control this.  If it does not, deadlock of
the fabric may occur.  To prevent deadlock, the SA can load certain IB tables
that would cause the offending packets to be discarded.  This protects the
fabric, but would cause communication to fail between the offending nodes.

So we could argue all this forever, or maybe there is another solution.  The IB
spec already talks about how to determine both static rate and SL.  Therefore, we
don't need to rehash it in the RFC (other than maybe a note that says these are
determined IAW the InfiniBand standard).

wrt/ SL, this is pretty much what section 6.0 of
draft-kashyap-ipoib-ipv4-and-arp-01.txt says, except for the last paragraph
(which recommends using the SL from the broadcast group).  Unfortunately, the
recommendation is directly contrary to the some intended usage of IB and is
likely to either be prohibited or at least strongly discouraged by a future
version of the specification.

The LWG voted unanimously to recommend the addition of language that strongly
discourages the use of parameters that were not obtained by an applicable path
record and stating that the parameters from multicast and unicast should not be
mixed (see the LWG meeting minutes dated 8/16/01).

If we just remove the recommendation from the draft, then we can move forward
without further discussion.  It just seems bazaar to continue to push something
as "recommended" that causes great concern to many of the architects that created
IB.  All we need to do is leave out the recommendation and we are done with this
issue.  Those that wish to ignore the SL assignment from the SA and use the
multicast one can still do so without technically violating the current IBTA
spec.  There seems to be little downside to this.

Besides, you need to get static rate anyway....

Another $0.02 worth...
Regards,
Joe Pelissier

Michael Krause wrote:

> At 05:10 PM 1/14/2002 -0600, Ashwin Kamath wrote:
> >There is a general thought that ARP is a one step solution for obtaining the
> >parameters required for establishing IPoIB over UD. But I think that even
> >for using ARP, one will require to get the Path record information. I do not
> >think that ARP is a one stop solution. What happens if both nodes determine
> >that their MTU is 2K without considering that a switch component in between
> >might restrict the MTU to 256 bytes?
>
> The minimum supported MTU for UD traffic is 2KB so a path with a 256 bytes
> MTU should not be used by the IP over IB service.  The SM working with the
> partition manager will establish a set of paths that can be used for IP
> over IB traffic which will include the setting up of an all nodes multicast
> group per partition that support IP traffic.
>
> >Such information can be obtained only
> >from the central SM via a path record. Am I missing something here? If
> >address resolution is all that ARP is required to do, without trying to
> >establish parameters for communication one may not want to know the MTU, but
> >from the discussions I see that the Address resolution is being used for
> >establishing communication parameters (like the QPN)
>
> QP identification is an optimization that allows one to eliminate the
> Ethertype field.  It also allows an implementation to map different IP
> addresses to different QP thereby providing an implementation optimization
> through more fine grain hardware-based de-multiplexing.  An example use
> might be to assign an IP address per OS instance which shares a HCA port
> and then allow all traffic to target a corresponding QP without having to
> flow through an intermediate hypervisor that would have to de-multiplex the
> traffic.  This can provide both performance and scalability improvements
> for vendor offerings.
>
> >So the only real problem with having the LID in the ARP response, that I see
> >is when the ARP response needs to cross IB Routers. Can I assume that
> >changes in LID assignments are not a big problem because such changes are
> >not intended to be frequent and current ARP cache implementations already
> >factor in the possibility of a link level address change. Changing LIDs
> >would have a bigger impact on shared devices using RC than on IPoIB using
> >UD. Is it ok if I consider a change in LID assignment to be equivalent of a
> >server reboot?
>
> No.  Applications should not be impacted by a change in LID assignment as
> there are defined methods to migrate applications from one LID or GID or
> port to another that are completely transparent to the application.  For
> UD-based services, one could simply re-ARP the endnode via the all-nodes
> multicast and provide a simple update of the LID mapping as this can be
> derived from the IB headers if desired.
>
> >If I understood what Jerry, one will have a problem if the ARP responder
> >lies in another IB Subnet, in which case the LID in the response will not be
> >relavant to the originator of the ARP request, in-fact the LID will have to
> >be the LID of the router. Was there any resolution reached for this problem?
> >
> >In short you do need an ARP + Path record diagnosis to determine all the
> >parameters for communication. What would happen if we bring an IB router in
> >between? One cannot obtain a path record to a node in another subnet. One
> >can obtain the path record to the IB Router. Should we be talking about the
> >behaviour of the IB router in case of mismatch of MTUs or should we say that
> >we will always use the smallest mtu for communicating across an IB router.
> >Or should we say that this could fall on the plate of the Infiniband WGs.
>
> The IB router protocol will define how one routes across IB subnets - an
> IBTA issue.  An IP router may be implemented along side an IB router or it
> may be an edge device such as a multi-protocol router.  The fact that IP
> subnets may span multiple IB subnets isn't really that big of an issue per
> se.  By keeping the LID not part of the link address structure, the ARP
> should be able to transparently flow across multiple IB subnets.
>
> Mike
>
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 14 23:59:31 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA04601
	for <ipoverib-archive@odin.ietf.org>; Mon, 14 Jan 2002 23:59:17 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id XAA10766
	for ipoverib-archive@odin.ietf.org; Mon, 14 Jan 2002 23:59:19 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id XAA10745;
	Mon, 14 Jan 2002 23:58:32 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id XAA10716
	for <ipoverib@optimus.ietf.org>; Mon, 14 Jan 2002 23:58:30 -0500 (EST)
Received: from mxout1.netvision.net.il (mxout1.netvision.net.il [194.90.9.20])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA04594
	for <ipoverib@ietf.org>; Mon, 14 Jan 2002 23:58:27 -0500 (EST)
Received: from vaio505he ([32.101.212.35]) by mxout1.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GPY00A3ER4E2E@mxout1.netvision.net.il> for ipoverib@ietf.org;
 Tue, 15 Jan 2002 06:57:59 +0200 (IST)
Date: Tue, 15 Jan 2002 06:58:23 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
In-reply-to: <200201142245.g0EMjiV08688@eng4.beaverton.ibm.com>
To: "'Vivek Kashyap'" <kashyapv@us.ibm.com>, Jerry.Chu@eng.sun.com
Cc: ipoverib@ietf.org
Reply-to: yaronh@voltaire.com
Message-id: <014f01c19d81$4877b030$0100007f@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7BIT

Vivek,

I agree with your assumptions
We need them all: LID + capability flags + GID + QPN
Mikes remark that you could extract the LID from the incoming packet however
looks interesting (as long as we make sure that people don't play with it,
e.g. redirect traffic etc')

As far as I understand the only open issues currently are (if someone thinks
there are more please respond):
LID: Yes/No
Ethertype
Capability flags

As for the why LID:
I think that going to the SM is not so efficient given the fact that it is a
central resource and all the nodes in the subnet go to if we could avoid it
why not
When IP will span IB the problem becomes more serious: Unlike IP routers
which has a default route entry, currently IB has explicit entries for each
path this might require doing too many PathRecord operations, which are also
currently very slow when going outside an IB subnet
(I had posted a proposal to the MSO team trying to resolve those issues
somehow however it is under discussion)
I would tend to send this info anyway since the extra 16 bits wouldn't harm
anyone, one might choose not to use them if he wants to go to the SM, this
way everybody will be happy and over time people will use the best solution

As for the Ethertype:
After giving Vivek a hard time on this issue I accept his proposal
Not because of the performance issues raised (since I don't think they are
real)
But because it is not really needed, and if it simplifies things, why not

As for the capability bits:
Avoiding the SIDR is a smart move
I'm not sure if the UC option will be used, if using connection why not
reliable connection
I just wanted to add SDP (Socket Direct Protocol) bit, Again ones might
choose not to use it, for the one that do it really simplifies the process a
lot

I'm hoping we get a quick convergence on those issues so we would move
forward
The issues are not serious and I think if an alternative enable someone to
do something he wants without violating the other's ability to do what they
want just ignoring the extra info, we should go for it

Yaron

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Vivek Kashyap
Sent: Tuesday, January 15, 2002 12:46 AM
To: Jerry.Chu@eng.sun.com
Cc: ipoverib@ietf.org
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?

Folks,

       It is close to the last call. In this regard I would like to
       state that we have been discussing this issue for a long time
       now. The most recent discussion is what I presented at the SLC
       last december. (I certainly felt that the room was in agreement
       - no queries, objection etc.)

       I believe the following link-description provides the best
       assimilation of all the choices that have been put forth. If we
       step back and restart the whole discussion again we are likely
       to come back to the following - so here is a brief compilation
       of issues.


        Suggested Link description to be used in address resolution:

                LID + capability flags + GID + QPN



       Why the LID?

             People said they don't want to go to the SM if possible
             (LID allows for that. As noted here and LWG the SL can be
             determined based on QoS or other parameters. The SL may
             also be determined from the SA/SM)

             People said they want the process to end in small number
             of messages (LID allows a query/response to settle the
             issue. As noted one doesn't necessarily have to go to the
             SM to get the SL).

             LID is always needed. (you can't escape it).

             BTW, people said they want the 'link-address' to be 48 bits.
             (LID + QPN + cap_flags is 48 bits).


       Why the GID ?

                People want IP to span across IB subnets. We are
                stretching stretching here since such implementations
                are not possible yet.
                However, we can define it such that it will work in
                the future. Therefore include the GID.

        GUID

                Can be made part of the GID.


        Why the QPN:

                1) There is only one other way to get the QPN. That is
                the SIDR protocol. However, this has a drawback.

                SIDR is NOT REQUIRED by IBTA.

                I had raised this point last August 'Feasibility of
                SIDR':

                ***** Section 12.11 essentially states that SIDR is
                optional. As the specification states:

               << GSAs shall support this protocol if non-management
               services are pro-vided on the Channel Adapter at other
               than fixed QPNs. If this protocol is *not supported*,
               the status "The method/attribute combination is not
               sup-ported" shall be returned, as described in section
               Table 101 MAD Common Status Field Bit Values >>

                *****

                That still holds. SIDR is NOT required by IBTA. A
                device can always choose to provide all non-management
                services at fixed QPNs. It may just choose to provide
                IP and nothing else at a particular QP and hence won't
                do SIDR.

                That is why I propose that the QPN be always included
                for the default case. SIDR usage is gated through the
                QPN flag and is in the realm of the advanced
                capabilities draft as has been suggested by many on
                this mailing list. And I agree.


                2) Address resolution that returns the QPN is done in
                one request/reply exchange. Any other method, while
                talking to the same endpoint, requires 6 messages. 2
                for ARP/ND and 2 each for each of the lower layer
                request/reply. IT is wasteful, more complicated,
                more time consuming wrt fast failover etc.


        Capability flags :

                Multiple people have opined that they would prefer in
                certain cases to map the protocol/ULP endpoints to
                specific QPs. People also want to use IP over RC mode
                (and others maybe) due to the advantages of larger MTU
                and Automatic path failover etc.

                The capability flags allow for interoperation between
                the default/dumb devices and the more sophisticated
                devices. IT also allows the endpoints to avoid
                unnecessary trial and error/to and fro when trying to
                utilise these features. The flag indicates the
                advanced capabilities supported and the receiver can
                choose to utilies them.


Result:

        We cannot depend on SIDR. We need to include the default QPN.
        We should not force implementations to go to the SA/SM (IBTA
        does not) and so allow LID/GID to be determined from the
        remote host. The default case works with :

        LID+Capability flags(all set to 0)+QPN+GID


Vivek




_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 00:16:08 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA04757
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 00:16:08 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id AAA11695
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 00:15:49 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id AAA11620;
	Tue, 15 Jan 2002 00:13:21 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id AAA11572
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 00:13:18 -0500 (EST)
Received: from mxout1.netvision.net.il (mxout1.netvision.net.il [194.90.9.20])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA04736
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 00:13:16 -0500 (EST)
Received: from vaio505he ([32.101.212.35]) by mxout1.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GPY00A3FRT14A@mxout1.netvision.net.il> for ipoverib@ietf.org;
 Tue, 15 Jan 2002 07:12:46 +0200 (IST)
Date: Tue, 15 Jan 2002 07:13:20 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
In-reply-to: <3C43B187.9C2BB72C@sanera.net>
To: "'Joe Pelissier'" <joep@sanera.net>, ipoverib@ietf.org
Reply-to: yaronh@voltaire.com
Message-id: <015201c19d83$592702d0$0100007f@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7BIT

Joe, welcome back to the discussion
Regarding the IPD (static rate control) issue you mentioned
I don't see any host managing to fill a 1X pipe doing IPoIB (the TCP stack
would die much sooner); those bandwidths are for native IB protocols such as
SDP and SRP, setting it to 1X as a default will probably wont create any
problem

Another issue around it, currently the LWG is discussing using the IPD for
congestion avoidance with a thinner granularity that will be dynamically
adjusted (through FECN's, MAD's)

Yaron
-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Joe Pelissier
Sent: Tuesday, January 15, 2002 6:35 AM
To: ipoverib@ietf.org
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?

Greetings:
I unfortunately have had to reduce my participation in this effort for a
while,
but have now been studying the email threads attempting to get back up to
speed.
My apologies in advance if my comments below are missing some context from
previous disucssions.

It seems that there continues to be a great deal of discussion as to whether
ARP
can be accomplished without consulting the SA (or at least cached
information
from a previous consultation).  There seems to be two major issues: static
rate
and service level.

All nodes that operate at greater than x1 speed are required to support
static
rate.  The static rate setting is a function of the path between two
endpoints.
It is important to note that the path multicast uses is very likely to be
quite
different than the path that unicast traffic uses.  Therefore, the static
rate
may well be different.  The only way to get this information is to consult
the SA
(actually, one could always set their static rate to x1 speed and avoid the
consultation, but if you got a x12 link, it seems a shame not to use it just
to
avoid talking to your friendly SA ;-).

The other is SL.  It has repeatedly been argued that the spec does not
require
getting the SL from the SA.  This is true.  This has been recognized by the
IBTA
and steps are being taken to correct it.  In order to support multiple paths
through the IB fabric, it was a fundamental assumption that the SA controls
which
SLs a node may use based on path.  The SA may provide a set of SLs from
which a
node may choose, but the SA needs to control this.  If it does not, deadlock
of
the fabric may occur.  To prevent deadlock, the SA can load certain IB
tables
that would cause the offending packets to be discarded.  This protects the
fabric, but would cause communication to fail between the offending nodes.

So we could argue all this forever, or maybe there is another solution.  The
IB
spec already talks about how to determine both static rate and SL.
Therefore, we
don't need to rehash it in the RFC (other than maybe a note that says these
are
determined IAW the InfiniBand standard).

wrt/ SL, this is pretty much what section 6.0 of
draft-kashyap-ipoib-ipv4-and-arp-01.txt says, except for the last paragraph
(which recommends using the SL from the broadcast group).  Unfortunately,
the
recommendation is directly contrary to the some intended usage of IB and is
likely to either be prohibited or at least strongly discouraged by a future
version of the specification.

The LWG voted unanimously to recommend the addition of language that
strongly
discourages the use of parameters that were not obtained by an applicable
path
record and stating that the parameters from multicast and unicast should not
be
mixed (see the LWG meeting minutes dated 8/16/01).

If we just remove the recommendation from the draft, then we can move
forward
without further discussion.  It just seems bazaar to continue to push
something
as "recommended" that causes great concern to many of the architects that
created
IB.  All we need to do is leave out the recommendation and we are done with
this
issue.  Those that wish to ignore the SL assignment from the SA and use the
multicast one can still do so without technically violating the current IBTA

spec.  There seems to be little downside to this.

Besides, you need to get static rate anyway....

Another $0.02 worth...
Regards,
Joe Pelissier

Michael Krause wrote:

> At 05:10 PM 1/14/2002 -0600, Ashwin Kamath wrote:
> >There is a general thought that ARP is a one step solution for obtaining
the
> >parameters required for establishing IPoIB over UD. But I think that even
> >for using ARP, one will require to get the Path record information. I do
not
> >think that ARP is a one stop solution. What happens if both nodes
determine
> >that their MTU is 2K without considering that a switch component in
between
> >might restrict the MTU to 256 bytes?
>
> The minimum supported MTU for UD traffic is 2KB so a path with a 256 bytes
> MTU should not be used by the IP over IB service.  The SM working with the
> partition manager will establish a set of paths that can be used for IP
> over IB traffic which will include the setting up of an all nodes
multicast
> group per partition that support IP traffic.
>
> >Such information can be obtained only
> >from the central SM via a path record. Am I missing something here? If
> >address resolution is all that ARP is required to do, without trying to
> >establish parameters for communication one may not want to know the MTU,
but
> >from the discussions I see that the Address resolution is being used for
> >establishing communication parameters (like the QPN)
>
> QP identification is an optimization that allows one to eliminate the
> Ethertype field.  It also allows an implementation to map different IP
> addresses to different QP thereby providing an implementation optimization
> through more fine grain hardware-based de-multiplexing.  An example use
> might be to assign an IP address per OS instance which shares a HCA port
> and then allow all traffic to target a corresponding QP without having to
> flow through an intermediate hypervisor that would have to de-multiplex
the
> traffic.  This can provide both performance and scalability improvements
> for vendor offerings.
>
> >So the only real problem with having the LID in the ARP response, that I
see
> >is when the ARP response needs to cross IB Routers. Can I assume that
> >changes in LID assignments are not a big problem because such changes are
> >not intended to be frequent and current ARP cache implementations already
> >factor in the possibility of a link level address change. Changing LIDs
> >would have a bigger impact on shared devices using RC than on IPoIB using
> >UD. Is it ok if I consider a change in LID assignment to be equivalent of
a
> >server reboot?
>
> No.  Applications should not be impacted by a change in LID assignment as
> there are defined methods to migrate applications from one LID or GID or
> port to another that are completely transparent to the application.  For
> UD-based services, one could simply re-ARP the endnode via the all-nodes
> multicast and provide a simple update of the LID mapping as this can be
> derived from the IB headers if desired.
>
> >If I understood what Jerry, one will have a problem if the ARP responder
> >lies in another IB Subnet, in which case the LID in the response will not
be
> >relavant to the originator of the ARP request, in-fact the LID will have
to
> >be the LID of the router. Was there any resolution reached for this
problem?
> >
> >In short you do need an ARP + Path record diagnosis to determine all the
> >parameters for communication. What would happen if we bring an IB router
in
> >between? One cannot obtain a path record to a node in another subnet. One
> >can obtain the path record to the IB Router. Should we be talking about
the
> >behaviour of the IB router in case of mismatch of MTUs or should we say
that
> >we will always use the smallest mtu for communicating across an IB
router.
> >Or should we say that this could fall on the plate of the Infiniband WGs.
>
> The IB router protocol will define how one routes across IB subnets - an
> IBTA issue.  An IP router may be implemented along side an IB router or it
> may be an edge device such as a multi-protocol router.  The fact that IP
> subnets may span multiple IB subnets isn't really that big of an issue per
> se.  By keeping the LID not part of the link address structure, the ARP
> should be able to transparently flow across multiple IB subnets.
>
> Mike
>
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 01:23:05 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA05607
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 01:23:04 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id BAA22070
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 01:23:05 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id BAA22053;
	Tue, 15 Jan 2002 01:22:04 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id BAA22023
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 01:22:03 -0500 (EST)
Received: from pdxpo.dsl-only.net (pdxpo.dsl-only.net [63.105.16.3])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA05594
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 01:22:01 -0500 (EST)
Received: from bonfire (unverified [63.105.18.201]) by pdxpo.dsl-only.net
 (Rockliffe SMTPRA 4.5.4) with SMTP id <B0009159506@pdxpo.dsl-only.net>;
 Mon, 14 Jan 2002 22:21:09 -0800
From: "Bill Strahm" <bill@strahm.net>
To: <yaronh@voltaire.com>, "'Joe Pelissier'" <joep@sanera.net>,
        <ipoverib@ietf.org>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
Date: Mon, 14 Jan 2002 22:30:15 -0800
Message-ID: <HHEDIJGOEFLPCPNILMPAMENOCEAA.bill@strahm.net>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0)
Importance: Normal
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
In-Reply-To: <015201c19d83$592702d0$0100007f@vaio505he>
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Why do you say that... Current stacks exist that can push TCP traffic well
above 1 Gbit, even with standard hardware (1Ghz proc, standard nic etc)
Intel was showing TCP running at around 800 Mbit.  I go with a TOE and all
the sudden my throughput becomes limited by the bus bandwidth, guess what
happens when someone puts a 12x connection in the chipset running at full
throttle...

I don't think we want to limit ourselves to 1x just because that is all
people are thinking about today.

Bill

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Yaron Haviv
Sent: Monday, January 14, 2002 9:13 PM
To: 'Joe Pelissier'; ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?


Joe, welcome back to the discussion
Regarding the IPD (static rate control) issue you mentioned
I don't see any host managing to fill a 1X pipe doing IPoIB (the TCP stack
would die much sooner); those bandwidths are for native IB protocols such as
SDP and SRP, setting it to 1X as a default will probably wont create any
problem

Another issue around it, currently the LWG is discussing using the IPD for
congestion avoidance with a thinner granularity that will be dynamically
adjusted (through FECN's, MAD's)

Yaron
-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Joe Pelissier
Sent: Tuesday, January 15, 2002 6:35 AM
To: ipoverib@ietf.org
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?

Greetings:
I unfortunately have had to reduce my participation in this effort for a
while,
but have now been studying the email threads attempting to get back up to
speed.
My apologies in advance if my comments below are missing some context from
previous disucssions.

It seems that there continues to be a great deal of discussion as to whether
ARP
can be accomplished without consulting the SA (or at least cached
information
from a previous consultation).  There seems to be two major issues: static
rate
and service level.

All nodes that operate at greater than x1 speed are required to support
static
rate.  The static rate setting is a function of the path between two
endpoints.
It is important to note that the path multicast uses is very likely to be
quite
different than the path that unicast traffic uses.  Therefore, the static
rate
may well be different.  The only way to get this information is to consult
the SA
(actually, one could always set their static rate to x1 speed and avoid the
consultation, but if you got a x12 link, it seems a shame not to use it just
to
avoid talking to your friendly SA ;-).

The other is SL.  It has repeatedly been argued that the spec does not
require
getting the SL from the SA.  This is true.  This has been recognized by the
IBTA
and steps are being taken to correct it.  In order to support multiple paths
through the IB fabric, it was a fundamental assumption that the SA controls
which
SLs a node may use based on path.  The SA may provide a set of SLs from
which a
node may choose, but the SA needs to control this.  If it does not, deadlock
of
the fabric may occur.  To prevent deadlock, the SA can load certain IB
tables
that would cause the offending packets to be discarded.  This protects the
fabric, but would cause communication to fail between the offending nodes.

So we could argue all this forever, or maybe there is another solution.  The
IB
spec already talks about how to determine both static rate and SL.
Therefore, we
don't need to rehash it in the RFC (other than maybe a note that says these
are
determined IAW the InfiniBand standard).

wrt/ SL, this is pretty much what section 6.0 of
draft-kashyap-ipoib-ipv4-and-arp-01.txt says, except for the last paragraph
(which recommends using the SL from the broadcast group).  Unfortunately,
the
recommendation is directly contrary to the some intended usage of IB and is
likely to either be prohibited or at least strongly discouraged by a future
version of the specification.

The LWG voted unanimously to recommend the addition of language that
strongly
discourages the use of parameters that were not obtained by an applicable
path
record and stating that the parameters from multicast and unicast should not
be
mixed (see the LWG meeting minutes dated 8/16/01).

If we just remove the recommendation from the draft, then we can move
forward
without further discussion.  It just seems bazaar to continue to push
something
as "recommended" that causes great concern to many of the architects that
created
IB.  All we need to do is leave out the recommendation and we are done with
this
issue.  Those that wish to ignore the SL assignment from the SA and use the
multicast one can still do so without technically violating the current IBTA

spec.  There seems to be little downside to this.

Besides, you need to get static rate anyway....

Another $0.02 worth...
Regards,
Joe Pelissier

Michael Krause wrote:

> At 05:10 PM 1/14/2002 -0600, Ashwin Kamath wrote:
> >There is a general thought that ARP is a one step solution for obtaining
the
> >parameters required for establishing IPoIB over UD. But I think that even
> >for using ARP, one will require to get the Path record information. I do
not
> >think that ARP is a one stop solution. What happens if both nodes
determine
> >that their MTU is 2K without considering that a switch component in
between
> >might restrict the MTU to 256 bytes?
>
> The minimum supported MTU for UD traffic is 2KB so a path with a 256 bytes
> MTU should not be used by the IP over IB service.  The SM working with the
> partition manager will establish a set of paths that can be used for IP
> over IB traffic which will include the setting up of an all nodes
multicast
> group per partition that support IP traffic.
>
> >Such information can be obtained only
> >from the central SM via a path record. Am I missing something here? If
> >address resolution is all that ARP is required to do, without trying to
> >establish parameters for communication one may not want to know the MTU,
but
> >from the discussions I see that the Address resolution is being used for
> >establishing communication parameters (like the QPN)
>
> QP identification is an optimization that allows one to eliminate the
> Ethertype field.  It also allows an implementation to map different IP
> addresses to different QP thereby providing an implementation optimization
> through more fine grain hardware-based de-multiplexing.  An example use
> might be to assign an IP address per OS instance which shares a HCA port
> and then allow all traffic to target a corresponding QP without having to
> flow through an intermediate hypervisor that would have to de-multiplex
the
> traffic.  This can provide both performance and scalability improvements
> for vendor offerings.
>
> >So the only real problem with having the LID in the ARP response, that I
see
> >is when the ARP response needs to cross IB Routers. Can I assume that
> >changes in LID assignments are not a big problem because such changes are
> >not intended to be frequent and current ARP cache implementations already
> >factor in the possibility of a link level address change. Changing LIDs
> >would have a bigger impact on shared devices using RC than on IPoIB using
> >UD. Is it ok if I consider a change in LID assignment to be equivalent of
a
> >server reboot?
>
> No.  Applications should not be impacted by a change in LID assignment as
> there are defined methods to migrate applications from one LID or GID or
> port to another that are completely transparent to the application.  For
> UD-based services, one could simply re-ARP the endnode via the all-nodes
> multicast and provide a simple update of the LID mapping as this can be
> derived from the IB headers if desired.
>
> >If I understood what Jerry, one will have a problem if the ARP responder
> >lies in another IB Subnet, in which case the LID in the response will not
be
> >relavant to the originator of the ARP request, in-fact the LID will have
to
> >be the LID of the router. Was there any resolution reached for this
problem?
> >
> >In short you do need an ARP + Path record diagnosis to determine all the
> >parameters for communication. What would happen if we bring an IB router
in
> >between? One cannot obtain a path record to a node in another subnet. One
> >can obtain the path record to the IB Router. Should we be talking about
the
> >behaviour of the IB router in case of mismatch of MTUs or should we say
that
> >we will always use the smallest mtu for communicating across an IB
router.
> >Or should we say that this could fall on the plate of the Infiniband WGs.
>
> The IB router protocol will define how one routes across IB subnets - an
> IBTA issue.  An IP router may be implemented along side an IB router or it
> may be an edge device such as a multi-protocol router.  The fact that IP
> subnets may span multiple IB subnets isn't really that big of an issue per
> se.  By keeping the LID not part of the link address structure, the ARP
> should be able to transparently flow across multiple IB subnets.
>
> Mike
>
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 01:36:33 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA05742
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 01:36:32 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id BAA22629
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 01:36:33 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id BAA22597;
	Tue, 15 Jan 2002 01:35:05 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id BAA22566
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 01:35:04 -0500 (EST)
Received: from mxout1.netvision.net.il (mxout1.netvision.net.il [194.90.9.20])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA05733
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 01:35:01 -0500 (EST)
Received: from vaio505he ([32.100.203.229]) by mxout1.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GPY00AF8VL9CB@mxout1.netvision.net.il> for ipoverib@ietf.org;
 Tue, 15 Jan 2002 08:34:31 +0200 (IST)
Date: Tue, 15 Jan 2002 08:35:01 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
In-reply-to: <HHEDIJGOEFLPCPNILMPAMENOCEAA.bill@strahm.net>
To: "'Bill Strahm'" <bill@strahm.net>, ipoverib@ietf.org
Reply-to: yaronh@voltaire.com
Message-id: <017501c19d8e$c366b6d0$0100007f@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7BIT

I would be happy to see any benchmark running TCP in 1Gbit without any form
of offload, and regular MTU
The ones that you see are using special techniques, and are in ideal cases
And they consume the entire CPU processing power
(1X is also 2Gbit)

I heard a wise man saying once that if eventually TCP/IP traffic will
dominant IB fabric we probably did something wrong here
And could have stayed with Ethernet

If you would insist on killing your CPU exceeding the 2Gbit, you could still
do the PathRecord query
My notion is that the IPoIB will be used to initialize the fabric, low
bandwidth applications
The data movement applications will be using RDMA (the IP guys already got
it and now imitating IB stuff, see iWarp)

Yaron

-----Original Message-----
From: Bill Strahm [mailto:bill@strahm.net]
Sent: Tuesday, January 15, 2002 8:30 AM
To: yaronh@voltaire.com; 'Joe Pelissier'; ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?

Why do you say that... Current stacks exist that can push TCP traffic well
above 1 Gbit, even with standard hardware (1Ghz proc, standard nic etc)
Intel was showing TCP running at around 800 Mbit.  I go with a TOE and all
the sudden my throughput becomes limited by the bus bandwidth, guess what
happens when someone puts a 12x connection in the chipset running at full
throttle...

I don't think we want to limit ourselves to 1x just because that is all
people are thinking about today.

Bill

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Yaron Haviv
Sent: Monday, January 14, 2002 9:13 PM
To: 'Joe Pelissier'; ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?


Joe, welcome back to the discussion
Regarding the IPD (static rate control) issue you mentioned
I don't see any host managing to fill a 1X pipe doing IPoIB (the TCP stack
would die much sooner); those bandwidths are for native IB protocols such as
SDP and SRP, setting it to 1X as a default will probably wont create any
problem

Another issue around it, currently the LWG is discussing using the IPD for
congestion avoidance with a thinner granularity that will be dynamically
adjusted (through FECN's, MAD's)

Yaron
-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Joe Pelissier
Sent: Tuesday, January 15, 2002 6:35 AM
To: ipoverib@ietf.org
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?

Greetings:
I unfortunately have had to reduce my participation in this effort for a
while,
but have now been studying the email threads attempting to get back up to
speed.
My apologies in advance if my comments below are missing some context from
previous disucssions.

It seems that there continues to be a great deal of discussion as to whether
ARP
can be accomplished without consulting the SA (or at least cached
information
from a previous consultation).  There seems to be two major issues: static
rate
and service level.

All nodes that operate at greater than x1 speed are required to support
static
rate.  The static rate setting is a function of the path between two
endpoints.
It is important to note that the path multicast uses is very likely to be
quite
different than the path that unicast traffic uses.  Therefore, the static
rate
may well be different.  The only way to get this information is to consult
the SA
(actually, one could always set their static rate to x1 speed and avoid the
consultation, but if you got a x12 link, it seems a shame not to use it just
to
avoid talking to your friendly SA ;-).

The other is SL.  It has repeatedly been argued that the spec does not
require
getting the SL from the SA.  This is true.  This has been recognized by the
IBTA
and steps are being taken to correct it.  In order to support multiple paths
through the IB fabric, it was a fundamental assumption that the SA controls
which
SLs a node may use based on path.  The SA may provide a set of SLs from
which a
node may choose, but the SA needs to control this.  If it does not, deadlock
of
the fabric may occur.  To prevent deadlock, the SA can load certain IB
tables
that would cause the offending packets to be discarded.  This protects the
fabric, but would cause communication to fail between the offending nodes.

So we could argue all this forever, or maybe there is another solution.  The
IB
spec already talks about how to determine both static rate and SL.
Therefore, we
don't need to rehash it in the RFC (other than maybe a note that says these
are
determined IAW the InfiniBand standard).

wrt/ SL, this is pretty much what section 6.0 of
draft-kashyap-ipoib-ipv4-and-arp-01.txt says, except for the last paragraph
(which recommends using the SL from the broadcast group).  Unfortunately,
the
recommendation is directly contrary to the some intended usage of IB and is
likely to either be prohibited or at least strongly discouraged by a future
version of the specification.

The LWG voted unanimously to recommend the addition of language that
strongly
discourages the use of parameters that were not obtained by an applicable
path
record and stating that the parameters from multicast and unicast should not
be
mixed (see the LWG meeting minutes dated 8/16/01).

If we just remove the recommendation from the draft, then we can move
forward
without further discussion.  It just seems bazaar to continue to push
something
as "recommended" that causes great concern to many of the architects that
created
IB.  All we need to do is leave out the recommendation and we are done with
this
issue.  Those that wish to ignore the SL assignment from the SA and use the
multicast one can still do so without technically violating the current IBTA

spec.  There seems to be little downside to this.

Besides, you need to get static rate anyway....

Another $0.02 worth...
Regards,
Joe Pelissier

Michael Krause wrote:

> At 05:10 PM 1/14/2002 -0600, Ashwin Kamath wrote:
> >There is a general thought that ARP is a one step solution for obtaining
the
> >parameters required for establishing IPoIB over UD. But I think that even
> >for using ARP, one will require to get the Path record information. I do
not
> >think that ARP is a one stop solution. What happens if both nodes
determine
> >that their MTU is 2K without considering that a switch component in
between
> >might restrict the MTU to 256 bytes?
>
> The minimum supported MTU for UD traffic is 2KB so a path with a 256 bytes
> MTU should not be used by the IP over IB service.  The SM working with the
> partition manager will establish a set of paths that can be used for IP
> over IB traffic which will include the setting up of an all nodes
multicast
> group per partition that support IP traffic.
>
> >Such information can be obtained only
> >from the central SM via a path record. Am I missing something here? If
> >address resolution is all that ARP is required to do, without trying to
> >establish parameters for communication one may not want to know the MTU,
but
> >from the discussions I see that the Address resolution is being used for
> >establishing communication parameters (like the QPN)
>
> QP identification is an optimization that allows one to eliminate the
> Ethertype field.  It also allows an implementation to map different IP
> addresses to different QP thereby providing an implementation optimization
> through more fine grain hardware-based de-multiplexing.  An example use
> might be to assign an IP address per OS instance which shares a HCA port
> and then allow all traffic to target a corresponding QP without having to
> flow through an intermediate hypervisor that would have to de-multiplex
the
> traffic.  This can provide both performance and scalability improvements
> for vendor offerings.
>
> >So the only real problem with having the LID in the ARP response, that I
see
> >is when the ARP response needs to cross IB Routers. Can I assume that
> >changes in LID assignments are not a big problem because such changes are
> >not intended to be frequent and current ARP cache implementations already
> >factor in the possibility of a link level address change. Changing LIDs
> >would have a bigger impact on shared devices using RC than on IPoIB using
> >UD. Is it ok if I consider a change in LID assignment to be equivalent of
a
> >server reboot?
>
> No.  Applications should not be impacted by a change in LID assignment as
> there are defined methods to migrate applications from one LID or GID or
> port to another that are completely transparent to the application.  For
> UD-based services, one could simply re-ARP the endnode via the all-nodes
> multicast and provide a simple update of the LID mapping as this can be
> derived from the IB headers if desired.
>
> >If I understood what Jerry, one will have a problem if the ARP responder
> >lies in another IB Subnet, in which case the LID in the response will not
be
> >relavant to the originator of the ARP request, in-fact the LID will have
to
> >be the LID of the router. Was there any resolution reached for this
problem?
> >
> >In short you do need an ARP + Path record diagnosis to determine all the
> >parameters for communication. What would happen if we bring an IB router
in
> >between? One cannot obtain a path record to a node in another subnet. One
> >can obtain the path record to the IB Router. Should we be talking about
the
> >behaviour of the IB router in case of mismatch of MTUs or should we say
that
> >we will always use the smallest mtu for communicating across an IB
router.
> >Or should we say that this could fall on the plate of the Infiniband WGs.
>
> The IB router protocol will define how one routes across IB subnets - an
> IBTA issue.  An IP router may be implemented along side an IB router or it
> may be an edge device such as a multi-protocol router.  The fact that IP
> subnets may span multiple IB subnets isn't really that big of an issue per
> se.  By keeping the LID not part of the link address structure, the ARP
> should be able to transparently flow across multiple IB subnets.
>
> Mike
>
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 03:16:22 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA15131
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 03:16:17 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id DAA25282
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 03:16:19 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id DAA25239;
	Tue, 15 Jan 2002 03:15:07 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id DAA25188
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 03:15:04 -0500 (EST)
Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA15112
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 03:15:01 -0500 (EST)
Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23])
	by e31.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id DAA302386
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 03:12:02 -0500
Received: from d03nm036.boulder.ibm.com (d03nm036.boulder.ibm.com [9.99.140.36])
	by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0F8F2F105232
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 01:15:02 -0700
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
To: ipoverib@ietf.org
X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000
Message-ID: <OFCA25676A.DCAD4019-ON88256B42.001DFDD0@boulder.ibm.com>
From: "Vivek Kashyap" <vivk@us.ibm.com>
Date: Tue, 15 Jan 2002 00:12:16 -0800
X-MIMETrack: Serialize by Router on D03NM036/03/M/IBM(Release 5.0.9 |November 16, 2001) at
 01/15/2002 01:15:01 AM
MIME-Version: 1.0
Content-type: text/plain; charset=us-ascii
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org


Greetings Joe,

      A couple of clarifications are in order.

      The idea of avoiding the SM is a very valid one - I wonder what all
this SM/SA will do ? Everything on the CA will go over the same QPs(0/1)
and will go over VL15 (which doesn't have congestion control, looses
packets etc.), will go the SM/SA at its QP0/1 in a large SM. The SM will be
consulted for path records, for service id resolutions, for this and that
and that also. It seems a bit overmuch. That is an aside anyway for one to
think about.

      There are other reasons for avoiding the SM/SA that I  have mentioned
previously. However, nowhere has it been suggested that SM/SA cannot be
consulted. The counter-suggestion from you though says that one must choose
to go to  SM/SA - that is something that IBTA itself doesn't agree with.
Those that want to consult the SM/SA can continue to do so but those that
don't want to because they certainly see advantages in avoiding SM/SA when
possible should be allowed to do so; exactly as IBTA does.

      Regarding your comments on SL - I am a member of IBTA. IBTA rules do
not allow to one to give out the statements and the voting results  to
non-IBTA forums, so I'll only state that IBTA did not change the
specification - it is available to anyone on www.infinibandta.org  and did
not modify the rules regarding SL determination as Mike Krause in the mail
before yours has clarified. Mike is also a member(chair) of LWG.  Others
who are members of the IBTA can verify for themselves the final decision of
LWG. It did not preclude determining SL from sources other than the SA.

      You are right in stating that protection from a random SL is required
but no where is it being suggested that the SL used by the endnodes will be
random one. It will be a well thought of one and a correct one except it
will be determined without consulting the SA/SM. Nothing in IBTA precludes
it. Rather IBA specification promotes it.  BTW, the use SL from multicast
or determination of it  in any other way has been left to the choice of the
implementor. Not part of IPoIB architecture. Also, again, IBTA does not
specify a particular method either.

      All nodes if working greater than 1x are required to support static
rate iff any destination in the fabric works at < the node's rate.  It
merely says that the values needs to be determined but does not make the
method for determining it mandatory. Consulting the SA to get the path
record is one of the methods. Additionally IBTA is defining a congestion
control mechanism disjoint from injection control.

      An example:  do you mean to say that if I know that my subnet is made
of 4x through and through I need to consult the SA to get the information
always ? I don't think so.

      The suggestion here that some implementations will avoid the SA/SM in
a particular subnet if they think it is right in no way precludes
consulting the SA/SM or any other entity when required. That is not the
contention at all.  It is not being mandated on everyone. Similarly there
is no need to require every implementation to throw a look toward the SA/SM
for SL/path rate or any other information.

Vivek

--
Vivek Kashyap
Linux Technology Center, IBM
vivk@us.ibm.com
kashyapv@us.ibm.com
Ph: 503 578 3422 T/L: 775 3422



                                                                                                                                  
                      Joe Pelissier                                                                                               
                      <joep@Sanera.net>        To:       ipoverib@ietf.org                                                        
                      Sent by:                 cc:                                                                                
                      ipoverib-admin@ie        Subject:  Re: [Ipoverib] What's inside an IPoIB link layer address?                
                      tf.org                                                                                                      
                                                                                                                                  
                                                                                                                                  
                      01/14/2002 08:35                                                                                            
                      PM                                                                                                          
                                                                                                                                  
                                                                                                                                  



Greetings:
I unfortunately have had to reduce my participation in this effort for a
while,
but have now been studying the email threads attempting to get back up to
speed.
My apologies in advance if my comments below are missing some context from
previous disucssions.

It seems that there continues to be a great deal of discussion as to
whether ARP
can be accomplished without consulting the SA (or at least cached
information
from a previous consultation).  There seems to be two major issues: static
rate
and service level.

All nodes that operate at greater than x1 speed are required to support
static
rate.  The static rate setting is a function of the path between two
endpoints.
It is important to note that the path multicast uses is very likely to be
quite
different than the path that unicast traffic uses.  Therefore, the static
rate
may well be different.  The only way to get this information is to consult
the SA
(actually, one could always set their static rate to x1 speed and avoid the
consultation, but if you got a x12 link, it seems a shame not to use it
just to
avoid talking to your friendly SA ;-).

The other is SL.  It has repeatedly been argued that the spec does not
require
getting the SL from the SA.  This is true.  This has been recognized by the
IBTA
and steps are being taken to correct it.  In order to support multiple
paths
through the IB fabric, it was a fundamental assumption that the SA controls
which
SLs a node may use based on path.  The SA may provide a set of SLs from
which a
node may choose, but the SA needs to control this.  If it does not,
deadlock of
the fabric may occur.  To prevent deadlock, the SA can load certain IB
tables
that would cause the offending packets to be discarded.  This protects the
fabric, but would cause communication to fail between the offending nodes.

So we could argue all this forever, or maybe there is another solution.
The IB
spec already talks about how to determine both static rate and SL.
Therefore, we
don't need to rehash it in the RFC (other than maybe a note that says these
are
determined IAW the InfiniBand standard).

wrt/ SL, this is pretty much what section 6.0 of
draft-kashyap-ipoib-ipv4-and-arp-01.txt says, except for the last paragraph
(which recommends using the SL from the broadcast group).  Unfortunately,
the
recommendation is directly contrary to the some intended usage of IB and is
likely to either be prohibited or at least strongly discouraged by a future
version of the specification.

The LWG voted unanimously to recommend the addition of language that
strongly
discourages the use of parameters that were not obtained by an applicable
path
record and stating that the parameters from multicast and unicast should
not be
mixed (see the LWG meeting minutes dated 8/16/01).

If we just remove the recommendation from the draft, then we can move
forward
without further discussion.  It just seems bazaar to continue to push
something
as "recommended" that causes great concern to many of the architects that
created
IB.  All we need to do is leave out the recommendation and we are done with
this
issue.  Those that wish to ignore the SL assignment from the SA and use the
multicast one can still do so without technically violating the current
IBTA
spec.  There seems to be little downside to this.

Besides, you need to get static rate anyway....

Another $0.02 worth...
Regards,
Joe Pelissier

Michael Krause wrote:

> At 05:10 PM 1/14/2002 -0600, Ashwin Kamath wrote:
> >There is a general thought that ARP is a one step solution for obtaining
the
> >parameters required for establishing IPoIB over UD. But I think that
even
> >for using ARP, one will require to get the Path record information. I do
not
> >think that ARP is a one stop solution. What happens if both nodes
determine
> >that their MTU is 2K without considering that a switch component in
between
> >might restrict the MTU to 256 bytes?
>
> The minimum supported MTU for UD traffic is 2KB so a path with a 256
bytes
> MTU should not be used by the IP over IB service.  The SM working with
the
> partition manager will establish a set of paths that can be used for IP
> over IB traffic which will include the setting up of an all nodes
multicast
> group per partition that support IP traffic.
>
> >Such information can be obtained only
> >from the central SM via a path record. Am I missing something here? If
> >address resolution is all that ARP is required to do, without trying to
> >establish parameters for communication one may not want to know the MTU,
but
> >from the discussions I see that the Address resolution is being used for
> >establishing communication parameters (like the QPN)
>
> QP identification is an optimization that allows one to eliminate the
> Ethertype field.  It also allows an implementation to map different IP
> addresses to different QP thereby providing an implementation
optimization
> through more fine grain hardware-based de-multiplexing.  An example use
> might be to assign an IP address per OS instance which shares a HCA port
> and then allow all traffic to target a corresponding QP without having to
> flow through an intermediate hypervisor that would have to de-multiplex
the
> traffic.  This can provide both performance and scalability improvements
> for vendor offerings.
>
> >So the only real problem with having the LID in the ARP response, that I
see
> >is when the ARP response needs to cross IB Routers. Can I assume that
> >changes in LID assignments are not a big problem because such changes
are
> >not intended to be frequent and current ARP cache implementations
already
> >factor in the possibility of a link level address change. Changing LIDs
> >would have a bigger impact on shared devices using RC than on IPoIB
using
> >UD. Is it ok if I consider a change in LID assignment to be equivalent
of a
> >server reboot?
>
> No.  Applications should not be impacted by a change in LID assignment as
> there are defined methods to migrate applications from one LID or GID or
> port to another that are completely transparent to the application.  For
> UD-based services, one could simply re-ARP the endnode via the all-nodes
> multicast and provide a simple update of the LID mapping as this can be
> derived from the IB headers if desired.
>
> >If I understood what Jerry, one will have a problem if the ARP responder
> >lies in another IB Subnet, in which case the LID in the response will
not be
> >relavant to the originator of the ARP request, in-fact the LID will have
to
> >be the LID of the router. Was there any resolution reached for this
problem?
> >
> >In short you do need an ARP + Path record diagnosis to determine all the
> >parameters for communication. What would happen if we bring an IB router
in
> >between? One cannot obtain a path record to a node in another subnet.
One
> >can obtain the path record to the IB Router. Should we be talking about
the
> >behaviour of the IB router in case of mismatch of MTUs or should we say
that
> >we will always use the smallest mtu for communicating across an IB
router.
> >Or should we say that this could fall on the plate of the Infiniband
WGs.
>
> The IB router protocol will define how one routes across IB subnets - an
> IBTA issue.  An IP router may be implemented along side an IB router or
it
> may be an edge device such as a multi-protocol router.  The fact that IP
> subnets may span multiple IB subnets isn't really that big of an issue
per
> se.  By keeping the LID not part of the link address structure, the ARP
> should be able to transparently flow across multiple IB subnets.
>
> Mike
>
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib





_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 12:26:56 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA06131
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 12:26:52 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id MAA14605
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 12:26:54 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA14566;
	Tue, 15 Jan 2002 12:25:55 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id GAA00859
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 06:26:48 -0500 (EST)
Received: from CNRI.Reston.VA.US (localhost [127.0.0.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA17572;
	Tue, 15 Jan 2002 06:26:46 -0500 (EST)
Message-Id: <200201151126.GAA17572@ietf.org>
Mime-Version: 1.0
Content-Type: Multipart/Mixed; Boundary="NextPart"
To: IETF-Announce: ;
Cc: ipoverib@ietf.org
From: Internet-Drafts@ietf.org
Reply-to: Internet-Drafts@ietf.org
Date: Tue, 15 Jan 2002 06:26:46 -0500
Subject: [Ipoverib] I-D ACTION:draft-ietf-ipoib-link-multicast-00.txt
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the IP over InfiniBand Working Group of the IETF.

	Title		: IP link and multicast over InfiniBand networks
	Author(s)	: J. Chu, V. Kashyap
	Filename	: draft-ietf-ipoib-link-multicast-00.txt
	Pages		: 13
	Date		: 14-Jan-02
	
This document specifies a method for setting up IP subnets and
multicast services over InfiniBand(TM) networks. Discussions in this
document are applicable to both IPv4 and IPv6, unless explicitly
specified. A separate document will cover unicast and encapsulation
of IP datagrams over InfiniBand networks.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-ipoib-link-multicast-00.txt

To remove yourself from the IETF Announcement list, send a message to 
ietf-announce-request with the word unsubscribe in the body of the message.

Internet-Drafts are also available by anonymous FTP. Login with the username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-ietf-ipoib-link-multicast-00.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt


Internet-Drafts can also be obtained by e-mail.

Send a message to:
	mailserv@ietf.org.
In the body type:
	"FILE /internet-drafts/draft-ietf-ipoib-link-multicast-00.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.
		
		
Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Multipart/Alternative; Boundary="OtherAccess"

--OtherAccess
Content-Type: Message/External-body;
	access-type="mail-server";
	server="mailserv@ietf.org"

Content-Type: text/plain
Content-ID:	<20020114174829.I-D@ietf.org>

ENCODING mime
FILE /internet-drafts/draft-ietf-ipoib-link-multicast-00.txt

--OtherAccess
Content-Type: Message/External-body;
	name="draft-ietf-ipoib-link-multicast-00.txt";
	site="ftp.ietf.org";
	access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID:	<20020114174829.I-D@ietf.org>

--OtherAccess--

--NextPart--




_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 12:44:21 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA06832
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 12:44:21 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id MAA15406
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 12:44:22 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA15303;
	Tue, 15 Jan 2002 12:41:11 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA15269
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 12:41:09 -0500 (EST)
Received: from rwcrmhc53.attbi.com (rwcrmhc53.attbi.com [204.127.198.39])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA06585
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 12:41:03 -0500 (EST)
Received: from spike ([12.224.213.32]) by rwcrmhc53.attbi.com
          (InterMail vM.4.01.03.27 201-229-121-127-20010626) with SMTP
          id <20020115174036.YWEZ10199.rwcrmhc53.attbi.com@spike>;
          Tue, 15 Jan 2002 17:40:36 +0000
From: "Bill Swortwood" <billiam33@attbi.com>
To: "Vivek Kashyap" <vivk@us.ibm.com>, <ipoverib@ietf.org>
Subject: Some annotation (was RE: [Ipoverib] What's inside an IPoIB link layer address?)
Date: Tue, 15 Jan 2002 09:41:50 -0800
Message-ID: <HFEHJGAKENECEAHPNJECAEDLCBAA.billiam33@attbi.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0)
Importance: Normal
In-Reply-To: <OFCA25676A.DCAD4019-ON88256B42.001DFDD0@boulder.ibm.com>
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Greetings IPoverIB subscribers -

A few conceptual clarifications. There have been a number of
misnomers here, not central to the main discussion, but these do
muddy the waters of understanding. These are marked with [whs] below:

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Vivek Kashyap
Sent: Tuesday, January 15, 2002 12:12 AM
To: ipoverib@ietf.org
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?



Greetings Joe,

      A couple of clarifications are in order.

      The idea of avoiding the SM is a very valid one - I wonder what all
this SM/SA will do ?

[whs] Set up the topology of the IB subnet, keep the subnet topology intact,
collect and disperse subnet event information, set up partitions within the
subnet (part of topology), provide path information within the subnet, act
as the subnet multicast setup and join agency, and act as a low level
service
registry.

Everything on the CA will go over the same QPs(0/1)
and will go over VL15 (which doesn't have congestion control, looses
packets etc.), will go the SM/SA at its QP0/1 in a large SM.

[whs] Only SMPs (Subnet Management Packets) go on VL15. GMPS (General
Management Packets) go on any other assigned VL that is not VL15. The
SM communications are those that set up and keep the topology within
a subnet intact, and are the controlled domain of the SM, for the most
part. GMP communications are used by entities wanting information --
like topology or path record information, from the SA. The SA can also
redirect its communications to any other QP/Port within the subnet, so
communications can be balanced or serviced at other locales within the
subnet. Point here - it is NOT all VL15, and NOT all QP0/1. Further,
SL and static rate information are specific to the path in question.
--

The SM will be
consulted for path records, for service id resolutions, for this and that
and that also. It seems a bit overmuch. That is an aside anyway for one to
think about.

[whs] Suggest you chat with Dr. Pfister [IBM] on this. PathRecords need to
come
from somewhere. Paths can change. SLs and static rate are particular to a
Path.

Service records
exist at the SA as it is the only well known address at the time any node
comes
up on an IB subnet. Note: if you hang a name service or directory service
as a service record with the SA, and that service should also contain all
information (like SL and static rate data), you would only have to
communicate
within the subnet to the SA when you need to establish a communication link
to that service.
--

      There are other reasons for avoiding the SM/SA that I  have mentioned
previously. However, nowhere has it been suggested that SM/SA cannot be
consulted. The counter-suggestion from you though says that one must choose
to go to  SM/SA - that is something that IBTA itself doesn't agree with.
Those that want to consult the SM/SA can continue to do so but those that
don't want to because they certainly see advantages in avoiding SM/SA when
possible should be allowed to do so; exactly as IBTA does.

[whs] I agree with you on keeping your SA communications to a minimum. Joe's
point is that the SM/SA is the origination point of path data. Some other
service can obtain that data from the SM/SA and outwardly provide it. I
am not, and would not think that Joe would be (correct me if I am wrong)
adverse to someone postulating such a service. But as things stand today
with the 1.0a spec, the SA now is the only defined interface.
--

---[snip]---

regards all -

-------------------------------
Bill Swortwood
(Office)(503) 690 8578
(CELL)  (503) 819-5281
--------------------------------


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 13:06:08 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA07495
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 13:06:08 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id NAA16405
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 13:06:09 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA16231;
	Tue, 15 Jan 2002 13:04:19 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA16173
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 13:04:17 -0500 (EST)
Received: from palrel13.hp.com (palrel13.hp.com [156.153.255.238])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA07426
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 13:04:15 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel13.hp.com (Postfix) with ESMTP id 73C16E00647
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 10:03:42 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id KAA23523
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 10:03:42 -0800 (PST)
Message-Id: <5.1.0.14.2.20020115091256.00ad74c0@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Tue, 15 Jan 2002 09:16:53 -0800
To: ipoverib@ietf.org
From: Michael Krause <krause@cup.hp.com>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
In-Reply-To: <015201c19d83$592702d0$0100007f@vaio505he>
References: <3C43B187.9C2BB72C@sanera.net>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

At 07:13 AM 1/15/2002 +0200, Yaron Haviv wrote:
>Joe, welcome back to the discussion
>Regarding the IPD (static rate control) issue you mentioned
>I don't see any host managing to fill a 1X pipe doing IPoIB (the TCP stack
>would die much sooner); those bandwidths are for native IB protocols such as
>SDP and SRP, setting it to 1X as a default will probably wont create any
>problem

I strongly disagree with the above assertion.  There are many servers that 
today can drive 1x at link rate; in fact, many that can drive 4x using TCP 
/ IP implemented in software - keep in mind that Moore's Law combined with 
new I/O chip interconnects such as PCI-X, PCI-X 2.0, and 3GIO one will have 
the processing power and I/O bandwidth to drive the very high bandwidths 
that are supported by IB fabrics.

>Another issue around it, currently the LWG is discussing using the IPD for
>congestion avoidance with a thinner granularity that will be dynamically
>adjusted (through FECN's, MAD's)

What the LWG is doing is separate from this effort as it is in another 
standards group.  The IB spec states any link that supports greater than 
2.5 Gbps must implement static rate control. This means that any IP over IB 
specification must not preclude or prescribe any implementation from using 
static rate control in compliance with the current IB specification.

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 13:06:54 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA07534
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 13:06:54 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id NAA16435
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 13:06:55 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA16284;
	Tue, 15 Jan 2002 13:04:22 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA16171
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 13:04:17 -0500 (EST)
Received: from palrel11.hp.com (palrel11.hp.com [156.153.255.246])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA07425
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 13:04:15 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel11.hp.com (Postfix) with ESMTP id 26F7EE0063E
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 10:03:43 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id KAA23532
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 10:03:42 -0800 (PST)
Message-Id: <5.1.0.14.2.20020115092648.029a1008@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Tue, 15 Jan 2002 09:28:16 -0800
To: <ipoverib@ietf.org>
From: Michael Krause <krause@cup.hp.com>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
In-Reply-To: <HHEDIJGOEFLPCPNILMPAMENOCEAA.bill@strahm.net>
References: <015201c19d83$592702d0$0100007f@vaio505he>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

At 10:30 PM 1/14/2002 -0800, Bill Strahm wrote:
>Why do you say that... Current stacks exist that can push TCP traffic well
>above 1 Gbit, even with standard hardware (1Ghz proc, standard nic etc)
>Intel was showing TCP running at around 800 Mbit.  I go with a TOE and all
>the sudden my throughput becomes limited by the bus bandwidth, guess what
>happens when someone puts a 12x connection in the chipset running at full
>throttle...

As another data point, HP has a PCI GbE NIC running full-duplex at link 
rate (i.e. over 1800 Mbps aggregate of application data - user-space to 
user-space) and over 200,000 packets per second both measured using netperf 
TCP_STREAM and TCP_RR respectively.  This bandwidth was achieved using 
standard Ethernet MTU and not protocol off-load sans checksum 
off-load.  Less than 1 CPU was used to drive this bandwidth.  BTW, we can 
drive 6 of these links on a 8-way SMP without consuming all of the CPU 
power.  Scaling is fairly linear within the TCP implementation and with the 
point-to-point non-shared PCI implementation, we don't have any of the I/O 
scaling problems found in many PCI implementations.

>I don't think we want to limit ourselves to 1x just because that is all
>people are thinking about today.

Agreed.

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 13:16:31 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA07903
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 13:16:26 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id NAA16752
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 13:16:28 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA16721;
	Tue, 15 Jan 2002 13:15:38 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA16685
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 13:15:36 -0500 (EST)
Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA07851
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 13:15:34 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id MAA61744;
	Tue, 15 Jan 2002 12:11:23 -0600
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0FIFYe38984;
	Tue, 15 Jan 2002 13:15:34 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0FIFWV16017;
	Tue, 15 Jan 2002 10:15:32 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0FIFVY03360;
	Tue, 15 Jan 2002 10:15:31 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201151815.g0FIFVY03360@eng4.beaverton.ibm.com>
Subject: Re: Some annotation (was RE: [Ipoverib] What's inside an IPoIB link layer address?)
To: billiam33@attbi.com (Bill Swortwood)
Date: Tue, 15 Jan 2002 10:15:30 -0800 (PST)
Cc: vivk@us.ibm.com (Vivek Kashyap), ipoverib@ietf.org
In-Reply-To: <HFEHJGAKENECEAHPNJECAEDLCBAA.billiam33@attbi.com> from "Bill Swortwood" at Jan 15, 2002 08:41:50 AM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

> 
>       The idea of avoiding the SM is a very valid one - I wonder what all
> this SM/SA will do ?
> 
> [whs] Set up the topology of the IB subnet, keep the subnet topology intact,
> collect and disperse subnet event information, set up partitions within the
> subnet (part of topology), provide path information within the subnet, act
> as the subnet multicast setup and join agency, and act as a low level
> service
> registry.
> 
> Everything on the CA will go over the same QPs(0/1)
> and will go over VL15 (which doesn't have congestion control, looses
> packets etc.), will go the SM/SA at its QP0/1 in a large SM.
> 
> [whs] Only SMPs (Subnet Management Packets) go on VL15. GMPS (General
> Management Packets) go on any other assigned VL that is not VL15. The
> SM communications are those that set up and keep the topology within
> a subnet intact, and are the controlled domain of the SM, for the most
> part. GMP communications are used by entities wanting information --
> like topology or path record information, from the SA. The SA can also
> redirect its communications to any other QP/Port within the subnet, so
> communications can be balanced or serviced at other locales within the
> subnet. Point here - it is NOT all VL15, and NOT all QP0/1. Further,
> SL and static rate information are specific to the path in question.
> --

        OK so there is some respite. However, how many paths lead to
        SA/SM in a subnet ? It still stands to reason if one wants to
        avoid it and can avoid it why force them ? 

        As noted earlier, SLs need not come from the SA. They of
        course will be recorded there in the path record but that does
        not imply that they can't be determined differently e.g. QoS
        mappings. 

        Similarly the path rate needs to be determined and will be
        recorded at the SA/SM. That does not mean that one has to go
        to the SA at the time of the address resolution to determine
        it. 
	
        I am arguing against the notion that because SL or path rate
        need to be known the LID need not be included in the address
        resolution. I don't see why even though based on your
        communication. 

> 
> The SM will be
> consulted for path records, for service id resolutions, for this and that
> and that also. It seems a bit overmuch. That is an aside anyway for one to
> think about.
> 
> [whs] Suggest you chat with Dr. Pfister [IBM] on this. PathRecords need to
> come
> from somewhere. Paths can change. SLs and static rate are particular to a
> Path.

        Please re-read what I have said. No where do I say that path
        records won't be recorded at the SA. Nor is it suggested that
        you preclude it. The discussion is different. It is not
        correct to use the argument that the path record is at the
        SA/SM and hence we need to get the LID from there.

        Inclusion of LID is an IP requirement. It needs to be
        determined at every address resolution. SL/path rate can be
        static in a subnet, can be determined differently including
        indirectly/directly from the SA.


> 
> Service records
> exist at the SA as it is the only well known address at the time any node
> comes
> up on an IB subnet. Note: if you hang a name service or directory service
> as a service record with the SA, and that service should also contain all
> information (like SL and static rate data), you would only have to
> communicate
> within the subnet to the SA when you need to establish a communication link
> to that service.
> --
> 
>       There are other reasons for avoiding the SM/SA that I  have mentioned
> previously. However, nowhere has it been suggested that SM/SA cannot be
> consulted. The counter-suggestion from you though says that one must choose
> to go to  SM/SA - that is something that IBTA itself doesn't agree with.
> Those that want to consult the SM/SA can continue to do so but those that
> don't want to because they certainly see advantages in avoiding SM/SA when
> possible should be allowed to do so; exactly as IBTA does.
> 
> [whs] I agree with you on keeping your SA communications to a minimum. Joe's
> point is that the SM/SA is the origination point of path data. Some other
> service can obtain that data from the SM/SA and outwardly provide it. I
> am not, and would not think that Joe would be (correct me if I am wrong)
> adverse to someone postulating such a service. But as things stand today
> with the 1.0a spec, the SA now is the only defined interface.
> --

        SA is the repository of the information. How you get it, when
        you get it is the implementations choice. It is not tied to
        address resolution. So it is not an argument for getting the
        LID from the SA at all times.


Vivek

> 
> ---[snip]---
> 
> regards all -
> 
> -------------------------------
> Bill Swortwood
> (Office)(503) 690 8578
> (CELL)  (503) 819-5281
> --------------------------------
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 13:31:49 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA08562
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 13:31:49 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id NAA17624
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 13:31:50 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA17115;
	Tue, 15 Jan 2002 13:29:07 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA17001
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 13:25:53 -0500 (EST)
Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA08312
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 13:25:46 -0500 (EST)
Received: from eastmail2.East.Sun.COM ([129.148.1.241])
	by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id KAA26976;
	Tue, 15 Jan 2002 10:25:47 -0800 (PST)
Received: from esp-bugs.east (lincoln.East.Sun.COM [129.148.183.45])
	by eastmail2.East.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id NAA18221;
	Tue, 15 Jan 2002 13:25:45 -0500 (EST)
Received: (from drc@localhost)
	by esp-bugs.east (8.9.3+Sun/8.9.1) id NAA28880;
	Tue, 15 Jan 2002 13:25:45 -0500 (EST)
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-ID: <15428.29736.711762.341888@esp-bugs.east>
Date: Tue, 15 Jan 2002 13:25:44 -0500 (EST)
From: drc@zanclea.east.Sun.COM
To: Joe Pelissier <joep@Sanera.net>
Cc: ipoverib@ietf.org
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
In-Reply-To: <3C43B187.9C2BB72C@sanera.net>
References: <200201142203.g0EM33qT126365@jurassic.eng.sun.com>
	<5.1.0.14.2.20020114164653.00ad7d40@esalpha2.cup.hp.com>
	<3C43B187.9C2BB72C@sanera.net>
X-Mailer: VM 6.72 under 21.1 (patch 8) "Bryce Canyon" XEmacs Lucid
Reply-To: daniel.cassiday@east.sun.com
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Joe Pelissier writes:
 > Greetings:
 > I unfortunately have had to reduce my participation in this effort for a while,
 > but have now been studying the email threads attempting to get back up to speed.
 > My apologies in advance if my comments below are missing some context from
 > previous disucssions.
 > 
 > It seems that there continues to be a great deal of discussion as to whether ARP
 > can be accomplished without consulting the SA (or at least cached information
 > from a previous consultation).  There seems to be two major issues: static rate
 > and service level.
 > 
 > All nodes that operate at greater than x1 speed are required to support static
 > rate.  The static rate setting is a function of the path between two endpoints.
 > It is important to note that the path multicast uses is very likely to be quite
 > different than the path that unicast traffic uses.  Therefore, the static rate
 > may well be different.  The only way to get this information is to consult the SA
 > (actually, one could always set their static rate to x1 speed and avoid the
 > consultation, but if you got a x12 link, it seems a shame not to use it just to
 > avoid talking to your friendly SA ;-).
 > 

Actually always using a static rate of 1x for ARP multicast may be a
good idea.  All this does is limit source multicast traffic to not
exceed ~200MB/s (surely this is enough).  I see no downside to this.

 > The other is SL.  It has repeatedly been argued that the spec does not require
 > getting the SL from the SA.  This is true.  This has been recognized by the IBTA
 > and steps are being taken to correct it.  In order to support multiple paths
 > through the IB fabric, it was a fundamental assumption that the SA controls which
 > SLs a node may use based on path.  The SA may provide a set of SLs from which a
 > node may choose, but the SA needs to control this.  If it does not, deadlock of
 > the fabric may occur.  To prevent deadlock, the SA can load certain IB tables
 > that would cause the offending packets to be discarded.  This protects the
 > fabric, but would cause communication to fail between the offending nodes.
 > 
 > So we could argue all this forever, or maybe there is another solution.  The IB
 > spec already talks about how to determine both static rate and SL.  Therefore, we
 > don't need to rehash it in the RFC (other than maybe a note that says these are
 > determined IAW the InfiniBand standard).
 > 
 > wrt/ SL, this is pretty much what section 6.0 of
 > draft-kashyap-ipoib-ipv4-and-arp-01.txt says, except for the last paragraph
 > (which recommends using the SL from the broadcast group).  Unfortunately, the
 > recommendation is directly contrary to the some intended usage of IB and is
 > likely to either be prohibited or at least strongly discouraged by a future
 > version of the specification.
 > 
 > The LWG voted unanimously to recommend the addition of language that strongly
 > discourages the use of parameters that were not obtained by an applicable path
 > record and stating that the parameters from multicast and unicast should not be
 > mixed (see the LWG meeting minutes dated 8/16/01).
 > 
 > If we just remove the recommendation from the draft, then we can move forward
 > without further discussion.  It just seems bazaar to continue to push something
 > as "recommended" that causes great concern to many of the architects that created
 > IB.  All we need to do is leave out the recommendation and we are done with this
 > issue.  Those that wish to ignore the SL assignment from the SA and use the
 > multicast one can still do so without technically violating the current IBTA
 > spec.  There seems to be little downside to this.
 > 
 > Besides, you need to get static rate anyway....
 > 
 > Another $0.02 worth...
 > Regards,
 > Joe Pelissier
 > 
 > Michael Krause wrote:
 > 
 > > At 05:10 PM 1/14/2002 -0600, Ashwin Kamath wrote:
 > > >There is a general thought that ARP is a one step solution for obtaining the
 > > >parameters required for establishing IPoIB over UD. But I think that even
 > > >for using ARP, one will require to get the Path record information. I do not
 > > >think that ARP is a one stop solution. What happens if both nodes determine
 > > >that their MTU is 2K without considering that a switch component in between
 > > >might restrict the MTU to 256 bytes?
 > >
 > > The minimum supported MTU for UD traffic is 2KB so a path with a 256 bytes
 > > MTU should not be used by the IP over IB service.  The SM working with the
 > > partition manager will establish a set of paths that can be used for IP
 > > over IB traffic which will include the setting up of an all nodes multicast
 > > group per partition that support IP traffic.
 > >
 > > >Such information can be obtained only
 > > >from the central SM via a path record. Am I missing something here? If
 > > >address resolution is all that ARP is required to do, without trying to
 > > >establish parameters for communication one may not want to know the MTU, but
 > > >from the discussions I see that the Address resolution is being used for
 > > >establishing communication parameters (like the QPN)
 > >
 > > QP identification is an optimization that allows one to eliminate the
 > > Ethertype field.  It also allows an implementation to map different IP
 > > addresses to different QP thereby providing an implementation optimization
 > > through more fine grain hardware-based de-multiplexing.  An example use
 > > might be to assign an IP address per OS instance which shares a HCA port
 > > and then allow all traffic to target a corresponding QP without having to
 > > flow through an intermediate hypervisor that would have to de-multiplex the
 > > traffic.  This can provide both performance and scalability improvements
 > > for vendor offerings.
 > >
 > > >So the only real problem with having the LID in the ARP response, that I see
 > > >is when the ARP response needs to cross IB Routers. Can I assume that
 > > >changes in LID assignments are not a big problem because such changes are
 > > >not intended to be frequent and current ARP cache implementations already
 > > >factor in the possibility of a link level address change. Changing LIDs
 > > >would have a bigger impact on shared devices using RC than on IPoIB using
 > > >UD. Is it ok if I consider a change in LID assignment to be equivalent of a
 > > >server reboot?
 > >
 > > No.  Applications should not be impacted by a change in LID assignment as
 > > there are defined methods to migrate applications from one LID or GID or
 > > port to another that are completely transparent to the application.  For
 > > UD-based services, one could simply re-ARP the endnode via the all-nodes
 > > multicast and provide a simple update of the LID mapping as this can be
 > > derived from the IB headers if desired.
 > >
 > > >If I understood what Jerry, one will have a problem if the ARP responder
 > > >lies in another IB Subnet, in which case the LID in the response will not be
 > > >relavant to the originator of the ARP request, in-fact the LID will have to
 > > >be the LID of the router. Was there any resolution reached for this problem?
 > > >
 > > >In short you do need an ARP + Path record diagnosis to determine all the
 > > >parameters for communication. What would happen if we bring an IB router in
 > > >between? One cannot obtain a path record to a node in another subnet. One
 > > >can obtain the path record to the IB Router. Should we be talking about the
 > > >behaviour of the IB router in case of mismatch of MTUs or should we say that
 > > >we will always use the smallest mtu for communicating across an IB router.
 > > >Or should we say that this could fall on the plate of the Infiniband WGs.
 > >
 > > The IB router protocol will define how one routes across IB subnets - an
 > > IBTA issue.  An IP router may be implemented along side an IB router or it
 > > may be an edge device such as a multi-protocol router.  The fact that IP
 > > subnets may span multiple IB subnets isn't really that big of an issue per
 > > se.  By keeping the LID not part of the link address structure, the ARP
 > > should be able to transparently flow across multiple IB subnets.
 > >
 > > Mike
 > >
 > > _______________________________________________
 > > IPoverIB mailing list
 > > IPoverIB@ietf.org
 > > http://www.ietf.org/mailman/listinfo/ipoverib
 > 
 > 
 > _______________________________________________
 > IPoverIB mailing list
 > IPoverIB@ietf.org
 > http://www.ietf.org/mailman/listinfo/ipoverib
 > 


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 13:51:22 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA09307
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 13:51:22 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id NAA18052
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 13:51:19 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA18043;
	Tue, 15 Jan 2002 13:50:32 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA17990
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 13:50:30 -0500 (EST)
Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA09271
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 13:50:28 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id MAA61330;
	Tue, 15 Jan 2002 12:46:17 -0600
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0FIoSe62252;
	Tue, 15 Jan 2002 13:50:28 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0FIoQV17137;
	Tue, 15 Jan 2002 10:50:26 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0FIoPF05262;
	Tue, 15 Jan 2002 10:50:25 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201151850.g0FIoPF05262@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
To: daniel.cassiday@east.sun.com
Date: Tue, 15 Jan 2002 10:50:24 -0800 (PST)
Cc: joep@Sanera.net (Joe Pelissier), ipoverib@ietf.org
In-Reply-To: <15428.29736.711762.341888@esp-bugs.east> from "drc@zanclea.east.Sun.COM" at Jan 15, 2002 12:25:44 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

>  > 
> 
> Actually always using a static rate of 1x for ARP multicast may be a
> good idea.  All this does is limit source multicast traffic to not
> exceed ~200MB/s (surely this is enough).  I see no downside to this.
> 

        Exactly, it is an implementation's choice on how to use path
        rates/ setup the subnets etc. It is not tied to determining
        the link address.


Vivek


>  > The other is SL.  It has repeatedly been argued that the spec does not require
>  > getting the SL from the SA.  This is true.  This has been recognized by the IBTA
>  > and steps are being taken to correct it.  In order to support multiple paths
>  > through the IB fabric, it was a fundamental assumption that the SA controls which
>  > SLs a node may use based on path.  The SA may provide a set of SLs from which a
>  > node may choose, but the SA needs to control this.  If it does not, deadlock of
>  > the fabric may occur.  To prevent deadlock, the SA can load certain IB tables
>  > that would cause the offending packets to be discarded.  This protects the
>  > fabric, but would cause communication to fail between the offending nodes.
>  > 
>  > So we could argue all this forever, or maybe there is another solution.  The IB
>  > spec already talks about how to determine both static rate and SL.  Therefore, we
>  > don't need to rehash it in the RFC (other than maybe a note that says these are
>  > determined IAW the InfiniBand standard).
>  > 
>  > wrt/ SL, this is pretty much what section 6.0 of
>  > draft-kashyap-ipoib-ipv4-and-arp-01.txt says, except for the last paragraph
>  > (which recommends using the SL from the broadcast group).  Unfortunately, the
>  > recommendation is directly contrary to the some intended usage of IB and is
>  > likely to either be prohibited or at least strongly discouraged by a future
>  > version of the specification.
>  > 
>  > The LWG voted unanimously to recommend the addition of language that strongly
>  > discourages the use of parameters that were not obtained by an applicable path
>  > record and stating that the parameters from multicast and unicast should not be
>  > mixed (see the LWG meeting minutes dated 8/16/01).
>  > 
>  > If we just remove the recommendation from the draft, then we can move forward
>  > without further discussion.  It just seems bazaar to continue to push something
>  > as "recommended" that causes great concern to many of the architects that created
>  > IB.  All we need to do is leave out the recommendation and we are done with this
>  > issue.  Those that wish to ignore the SL assignment from the SA and use the
>  > multicast one can still do so without technically violating the current IBTA
>  > spec.  There seems to be little downside to this.
>  > 
>  > Besides, you need to get static rate anyway....
>  > 
>  > Another $0.02 worth...
>  > Regards,
>  > Joe Pelissier
>  > 
>  > Michael Krause wrote:
>  > 
>  > > At 05:10 PM 1/14/2002 -0600, Ashwin Kamath wrote:
>  > > >There is a general thought that ARP is a one step solution for obtaining the
>  > > >parameters required for establishing IPoIB over UD. But I think that even
>  > > >for using ARP, one will require to get the Path record information. I do not
>  > > >think that ARP is a one stop solution. What happens if both nodes determine
>  > > >that their MTU is 2K without considering that a switch component in between
>  > > >might restrict the MTU to 256 bytes?
>  > >
>  > > The minimum supported MTU for UD traffic is 2KB so a path with a 256 bytes
>  > > MTU should not be used by the IP over IB service.  The SM working with the
>  > > partition manager will establish a set of paths that can be used for IP
>  > > over IB traffic which will include the setting up of an all nodes multicast
>  > > group per partition that support IP traffic.
>  > >
>  > > >Such information can be obtained only
>  > > >from the central SM via a path record. Am I missing something here? If
>  > > >address resolution is all that ARP is required to do, without trying to
>  > > >establish parameters for communication one may not want to know the MTU, but
>  > > >from the discussions I see that the Address resolution is being used for
>  > > >establishing communication parameters (like the QPN)
>  > >
>  > > QP identification is an optimization that allows one to eliminate the
>  > > Ethertype field.  It also allows an implementation to map different IP
>  > > addresses to different QP thereby providing an implementation optimization
>  > > through more fine grain hardware-based de-multiplexing.  An example use
>  > > might be to assign an IP address per OS instance which shares a HCA port
>  > > and then allow all traffic to target a corresponding QP without having to
>  > > flow through an intermediate hypervisor that would have to de-multiplex the
>  > > traffic.  This can provide both performance and scalability improvements
>  > > for vendor offerings.
>  > >
>  > > >So the only real problem with having the LID in the ARP response, that I see
>  > > >is when the ARP response needs to cross IB Routers. Can I assume that
>  > > >changes in LID assignments are not a big problem because such changes are
>  > > >not intended to be frequent and current ARP cache implementations already
>  > > >factor in the possibility of a link level address change. Changing LIDs
>  > > >would have a bigger impact on shared devices using RC than on IPoIB using
>  > > >UD. Is it ok if I consider a change in LID assignment to be equivalent of a
>  > > >server reboot?
>  > >
>  > > No.  Applications should not be impacted by a change in LID assignment as
>  > > there are defined methods to migrate applications from one LID or GID or
>  > > port to another that are completely transparent to the application.  For
>  > > UD-based services, one could simply re-ARP the endnode via the all-nodes
>  > > multicast and provide a simple update of the LID mapping as this can be
>  > > derived from the IB headers if desired.
>  > >
>  > > >If I understood what Jerry, one will have a problem if the ARP responder
>  > > >lies in another IB Subnet, in which case the LID in the response will not be
>  > > >relavant to the originator of the ARP request, in-fact the LID will have to
>  > > >be the LID of the router. Was there any resolution reached for this problem?
>  > > >
>  > > >In short you do need an ARP + Path record diagnosis to determine all the
>  > > >parameters for communication. What would happen if we bring an IB router in
>  > > >between? One cannot obtain a path record to a node in another subnet. One
>  > > >can obtain the path record to the IB Router. Should we be talking about the
>  > > >behaviour of the IB router in case of mismatch of MTUs or should we say that
>  > > >we will always use the smallest mtu for communicating across an IB router.
>  > > >Or should we say that this could fall on the plate of the Infiniband WGs.
>  > >
>  > > The IB router protocol will define how one routes across IB subnets - an
>  > > IBTA issue.  An IP router may be implemented along side an IB router or it
>  > > may be an edge device such as a multi-protocol router.  The fact that IP
>  > > subnets may span multiple IB subnets isn't really that big of an issue per
>  > > se.  By keeping the LID not part of the link address structure, the ARP
>  > > should be able to transparently flow across multiple IB subnets.
>  > >
>  > > Mike
>  > >
>  > > _______________________________________________
>  > > IPoverIB mailing list
>  > > IPoverIB@ietf.org
>  > > http://www.ietf.org/mailman/listinfo/ipoverib
>  > 
>  > 
>  > _______________________________________________
>  > IPoverIB mailing list
>  > IPoverIB@ietf.org
>  > http://www.ietf.org/mailman/listinfo/ipoverib
>  > 
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 13:59:43 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA07510
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 13:06:15 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id NAA16419
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 13:06:17 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA16290;
	Tue, 15 Jan 2002 13:04:22 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA16177
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 13:04:17 -0500 (EST)
Received: from palrel12.hp.com (palrel12.hp.com [156.153.255.237])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA07427
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 13:04:15 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel12.hp.com (Postfix) with ESMTP id B412CE00593
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 10:03:42 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id KAA23526
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 10:03:42 -0800 (PST)
Message-Id: <5.1.0.14.2.20020115091655.00ae5668@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Tue, 15 Jan 2002 09:25:04 -0800
To: ipoverib@ietf.org
From: Michael Krause <krause@cup.hp.com>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
In-Reply-To: <014f01c19d81$4877b030$0100007f@vaio505he>
References: <200201142245.g0EMjiV08688@eng4.beaverton.ibm.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

At 06:58 AM 1/15/2002 +0200, Yaron Haviv wrote:
>Vivek,
>
>I agree with your assumptions
>We need them all: LID + capability flags + GID + QPN
>Mikes remark that you could extract the LID from the incoming packet however
>looks interesting (as long as we make sure that people don't play with it,
>e.g. redirect traffic etc')
>
>As far as I understand the only open issues currently are (if someone thinks
>there are more please respond):
>LID: Yes/No
>Ethertype
>Capability flags
>
>As for the why LID:
>I think that going to the SM is not so efficient given the fact that it is a
>central resource and all the nodes in the subnet go to if we could avoid it
>why not

How often do people really think they are going to the SM/SA for any of 
this information?  This isn't the Internet; it is a data center fabric.  So 
with the exception of a reboot of an individual node and fabric state 
events, most implementations will be able to cache their respective IP to 
IB mappings for very long periods of time.  This means that the actual 
packet rate for IP implementations to the SM/SA will be quite low.

>When IP will span IB the problem becomes more serious: Unlike IP routers
>which has a default route entry, currently IB has explicit entries for each
>path this might require doing too many PathRecord operations, which are also
>currently very slow when going outside an IB subnet
>(I had posted a proposal to the MSO team trying to resolve those issues
>somehow however it is under discussion)

IBTA proposals remain confidential within the IBTA and should not be 
discussed within the IETF.  Currently, the IB router protocol is under 
development and as such is not for open discussion.  As to whether the IB 
router protocol will modified to support default route entries, uses 
explicit route entries, etc. that has not been finalized and is subject to 
member review, etc.

Again, I will argue that the rate of endnode change within the IP view is 
rather slow and thus the inquiry rate to the SM/SA even for a global view 
will not present scalability problems with an intelligent implementation.

>I would tend to send this info anyway since the extra 16 bits wouldn't harm
>anyone, one might choose not to use them if he wants to go to the SM, this
>way everybody will be happy and over time people will use the best solution

It is unnecessary data that could cause confusion and interoperability 
problems.

>As for the Ethertype:
>After giving Vivek a hard time on this issue I accept his proposal
>Not because of the performance issues raised (since I don't think they are
>real) But because it is not really needed, and if it simplifies things, 
>why not
>
>As for the capability bits:
>Avoiding the SIDR is a smart move
>I'm not sure if the UC option will be used, if using connection why not
>reliable connection
>I just wanted to add SDP (Socket Direct Protocol) bit, Again ones might
>choose not to use it, for the one that do it really simplifies the process a
>lot

Capability discovery has to work as a separate inquiry method as noted in 
my previous response.  Having two methods for a given task is a waste and 
creates potential interoperability problems.  It is simpler for the spec to 
only specify one method and let implementations do likewise.

>I'm hoping we get a quick convergence on those issues so we would move
>forward
>The issues are not serious and I think if an alternative enable someone to
>do something he wants without violating the other's ability to do what they
>want just ignoring the extra info, we should go for it

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 14:10:58 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA10071
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 14:10:58 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id OAA18943
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 14:10:59 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA18928;
	Tue, 15 Jan 2002 14:10:03 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA18896
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 14:10:02 -0500 (EST)
Received: from sentinel.sanera.net (ns1.sanera.net [208.253.254.10])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA10043
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 14:09:59 -0500 (EST)
Received: from icarus.sanera.net (icarus [192.168.254.11])
	by sentinel.sanera.net (8.11.2/8.11.2) with ESMTP id g0FJ9Vu20675
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 11:09:31 -0800
Received: from newton.sanera.net (newton.sanera.net [172.16.2.11])
	by icarus.sanera.net (8.11.6/8.11.6) with ESMTP id g0FJ9Pn16151
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 11:09:25 -0800
Received: from sanera.net (dhcp-pc-100-161.sanera.net [192.168.100.161])
	by newton.sanera.net (8.11.4/8.11.4) with ESMTP id g0FJ9PM18832
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 11:09:25 -0800 (PST)
Message-ID: <3C447E64.9A048F65@sanera.net>
Date: Tue, 15 Jan 2002 11:09:25 -0800
From: Joe Pelissier <joep@Sanera.net>
X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U)
X-Accept-Language: en
MIME-Version: 1.0
To: ipoverib@ietf.org
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
References: <200201151850.g0FIoPF05262@eng4.beaverton.ibm.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

I was not referring to the static rate for multicast.  I agree there is no downside to
limiting ARP traffic to 1x speeds.  I was referring to real IP traffic.  If I want to talk
IP from a to b at rates greater than 1x, I believe it was the intent that I must get a
static rate value from the SA.  Probably this is another case where the IBTA went through
all the trouble of providing a mechanism to determine a parameter but did not mandate its
use.

Cheers,
Joe Pelissier


Vivek Kashyap wrote:

> >  >
> >
> > Actually always using a static rate of 1x for ARP multicast may be a
> > good idea.  All this does is limit source multicast traffic to not
> > exceed ~200MB/s (surely this is enough).  I see no downside to this.
> >
>
>         Exactly, it is an implementation's choice on how to use path
>         rates/ setup the subnets etc. It is not tied to determining
>         the link address.
>
> Vivek
>
> >  > The other is SL.  It has repeatedly been argued that the spec does not require
> >  > getting the SL from the SA.  This is true.  This has been recognized by the IBTA
> >  > and steps are being taken to correct it.  In order to support multiple paths
> >  > through the IB fabric, it was a fundamental assumption that the SA controls which
> >  > SLs a node may use based on path.  The SA may provide a set of SLs from which a
> >  > node may choose, but the SA needs to control this.  If it does not, deadlock of
> >  > the fabric may occur.  To prevent deadlock, the SA can load certain IB tables
> >  > that would cause the offending packets to be discarded.  This protects the
> >  > fabric, but would cause communication to fail between the offending nodes.
> >  >
> >  > So we could argue all this forever, or maybe there is another solution.  The IB
> >  > spec already talks about how to determine both static rate and SL.  Therefore, we
> >  > don't need to rehash it in the RFC (other than maybe a note that says these are
> >  > determined IAW the InfiniBand standard).
> >  >
> >  > wrt/ SL, this is pretty much what section 6.0 of
> >  > draft-kashyap-ipoib-ipv4-and-arp-01.txt says, except for the last paragraph
> >  > (which recommends using the SL from the broadcast group).  Unfortunately, the
> >  > recommendation is directly contrary to the some intended usage of IB and is
> >  > likely to either be prohibited or at least strongly discouraged by a future
> >  > version of the specification.
> >  >
> >  > The LWG voted unanimously to recommend the addition of language that strongly
> >  > discourages the use of parameters that were not obtained by an applicable path
> >  > record and stating that the parameters from multicast and unicast should not be
> >  > mixed (see the LWG meeting minutes dated 8/16/01).
> >  >
> >  > If we just remove the recommendation from the draft, then we can move forward
> >  > without further discussion.  It just seems bazaar to continue to push something
> >  > as "recommended" that causes great concern to many of the architects that created
> >  > IB.  All we need to do is leave out the recommendation and we are done with this
> >  > issue.  Those that wish to ignore the SL assignment from the SA and use the
> >  > multicast one can still do so without technically violating the current IBTA
> >  > spec.  There seems to be little downside to this.
> >  >
> >  > Besides, you need to get static rate anyway....
> >  >
> >  > Another $0.02 worth...
> >  > Regards,
> >  > Joe Pelissier
> >  >
> >  > Michael Krause wrote:
> >  >
> >  > > At 05:10 PM 1/14/2002 -0600, Ashwin Kamath wrote:
> >  > > >There is a general thought that ARP is a one step solution for obtaining the
> >  > > >parameters required for establishing IPoIB over UD. But I think that even
> >  > > >for using ARP, one will require to get the Path record information. I do not
> >  > > >think that ARP is a one stop solution. What happens if both nodes determine
> >  > > >that their MTU is 2K without considering that a switch component in between
> >  > > >might restrict the MTU to 256 bytes?
> >  > >
> >  > > The minimum supported MTU for UD traffic is 2KB so a path with a 256 bytes
> >  > > MTU should not be used by the IP over IB service.  The SM working with the
> >  > > partition manager will establish a set of paths that can be used for IP
> >  > > over IB traffic which will include the setting up of an all nodes multicast
> >  > > group per partition that support IP traffic.
> >  > >
> >  > > >Such information can be obtained only
> >  > > >from the central SM via a path record. Am I missing something here? If
> >  > > >address resolution is all that ARP is required to do, without trying to
> >  > > >establish parameters for communication one may not want to know the MTU, but
> >  > > >from the discussions I see that the Address resolution is being used for
> >  > > >establishing communication parameters (like the QPN)
> >  > >
> >  > > QP identification is an optimization that allows one to eliminate the
> >  > > Ethertype field.  It also allows an implementation to map different IP
> >  > > addresses to different QP thereby providing an implementation optimization
> >  > > through more fine grain hardware-based de-multiplexing.  An example use
> >  > > might be to assign an IP address per OS instance which shares a HCA port
> >  > > and then allow all traffic to target a corresponding QP without having to
> >  > > flow through an intermediate hypervisor that would have to de-multiplex the
> >  > > traffic.  This can provide both performance and scalability improvements
> >  > > for vendor offerings.
> >  > >
> >  > > >So the only real problem with having the LID in the ARP response, that I see
> >  > > >is when the ARP response needs to cross IB Routers. Can I assume that
> >  > > >changes in LID assignments are not a big problem because such changes are
> >  > > >not intended to be frequent and current ARP cache implementations already
> >  > > >factor in the possibility of a link level address change. Changing LIDs
> >  > > >would have a bigger impact on shared devices using RC than on IPoIB using
> >  > > >UD. Is it ok if I consider a change in LID assignment to be equivalent of a
> >  > > >server reboot?
> >  > >
> >  > > No.  Applications should not be impacted by a change in LID assignment as
> >  > > there are defined methods to migrate applications from one LID or GID or
> >  > > port to another that are completely transparent to the application.  For
> >  > > UD-based services, one could simply re-ARP the endnode via the all-nodes
> >  > > multicast and provide a simple update of the LID mapping as this can be
> >  > > derived from the IB headers if desired.
> >  > >
> >  > > >If I understood what Jerry, one will have a problem if the ARP responder
> >  > > >lies in another IB Subnet, in which case the LID in the response will not be
> >  > > >relavant to the originator of the ARP request, in-fact the LID will have to
> >  > > >be the LID of the router. Was there any resolution reached for this problem?
> >  > > >
> >  > > >In short you do need an ARP + Path record diagnosis to determine all the
> >  > > >parameters for communication. What would happen if we bring an IB router in
> >  > > >between? One cannot obtain a path record to a node in another subnet. One
> >  > > >can obtain the path record to the IB Router. Should we be talking about the
> >  > > >behaviour of the IB router in case of mismatch of MTUs or should we say that
> >  > > >we will always use the smallest mtu for communicating across an IB router.
> >  > > >Or should we say that this could fall on the plate of the Infiniband WGs.
> >  > >
> >  > > The IB router protocol will define how one routes across IB subnets - an
> >  > > IBTA issue.  An IP router may be implemented along side an IB router or it
> >  > > may be an edge device such as a multi-protocol router.  The fact that IP
> >  > > subnets may span multiple IB subnets isn't really that big of an issue per
> >  > > se.  By keeping the LID not part of the link address structure, the ARP
> >  > > should be able to transparently flow across multiple IB subnets.
> >  > >
> >  > > Mike
> >  > >
> >  > > _______________________________________________
> >  > > IPoverIB mailing list
> >  > > IPoverIB@ietf.org
> >  > > http://www.ietf.org/mailman/listinfo/ipoverib
> >  >
> >  >
> >  > _______________________________________________
> >  > IPoverIB mailing list
> >  > IPoverIB@ietf.org
> >  > http://www.ietf.org/mailman/listinfo/ipoverib
> >  >
> >
> >
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> >
>
> --
> Vivek Kashyap
> Linux Technology Center, IBM
> kashyapv@us.ibm.com
> vivk@us.ibm.com
> 503 578 3422 (o)


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 14:21:23 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA10345
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 14:21:22 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id OAA19162
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 14:21:24 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA19144;
	Tue, 15 Jan 2002 14:20:46 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA19076
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 14:20:42 -0500 (EST)
Received: from palrel12.hp.com (palrel12.hp.com [156.153.255.237])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA10328
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 14:20:40 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel12.hp.com (Postfix) with ESMTP id AE3F8E00600
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 11:20:11 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id LAA09833
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 11:20:11 -0800 (PST)
Message-Id: <5.1.0.14.2.20020115111204.028a58e0@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Tue, 15 Jan 2002 11:20:00 -0800
To: ipoverib@ietf.org
From: Michael Krause <krause@cup.hp.com>
Subject: Re: Some annotation (was RE: [Ipoverib] What's inside an IPoIB
  link layer address?)
In-Reply-To: <200201151815.g0FIFVY03360@eng4.beaverton.ibm.com>
References: <HFEHJGAKENECEAHPNJECAEDLCBAA.billiam33@attbi.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

At 10:15 AM 1/15/2002 -0800, Vivek Kashyap wrote:
> >
> >       The idea of avoiding the SM is a very valid one - I wonder what all
> > this SM/SA will do ?
> >
> > [whs] Set up the topology of the IB subnet, keep the subnet topology 
> intact,
> > collect and disperse subnet event information, set up partitions within the
> > subnet (part of topology), provide path information within the subnet, act
> > as the subnet multicast setup and join agency, and act as a low level
> > service
> > registry.
> >
> > Everything on the CA will go over the same QPs(0/1)
> > and will go over VL15 (which doesn't have congestion control, looses
> > packets etc.), will go the SM/SA at its QP0/1 in a large SM.
> >
> > [whs] Only SMPs (Subnet Management Packets) go on VL15. GMPS (General
> > Management Packets) go on any other assigned VL that is not VL15. The
> > SM communications are those that set up and keep the topology within
> > a subnet intact, and are the controlled domain of the SM, for the most
> > part. GMP communications are used by entities wanting information --
> > like topology or path record information, from the SA. The SA can also
> > redirect its communications to any other QP/Port within the subnet, so
> > communications can be balanced or serviced at other locales within the
> > subnet. Point here - it is NOT all VL15, and NOT all QP0/1. Further,
> > SL and static rate information are specific to the path in question.
> > --
>
>         OK so there is some respite. However, how many paths lead to
>         SA/SM in a subnet ? It still stands to reason if one wants to
>         avoid it and can avoid it why force them ?

Interoperability by providing only one method to obtain a particular data 
object.  This does not preclude additional methods from being used but the 
spec will define one method guaranteed to always work for all.


>         As noted earlier, SLs need not come from the SA. They of
>         course will be recorded there in the path record but that does
>         not imply that they can't be determined differently e.g. QoS
>         mappings.

This would be an implementation issue as all implementations can rely upon 
the SA for this information.

>         Similarly the path rate needs to be determined and will be
>         recorded at the SA/SM. That does not mean that one has to go
>         to the SA at the time of the address resolution to determine
>         it.

Again an implementation issue.  The SA provides the one source that can be 
relied upon.  All other methods are potential optimizations that can be 
defined other adjunct documentation.

>         I am arguing against the notion that because SL or path rate
>         need to be known the LID need not be included in the address
>         resolution. I don't see why even though based on your
>         communication.

I do not believe there is any benefit to having the LID be a mandatory 
element for the base specification.  Define one method that provides 
guaranteed interoperability for all implementations and then come back and 
define optimizations as experienced is gained from real 
deployments.  Keeping this spec simple to implement is the first step to 
getting a solution deployed from which experience can be gained for future 
enhancements.  It also will allow new material to be added once the IBTA 
has completed its router protocol definition which may present additional 
issues yet to be raised within this workgroup.

> > The SM will be
> > consulted for path records, for service id resolutions, for this and that
> > and that also. It seems a bit overmuch. That is an aside anyway for one to
> > think about.
> >
> > [whs] Suggest you chat with Dr. Pfister [IBM] on this. PathRecords need to
> > come
> > from somewhere. Paths can change. SLs and static rate are particular to a
> > Path.
>
>         Please re-read what I have said. No where do I say that path
>         records won't be recorded at the SA. Nor is it suggested that
>         you preclude it. The discussion is different. It is not
>         correct to use the argument that the path record is at the
>         SA/SM and hence we need to get the LID from there.
>
>         Inclusion of LID is an IP requirement. It needs to be
>         determined at every address resolution. SL/path rate can be
>         static in a subnet, can be determined differently including
>         indirectly/directly from the SA.

The LID, SL, rate control, P_Key, Q_Key, etc. all need to be determined for 
address resolution.  All that is being recommended by some is to use the 
KISS principle for the base spec which is quite a reasonable request.

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 14:29:58 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA10651
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 14:29:58 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id OAA19305
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 14:30:00 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA19112;
	Tue, 15 Jan 2002 14:20:44 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA19068
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 14:20:42 -0500 (EST)
Received: from palrel11.hp.com (palrel11.hp.com [156.153.255.246])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA10327
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 14:20:39 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel11.hp.com (Postfix) with ESMTP id 64E7FE0030D
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 11:20:11 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id LAA09829
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 11:20:11 -0800 (PST)
Message-Id: <5.1.0.14.2.20020115110851.029bdd08@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Tue, 15 Jan 2002 11:11:42 -0800
To: ipoverib@ietf.org
From: Michael Krause <krause@cup.hp.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
In-Reply-To: <200201151850.g0FIoPF05262@eng4.beaverton.ibm.com>
References: <15428.29736.711762.341888@esp-bugs.east>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

At 10:50 AM 1/15/2002 -0800, Vivek Kashyap wrote:
> >  >
> >
> > Actually always using a static rate of 1x for ARP multicast may be a
> > good idea.  All this does is limit source multicast traffic to not
> > exceed ~200MB/s (surely this is enough).  I see no downside to this.
> >
>
>         Exactly, it is an implementation's choice on how to use path
>         rates/ setup the subnets etc. It is not tied to determining
>         the link address.

I do not believe anyone is contending that an implementation may not 
default to a static rate of 1x for any IP service; all that is being 
contended is that the IP over IB spec be required to specify that one must 
use static rate in compliance with the IB specification.  This will allow 
implementations to achieve interoperability independent of the physical IB 
attributes for all types of IP traffic.

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 14:33:33 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA10816
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 14:33:33 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id OAA19798
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 14:33:35 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA19756;
	Tue, 15 Jan 2002 14:32:36 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA19720
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 14:32:34 -0500 (EST)
Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA10761
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 14:32:32 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id NAA31576;
	Tue, 15 Jan 2002 13:28:22 -0600
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0FJWXe277778;
	Tue, 15 Jan 2002 14:32:33 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0FJWVV18488;
	Tue, 15 Jan 2002 11:32:31 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0FJWUg07583;
	Tue, 15 Jan 2002 11:32:30 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201151932.g0FJWUg07583@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
To: krause@cup.hp.com (Michael Krause)
Date: Tue, 15 Jan 2002 11:32:29 -0800 (PST)
Cc: ipoverib@ietf.org
In-Reply-To: <5.1.0.14.2.20020115110851.029bdd08@esalpha2.cup.hp.com> from "Michael Krause" at Jan 15, 2002 10:11:42 AM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

> 
> At 10:50 AM 1/15/2002 -0800, Vivek Kashyap wrote:
> > >  >
> > >
> > > Actually always using a static rate of 1x for ARP multicast may be a
> > > good idea.  All this does is limit source multicast traffic to not
> > > exceed ~200MB/s (surely this is enough).  I see no downside to this.
> > >
> >
> >         Exactly, it is an implementation's choice on how to use path
> >         rates/ setup the subnets etc. It is not tied to determining
> >         the link address.
> 
> I do not believe anyone is contending that an implementation may not 
> default to a static rate of 1x for any IP service; all that is being 
> contended is that the IP over IB spec be required to specify that one must 
> use static rate in compliance with the IB specification.  This will allow 
> implementations to achieve interoperability independent of the physical IB 
> attributes for all types of IP traffic.

No argument with using a specific or any IBTA method. Am only stating
that it is upto the implementation to follow any method to determine
the values. That still doesn't tie it to determining the LID. If one
has found the values due to any method - SL through QoS method, path
rate due to knowing the topology is homogenous in
speed/speeds/whatever - one may choose not to go to SA for those
values. This has nothing to do with the LID.

My objection is to the claim that the LID must be determined from the
SA/SM because some may choose to get the path-rate/SL from the SM/SA.
These are disjoint issues.

Vivek

> 
> Mike
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 14:44:51 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA11104
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 14:44:51 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id OAA20045
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 14:44:53 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA19982;
	Tue, 15 Jan 2002 14:44:06 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA19951
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 14:44:04 -0500 (EST)
Received: from vieo.com (vieo.com [216.30.79.131])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA11074
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 14:44:01 -0500 (EST)
Received: (from root@localhost)
	by vieo.com (8.11.2/8.11.2) id g0FJhT616708
	for ipoverib@ietf.org; Tue, 15 Jan 2002 13:43:29 -0600 (CST)
	(envelope-from rronald@VIEO.com)
Received: from Fishhead (dhcp-10-2-15-151.vieo.com [10.2.15.151])
	by vieo.com (8.11.2/8.11.2) with SMTP id g0FJhSK16626;
	Tue, 15 Jan 2002 13:43:28 -0600 (CST)
	(envelope-from rronald@VIEO.com)
Message-ID: <001101c19dfc$df6656b0$970f020a@Fishhead>
From: "Roger Ronald" <rronald@VIEO.com>
To: <yaronh@voltaire.com>, <ipoverib@ietf.org>
References: <015201c19d83$592702d0$0100007f@vaio505he>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
Date: Tue, 15 Jan 2002 13:43:14 -0600
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 5.50.4133.2400
X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400
X-scanner: scanned by Inflex 0.1.5c+ on vieo.com
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

There are benchmarked IP implementations that already exceed 
1x IB rates (work done by Jim Pinkerton at SGI using GSN). 
These are few and far between, but speeds of this sort
will be more common as time goes by. The reason that 
IP over IB is important is the opportunity to "go fast".

Any solution for IP over IB should NOT assume that progress
will not continue and that  requirements will not grow!

I will agree that the requirements for ARP multicast 
bandwidth are much lower.

RR
----- Original Message ----- 
From: "Yaron Haviv" <yaronh@voltaire.com>
To: "'Joe Pelissier'" <joep@sanera.net>; <ipoverib@ietf.org>
Sent: Monday, January 14, 2002 11:13 PM
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?


> Joe, welcome back to the discussion
> Regarding the IPD (static rate control) issue you mentioned
> I don't see any host managing to fill a 1X pipe doing IPoIB (the TCP stack
> would die much sooner); those bandwidths are for native IB protocols such as
> SDP and SRP, setting it to 1X as a default will probably wont create any
> problem
> 
> Another issue around it, currently the LWG is discussing using the IPD for
> congestion avoidance with a thinner granularity that will be dynamically
> adjusted (through FECN's, MAD's)
> 
> Yaron
> -----Original Message-----
> From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
> Joe Pelissier
> Sent: Tuesday, January 15, 2002 6:35 AM
> To: ipoverib@ietf.org
> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> 
> Greetings:
> I unfortunately have had to reduce my participation in this effort for a
> while,
> but have now been studying the email threads attempting to get back up to
> speed.
> My apologies in advance if my comments below are missing some context from
> previous disucssions.
> 
> It seems that there continues to be a great deal of discussion as to whether
> ARP
> can be accomplished without consulting the SA (or at least cached
> information
> from a previous consultation).  There seems to be two major issues: static
> rate
> and service level.
> 
> All nodes that operate at greater than x1 speed are required to support
> static
> rate.  The static rate setting is a function of the path between two
> endpoints.
> It is important to note that the path multicast uses is very likely to be
> quite
> different than the path that unicast traffic uses.  Therefore, the static
> rate
> may well be different.  The only way to get this information is to consult
> the SA
> (actually, one could always set their static rate to x1 speed and avoid the
> consultation, but if you got a x12 link, it seems a shame not to use it just
> to
> avoid talking to your friendly SA ;-).
> 
> The other is SL.  It has repeatedly been argued that the spec does not
> require
> getting the SL from the SA.  This is true.  This has been recognized by the
> IBTA
> and steps are being taken to correct it.  In order to support multiple paths
> through the IB fabric, it was a fundamental assumption that the SA controls
> which
> SLs a node may use based on path.  The SA may provide a set of SLs from
> which a
> node may choose, but the SA needs to control this.  If it does not, deadlock
> of
> the fabric may occur.  To prevent deadlock, the SA can load certain IB
> tables
> that would cause the offending packets to be discarded.  This protects the
> fabric, but would cause communication to fail between the offending nodes.
> 
> So we could argue all this forever, or maybe there is another solution.  The
> IB
> spec already talks about how to determine both static rate and SL.
> Therefore, we
> don't need to rehash it in the RFC (other than maybe a note that says these
> are
> determined IAW the InfiniBand standard).
> 
> wrt/ SL, this is pretty much what section 6.0 of
> draft-kashyap-ipoib-ipv4-and-arp-01.txt says, except for the last paragraph
> (which recommends using the SL from the broadcast group).  Unfortunately,
> the
> recommendation is directly contrary to the some intended usage of IB and is
> likely to either be prohibited or at least strongly discouraged by a future
> version of the specification.
> 
> The LWG voted unanimously to recommend the addition of language that
> strongly
> discourages the use of parameters that were not obtained by an applicable
> path
> record and stating that the parameters from multicast and unicast should not
> be
> mixed (see the LWG meeting minutes dated 8/16/01).
> 
> If we just remove the recommendation from the draft, then we can move
> forward
> without further discussion.  It just seems bazaar to continue to push
> something
> as "recommended" that causes great concern to many of the architects that
> created
> IB.  All we need to do is leave out the recommendation and we are done with
> this
> issue.  Those that wish to ignore the SL assignment from the SA and use the
> multicast one can still do so without technically violating the current IBTA
> 
> spec.  There seems to be little downside to this.
> 
> Besides, you need to get static rate anyway....
> 
> Another $0.02 worth...
> Regards,
> Joe Pelissier
> 
> Michael Krause wrote:
> 
> > At 05:10 PM 1/14/2002 -0600, Ashwin Kamath wrote:
> > >There is a general thought that ARP is a one step solution for obtaining
> the
> > >parameters required for establishing IPoIB over UD. But I think that even
> > >for using ARP, one will require to get the Path record information. I do
> not
> > >think that ARP is a one stop solution. What happens if both nodes
> determine
> > >that their MTU is 2K without considering that a switch component in
> between
> > >might restrict the MTU to 256 bytes?
> >
> > The minimum supported MTU for UD traffic is 2KB so a path with a 256 bytes
> > MTU should not be used by the IP over IB service.  The SM working with the
> > partition manager will establish a set of paths that can be used for IP
> > over IB traffic which will include the setting up of an all nodes
> multicast
> > group per partition that support IP traffic.
> >
> > >Such information can be obtained only
> > >from the central SM via a path record. Am I missing something here? If
> > >address resolution is all that ARP is required to do, without trying to
> > >establish parameters for communication one may not want to know the MTU,
> but
> > >from the discussions I see that the Address resolution is being used for
> > >establishing communication parameters (like the QPN)
> >
> > QP identification is an optimization that allows one to eliminate the
> > Ethertype field.  It also allows an implementation to map different IP
> > addresses to different QP thereby providing an implementation optimization
> > through more fine grain hardware-based de-multiplexing.  An example use
> > might be to assign an IP address per OS instance which shares a HCA port
> > and then allow all traffic to target a corresponding QP without having to
> > flow through an intermediate hypervisor that would have to de-multiplex
> the
> > traffic.  This can provide both performance and scalability improvements
> > for vendor offerings.
> >
> > >So the only real problem with having the LID in the ARP response, that I
> see
> > >is when the ARP response needs to cross IB Routers. Can I assume that
> > >changes in LID assignments are not a big problem because such changes are
> > >not intended to be frequent and current ARP cache implementations already
> > >factor in the possibility of a link level address change. Changing LIDs
> > >would have a bigger impact on shared devices using RC than on IPoIB using
> > >UD. Is it ok if I consider a change in LID assignment to be equivalent of
> a
> > >server reboot?
> >
> > No.  Applications should not be impacted by a change in LID assignment as
> > there are defined methods to migrate applications from one LID or GID or
> > port to another that are completely transparent to the application.  For
> > UD-based services, one could simply re-ARP the endnode via the all-nodes
> > multicast and provide a simple update of the LID mapping as this can be
> > derived from the IB headers if desired.
> >
> > >If I understood what Jerry, one will have a problem if the ARP responder
> > >lies in another IB Subnet, in which case the LID in the response will not
> be
> > >relavant to the originator of the ARP request, in-fact the LID will have
> to
> > >be the LID of the router. Was there any resolution reached for this
> problem?
> > >
> > >In short you do need an ARP + Path record diagnosis to determine all the
> > >parameters for communication. What would happen if we bring an IB router
> in
> > >between? One cannot obtain a path record to a node in another subnet. One
> > >can obtain the path record to the IB Router. Should we be talking about
> the
> > >behaviour of the IB router in case of mismatch of MTUs or should we say
> that
> > >we will always use the smallest mtu for communicating across an IB
> router.
> > >Or should we say that this could fall on the plate of the Infiniband WGs.
> >
> > The IB router protocol will define how one routes across IB subnets - an
> > IBTA issue.  An IP router may be implemented along side an IB router or it
> > may be an edge device such as a multi-protocol router.  The fact that IP
> > subnets may span multiple IB subnets isn't really that big of an issue per
> > se.  By keeping the LID not part of the link address structure, the ARP
> > should be able to transparently flow across multiple IB subnets.
> >
> > Mike
> >
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 
> 


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 14:49:41 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA11233
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 14:49:41 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id OAA20165
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 14:49:43 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA20137;
	Tue, 15 Jan 2002 14:48:28 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA20089
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 14:48:24 -0500 (EST)
Received: from sentinel.sanera.net (ns1.sanera.net [208.253.254.10])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA11217
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 14:48:21 -0500 (EST)
Received: from icarus.sanera.net (icarus [192.168.254.11])
	by sentinel.sanera.net (8.11.2/8.11.2) with ESMTP id g0FJlru21019
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 11:47:53 -0800
Received: from newton.sanera.net (newton.sanera.net [172.16.2.11])
	by icarus.sanera.net (8.11.6/8.11.6) with ESMTP id g0FJlmn17809
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 11:47:48 -0800
Received: from sanera.net (dhcp-pc-100-161.sanera.net [192.168.100.161])
	by newton.sanera.net (8.11.4/8.11.4) with ESMTP id g0FJllM21306
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 11:47:48 -0800 (PST)
Message-ID: <3C448762.6CCB8812@sanera.net>
Date: Tue, 15 Jan 2002 11:47:47 -0800
From: Joe Pelissier <joep@Sanera.net>
X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U)
X-Accept-Language: en
MIME-Version: 1.0
To: ipoverib@ietf.org
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
References: <OFCA25676A.DCAD4019-ON88256B42.001DFDD0@boulder.ibm.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Comments embedded marked <JEP>
Regards,
Joe Pelissier

Vivek Kashyap wrote:

> Greetings Joe,
>
>       A couple of clarifications are in order.
>
>       The idea of avoiding the SM is a very valid one - I wonder what all
> this SM/SA will do ? Everything on the CA will go over the same QPs(0/1)
> and will go over VL15 (which doesn't have congestion control, looses
> packets etc.), will go the SM/SA at its QP0/1 in a large SM. The SM will be
> consulted for path records, for service id resolutions, for this and that
> and that also. It seems a bit overmuch. That is an aside anyway for one to
> think about.
>

<JEP>  A very small amount of traffic goes over QP0/VL15.  None of it is relevant to the discussion here.  We are
concerned with communication with the SA over QP1 which does not use VL15 and is subject to flow control.  These
packets are not dropped due to congestion.

>
>       There are other reasons for avoiding the SM/SA that I  have mentioned
> previously. However, nowhere has it been suggested that SM/SA cannot be
> consulted. The counter-suggestion from you though says that one must choose
> to go to  SM/SA - that is something that IBTA itself doesn't agree with.
> Those that want to consult the SM/SA can continue to do so but those that
> don't want to because they certainly see advantages in avoiding SM/SA when
> possible should be allowed to do so; exactly as IBTA does.
>

<JEP>  I agree that no one is suggesting prohibiting consultation with the SM/SA.  What I object to is the
recommendation in the draft against such consultation.  If this recommendation was simply removed and the draft was
silent on the topic, then I would have no objection.  I think I am correct in stating the you believe it works and
interoperates either way (which, btw, I do not believe), so why push so hard for the recommendation?  It seems clear
the consensus is unlikely, so lets compromise by not promoting nor prohibiting either approach in the draft and future
RFC.

>
>       Regarding your comments on SL - I am a member of IBTA. IBTA rules do
> not allow to one to give out the statements and the voting results  to
> non-IBTA forums, so I'll only state that IBTA did not change the
> specification - it is available to anyone on www.infinibandta.org  and did
> not modify the rules regarding SL determination as Mike Krause in the mail
> before yours has clarified. Mike is also a member(chair) of LWG.  Others
> who are members of the IBTA can verify for themselves the final decision of
> LWG. It did not preclude determining SL from sources other than the SA.
>

<JEP>  You are correct about the rules, my apologies.  Yes, the specification does not require that one obtain the SL
from the SA and no public statement has been made by the IBTA regarding whether or not this is likely to change in the
future.

>
>       You are right in stating that protection from a random SL is required
> but no where is it being suggested that the SL used by the endnodes will be
> random one. It will be a well thought of one and a correct one except it
> will be determined without consulting the SA/SM. Nothing in IBTA precludes
> it. Rather IBA specification promotes it.  BTW, the use SL from multicast
> or determination of it  in any other way has been left to the choice of the
> implementor. Not part of IPoIB architecture. Also, again, IBTA does not
> specify a particular method either.
>

<JEP>  Hmmm, I'm not sure that the fact that the spec is silent on the topic means that it is promoting it.

>
>       All nodes if working greater than 1x are required to support static
> rate iff any destination in the fabric works at < the node's rate.  It
> merely says that the values needs to be determined but does not make the
> method for determining it mandatory. Consulting the SA to get the path
> record is one of the methods. Additionally IBTA is defining a congestion
> control mechanism disjoint from injection control.

<JEP> A minor nit - all nodes that are operating at a rate greater than 1x are required to support static rate.  I
have not checked the spec in detail, but my guess is your probably right - it probably does not specify that one gets
the static rate from the SA.  However, as in the SL case, a mechanism is provided to get this information from the
SA.  So again, I grant you that you are not technically violating the specification if you use a static rate other
than that provided by the SA.  My only objection would be any language in a draft or RFC that specifically required or
recommended that one not consult the SA for this information.  Likewise, I am not proposing that the the draft or RFC
specifically prohibit or recommend against it.

>
>
>       An example:  do you mean to say that if I know that my subnet is made
> of 4x through and through I need to consult the SA to get the information
> always ? I don't think so.

<JEP> The only mechanism provided by IB to make such a determination is the SA.  If you have some other way of knowing
that there are no 1x links and there never will be (say, e.g., by an administrative setting), then the fabric would
operate as intended if 4x nodes did not consult with the SA.  Such operation, however, is beyond the scope of the
specification.  And as I stated previously, I have no desire to prohibit such operation within the drafts or RFCs.  I
simply do not wish to promote it either.  I just do not believe that consensus is obtainable for a recommendation
either way so lets move on.

>
>
>       The suggestion here that some implementations will avoid the SA/SM in
> a particular subnet if they think it is right in no way precludes
> consulting the SA/SM or any other entity when required. That is not the
> contention at all.  It is not being mandated on everyone. Similarly there
> is no need to require every implementation to throw a look toward the SA/SM
> for SL/path rate or any other information.

<JEP>  I am not asking that we require or recommend that every implementation throw a look the the SA/SM.  I am asking
that we do not require or recommend that they do not.

>
>
> Vivek
>
> --
> Vivek Kashyap
> Linux Technology Center, IBM
> vivk@us.ibm.com
> kashyapv@us.ibm.com
> Ph: 503 578 3422 T/L: 775 3422
>
>
>                       Joe Pelissier
>                       <joep@Sanera.net>        To:       ipoverib@ietf.org
>                       Sent by:                 cc:
>                       ipoverib-admin@ie        Subject:  Re: [Ipoverib] What's inside an IPoIB link layer address?
>                       tf.org
>
>
>                       01/14/2002 08:35
>                       PM
>
>
>
> Greetings:
> I unfortunately have had to reduce my participation in this effort for a
> while,
> but have now been studying the email threads attempting to get back up to
> speed.
> My apologies in advance if my comments below are missing some context from
> previous disucssions.
>
> It seems that there continues to be a great deal of discussion as to
> whether ARP
> can be accomplished without consulting the SA (or at least cached
> information
> from a previous consultation).  There seems to be two major issues: static
> rate
> and service level.
>
> All nodes that operate at greater than x1 speed are required to support
> static
> rate.  The static rate setting is a function of the path between two
> endpoints.
> It is important to note that the path multicast uses is very likely to be
> quite
> different than the path that unicast traffic uses.  Therefore, the static
> rate
> may well be different.  The only way to get this information is to consult
> the SA
> (actually, one could always set their static rate to x1 speed and avoid the
> consultation, but if you got a x12 link, it seems a shame not to use it
> just to
> avoid talking to your friendly SA ;-).
>
> The other is SL.  It has repeatedly been argued that the spec does not
> require
> getting the SL from the SA.  This is true.  This has been recognized by the
> IBTA
> and steps are being taken to correct it.  In order to support multiple
> paths
> through the IB fabric, it was a fundamental assumption that the SA controls
> which
> SLs a node may use based on path.  The SA may provide a set of SLs from
> which a
> node may choose, but the SA needs to control this.  If it does not,
> deadlock of
> the fabric may occur.  To prevent deadlock, the SA can load certain IB
> tables
> that would cause the offending packets to be discarded.  This protects the
> fabric, but would cause communication to fail between the offending nodes.
>
> So we could argue all this forever, or maybe there is another solution.
> The IB
> spec already talks about how to determine both static rate and SL.
> Therefore, we
> don't need to rehash it in the RFC (other than maybe a note that says these
> are
> determined IAW the InfiniBand standard).
>
> wrt/ SL, this is pretty much what section 6.0 of
> draft-kashyap-ipoib-ipv4-and-arp-01.txt says, except for the last paragraph
> (which recommends using the SL from the broadcast group).  Unfortunately,
> the
> recommendation is directly contrary to the some intended usage of IB and is
> likely to either be prohibited or at least strongly discouraged by a future
> version of the specification.
>
> The LWG voted unanimously to recommend the addition of language that
> strongly
> discourages the use of parameters that were not obtained by an applicable
> path
> record and stating that the parameters from multicast and unicast should
> not be
> mixed (see the LWG meeting minutes dated 8/16/01).
>
> If we just remove the recommendation from the draft, then we can move
> forward
> without further discussion.  It just seems bazaar to continue to push
> something
> as "recommended" that causes great concern to many of the architects that
> created
> IB.  All we need to do is leave out the recommendation and we are done with
> this
> issue.  Those that wish to ignore the SL assignment from the SA and use the
> multicast one can still do so without technically violating the current
> IBTA
> spec.  There seems to be little downside to this.
>
> Besides, you need to get static rate anyway....
>
> Another $0.02 worth...
> Regards,
> Joe Pelissier
>
> Michael Krause wrote:
>
> > At 05:10 PM 1/14/2002 -0600, Ashwin Kamath wrote:
> > >There is a general thought that ARP is a one step solution for obtaining
> the
> > >parameters required for establishing IPoIB over UD. But I think that
> even
> > >for using ARP, one will require to get the Path record information. I do
> not
> > >think that ARP is a one stop solution. What happens if both nodes
> determine
> > >that their MTU is 2K without considering that a switch component in
> between
> > >might restrict the MTU to 256 bytes?
> >
> > The minimum supported MTU for UD traffic is 2KB so a path with a 256
> bytes
> > MTU should not be used by the IP over IB service.  The SM working with
> the
> > partition manager will establish a set of paths that can be used for IP
> > over IB traffic which will include the setting up of an all nodes
> multicast
> > group per partition that support IP traffic.
> >
> > >Such information can be obtained only
> > >from the central SM via a path record. Am I missing something here? If
> > >address resolution is all that ARP is required to do, without trying to
> > >establish parameters for communication one may not want to know the MTU,
> but
> > >from the discussions I see that the Address resolution is being used for
> > >establishing communication parameters (like the QPN)
> >
> > QP identification is an optimization that allows one to eliminate the
> > Ethertype field.  It also allows an implementation to map different IP
> > addresses to different QP thereby providing an implementation
> optimization
> > through more fine grain hardware-based de-multiplexing.  An example use
> > might be to assign an IP address per OS instance which shares a HCA port
> > and then allow all traffic to target a corresponding QP without having to
> > flow through an intermediate hypervisor that would have to de-multiplex
> the
> > traffic.  This can provide both performance and scalability improvements
> > for vendor offerings.
> >
> > >So the only real problem with having the LID in the ARP response, that I
> see
> > >is when the ARP response needs to cross IB Routers. Can I assume that
> > >changes in LID assignments are not a big problem because such changes
> are
> > >not intended to be frequent and current ARP cache implementations
> already
> > >factor in the possibility of a link level address change. Changing LIDs
> > >would have a bigger impact on shared devices using RC than on IPoIB
> using
> > >UD. Is it ok if I consider a change in LID assignment to be equivalent
> of a
> > >server reboot?
> >
> > No.  Applications should not be impacted by a change in LID assignment as
> > there are defined methods to migrate applications from one LID or GID or
> > port to another that are completely transparent to the application.  For
> > UD-based services, one could simply re-ARP the endnode via the all-nodes
> > multicast and provide a simple update of the LID mapping as this can be
> > derived from the IB headers if desired.
> >
> > >If I understood what Jerry, one will have a problem if the ARP responder
> > >lies in another IB Subnet, in which case the LID in the response will
> not be
> > >relavant to the originator of the ARP request, in-fact the LID will have
> to
> > >be the LID of the router. Was there any resolution reached for this
> problem?
> > >
> > >In short you do need an ARP + Path record diagnosis to determine all the
> > >parameters for communication. What would happen if we bring an IB router
> in
> > >between? One cannot obtain a path record to a node in another subnet.
> One
> > >can obtain the path record to the IB Router. Should we be talking about
> the
> > >behaviour of the IB router in case of mismatch of MTUs or should we say
> that
> > >we will always use the smallest mtu for communicating across an IB
> router.
> > >Or should we say that this could fall on the plate of the Infiniband
> WGs.
> >
> > The IB router protocol will define how one routes across IB subnets - an
> > IBTA issue.  An IP router may be implemented along side an IB router or
> it
> > may be an edge device such as a multi-protocol router.  The fact that IP
> > subnets may span multiple IB subnets isn't really that big of an issue
> per
> > se.  By keeping the LID not part of the link address structure, the ARP
> > should be able to transparently flow across multiple IB subnets.
> >
> > Mike
> >
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
>
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
>
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 15:05:53 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA11663
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 15:05:52 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id PAA20834
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 15:05:55 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA20332;
	Tue, 15 Jan 2002 14:58:48 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA20298
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 14:58:46 -0500 (EST)
Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA11448
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 14:58:42 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e31.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAA129264;
	Tue, 15 Jan 2002 14:55:32 -0500
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0FJwQr62218;
	Tue, 15 Jan 2002 12:58:26 -0700
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0FJwPV19258;
	Tue, 15 Jan 2002 11:58:25 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0FJwO408903;
	Tue, 15 Jan 2002 11:58:24 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201151958.g0FJwO408903@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
To: joep@Sanera.net (Joe Pelissier)
Date: Tue, 15 Jan 2002 11:58:23 -0800 (PST)
Cc: ipoverib@ietf.org
In-Reply-To: <3C448762.6CCB8812@sanera.net> from "Joe Pelissier" at Jan 15, 2002 10:47:47 AM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

> 
> >
> >       There are other reasons for avoiding the SM/SA that I  have mentioned
> > previously. However, nowhere has it been suggested that SM/SA cannot be
> > consulted. The counter-suggestion from you though says that one must choose
> > to go to  SM/SA - that is something that IBTA itself doesn't agree with.
> > Those that want to consult the SM/SA can continue to do so but those that
> > don't want to because they certainly see advantages in avoiding SM/SA when
> > possible should be allowed to do so; exactly as IBTA does.
> >
> 
> <JEP>  I agree that no one is suggesting prohibiting consultation with the SM/SA.  What I object to is the
> recommendation in the draft against such consultation.  If this recommendation was simply removed and the draft was
> silent on the topic, then I would have no objection.  I think I am correct in stating the you believe it works and
> interoperates either way (which, btw, I do not believe), so why push so hard for the recommendation?  It seems clear
> the consensus is unlikely, so lets compromise by not promoting nor prohibiting either approach in the draft and future
> RFC.

<VK> Joe, you have not been looking at the drafts latest drafts. There
are two working group drafts out - IPoIB architecture and on
multicast. Things have moved forward. Skip the old draft. It is no
longer in consideration.
<VK>

> 
> >
> >       Regarding your comments on SL - I am a member of IBTA. IBTA rules do
> > not allow to one to give out the statements and the voting results  to
> > non-IBTA forums, so I'll only state that IBTA did not change the
> > specification - it is available to anyone on www.infinibandta.org  and did
> > not modify the rules regarding SL determination as Mike Krause in the mail
> > before yours has clarified. Mike is also a member(chair) of LWG.  Others
> > who are members of the IBTA can verify for themselves the final decision of
> > LWG. It did not preclude determining SL from sources other than the SA.
> >
> 
> <JEP>  You are correct about the rules, my apologies.  Yes, the specification does not require that one obtain the SL
> from the SA and no public statement has been made by the IBTA regarding whether or not this is likely to change in the
> future.
> 
> >
> >       You are right in stating that protection from a random SL is required
> > but no where is it being suggested that the SL used by the endnodes will be
> > random one. It will be a well thought of one and a correct one except it
> > will be determined without consulting the SA/SM. Nothing in IBTA precludes
> > it. Rather IBA specification promotes it.  BTW, the use SL from multicast
> > or determination of it  in any other way has been left to the choice of the
> > implementor. Not part of IPoIB architecture. Also, again, IBTA does not
> > specify a particular method either.
> >
> 
> <JEP>  Hmmm, I'm not sure that the fact that the spec is silent on the topic means that it is promoting it.
> 

<VK> It refers to service levels under QoS and talks a lot about that. It hardly mentions deadlocks for example.
<VK>

> >
> >       All nodes if working greater than 1x are required to support static
> > rate iff any destination in the fabric works at < the node's rate.  It
> > merely says that the values needs to be determined but does not make the
> > method for determining it mandatory. Consulting the SA to get the path
> > record is one of the methods. Additionally IBTA is defining a congestion
> > control mechanism disjoint from injection control.
> 
> <JEP> A minor nit - all nodes that are operating at a rate greater than 1x are required to support static rate.  I
> have not checked the spec in detail, but my guess is your probably right - it probably does not specify that one gets
> the static rate from the SA.  However, as in the SL case, a mechanism is provided to get this information from the
> SA.  So again, I grant you that you are not technically violating the specification if you use a static rate other
> than that provided by the SA.  My only objection would be any language in a draft or RFC that specifically required or
> recommended that one not consult the SA for this information.  Likewise, I am not proposing that the the draft or RFC
> specifically prohibit or recommend against it.

<VK> Nothing is suggested to be added to the draft. All I am trying to
point out is that other solutions are possible. So let us not put it
into the draft that it must be through a particular method i.e.
consult the SA. I think we are in agreement to say that the method of
determinig the values is upto the implementations.

I am also suggesting that due to this reason and the reason that this
information may be received at a different 'time period' one must not
make it a reason to say that the LID must be got from the SA. That is
my main point.

<VK>


> >       The suggestion here that some implementations will avoid the SA/SM in
> > a particular subnet if they think it is right in no way precludes
> > consulting the SA/SM or any other entity when required. That is not the
> > contention at all.  It is not being mandated on everyone. Similarly there
> > is no need to require every implementation to throw a look toward the SA/SM
> > for SL/path rate or any other information.
> 
> <JEP>  I am not asking that we require or recommend that every implementation throw a look the the SA/SM.  I am asking
> that we do not require or recommend that they do not.

<VK> We are in agreement then.  

        Determining such values is upto the implementation.
        Determining the LID is a separate disjoint issue.

Vivek

> 
> >
> >
> > Vivek
> >
> > --
> > Vivek Kashyap
> > Linux Technology Center, IBM
> > vivk@us.ibm.com
> > kashyapv@us.ibm.com
> > Ph: 503 578 3422 T/L: 775 3422
> >
> >
> >                       Joe Pelissier
> >                       <joep@Sanera.net>        To:       ipoverib@ietf.org
> >                       Sent by:                 cc:
> >                       ipoverib-admin@ie        Subject:  Re: [Ipoverib] What's inside an IPoIB link layer address?
> >                       tf.org
> >
> >
> >                       01/14/2002 08:35
> >                       PM
> >
> >
> >
> > Greetings:
> > I unfortunately have had to reduce my participation in this effort for a
> > while,
> > but have now been studying the email threads attempting to get back up to
> > speed.
> > My apologies in advance if my comments below are missing some context from
> > previous disucssions.
> >
> > It seems that there continues to be a great deal of discussion as to
> > whether ARP
> > can be accomplished without consulting the SA (or at least cached
> > information
> > from a previous consultation).  There seems to be two major issues: static
> > rate
> > and service level.
> >
> > All nodes that operate at greater than x1 speed are required to support
> > static
> > rate.  The static rate setting is a function of the path between two
> > endpoints.
> > It is important to note that the path multicast uses is very likely to be
> > quite
> > different than the path that unicast traffic uses.  Therefore, the static
> > rate
> > may well be different.  The only way to get this information is to consult
> > the SA
> > (actually, one could always set their static rate to x1 speed and avoid the
> > consultation, but if you got a x12 link, it seems a shame not to use it
> > just to
> > avoid talking to your friendly SA ;-).
> >
> > The other is SL.  It has repeatedly been argued that the spec does not
> > require
> > getting the SL from the SA.  This is true.  This has been recognized by the
> > IBTA
> > and steps are being taken to correct it.  In order to support multiple
> > paths
> > through the IB fabric, it was a fundamental assumption that the SA controls
> > which
> > SLs a node may use based on path.  The SA may provide a set of SLs from
> > which a
> > node may choose, but the SA needs to control this.  If it does not,
> > deadlock of
> > the fabric may occur.  To prevent deadlock, the SA can load certain IB
> > tables
> > that would cause the offending packets to be discarded.  This protects the
> > fabric, but would cause communication to fail between the offending nodes.
> >
> > So we could argue all this forever, or maybe there is another solution.
> > The IB
> > spec already talks about how to determine both static rate and SL.
> > Therefore, we
> > don't need to rehash it in the RFC (other than maybe a note that says these
> > are
> > determined IAW the InfiniBand standard).
> >
> > wrt/ SL, this is pretty much what section 6.0 of
> > draft-kashyap-ipoib-ipv4-and-arp-01.txt says, except for the last paragraph
> > (which recommends using the SL from the broadcast group).  Unfortunately,
> > the
> > recommendation is directly contrary to the some intended usage of IB and is
> > likely to either be prohibited or at least strongly discouraged by a future
> > version of the specification.
> >
> > The LWG voted unanimously to recommend the addition of language that
> > strongly
> > discourages the use of parameters that were not obtained by an applicable
> > path
> > record and stating that the parameters from multicast and unicast should
> > not be
> > mixed (see the LWG meeting minutes dated 8/16/01).
> >
> > If we just remove the recommendation from the draft, then we can move
> > forward
> > without further discussion.  It just seems bazaar to continue to push
> > something
> > as "recommended" that causes great concern to many of the architects that
> > created
> > IB.  All we need to do is leave out the recommendation and we are done with
> > this
> > issue.  Those that wish to ignore the SL assignment from the SA and use the
> > multicast one can still do so without technically violating the current
> > IBTA
> > spec.  There seems to be little downside to this.
> >
> > Besides, you need to get static rate anyway....
> >
> > Another $0.02 worth...
> > Regards,
> > Joe Pelissier
> >
> > Michael Krause wrote:
> >
> > > At 05:10 PM 1/14/2002 -0600, Ashwin Kamath wrote:
> > > >There is a general thought that ARP is a one step solution for obtaining
> > the
> > > >parameters required for establishing IPoIB over UD. But I think that
> > even
> > > >for using ARP, one will require to get the Path record information. I do
> > not
> > > >think that ARP is a one stop solution. What happens if both nodes
> > determine
> > > >that their MTU is 2K without considering that a switch component in
> > between
> > > >might restrict the MTU to 256 bytes?
> > >
> > > The minimum supported MTU for UD traffic is 2KB so a path with a 256
> > bytes
> > > MTU should not be used by the IP over IB service.  The SM working with
> > the
> > > partition manager will establish a set of paths that can be used for IP
> > > over IB traffic which will include the setting up of an all nodes
> > multicast
> > > group per partition that support IP traffic.
> > >
> > > >Such information can be obtained only
> > > >from the central SM via a path record. Am I missing something here? If
> > > >address resolution is all that ARP is required to do, without trying to
> > > >establish parameters for communication one may not want to know the MTU,
> > but
> > > >from the discussions I see that the Address resolution is being used for
> > > >establishing communication parameters (like the QPN)
> > >
> > > QP identification is an optimization that allows one to eliminate the
> > > Ethertype field.  It also allows an implementation to map different IP
> > > addresses to different QP thereby providing an implementation
> > optimization
> > > through more fine grain hardware-based de-multiplexing.  An example use
> > > might be to assign an IP address per OS instance which shares a HCA port
> > > and then allow all traffic to target a corresponding QP without having to
> > > flow through an intermediate hypervisor that would have to de-multiplex
> > the
> > > traffic.  This can provide both performance and scalability improvements
> > > for vendor offerings.
> > >
> > > >So the only real problem with having the LID in the ARP response, that I
> > see
> > > >is when the ARP response needs to cross IB Routers. Can I assume that
> > > >changes in LID assignments are not a big problem because such changes
> > are
> > > >not intended to be frequent and current ARP cache implementations
> > already
> > > >factor in the possibility of a link level address change. Changing LIDs
> > > >would have a bigger impact on shared devices using RC than on IPoIB
> > using
> > > >UD. Is it ok if I consider a change in LID assignment to be equivalent
> > of a
> > > >server reboot?
> > >
> > > No.  Applications should not be impacted by a change in LID assignment as
> > > there are defined methods to migrate applications from one LID or GID or
> > > port to another that are completely transparent to the application.  For
> > > UD-based services, one could simply re-ARP the endnode via the all-nodes
> > > multicast and provide a simple update of the LID mapping as this can be
> > > derived from the IB headers if desired.
> > >
> > > >If I understood what Jerry, one will have a problem if the ARP responder
> > > >lies in another IB Subnet, in which case the LID in the response will
> > not be
> > > >relavant to the originator of the ARP request, in-fact the LID will have
> > to
> > > >be the LID of the router. Was there any resolution reached for this
> > problem?
> > > >
> > > >In short you do need an ARP + Path record diagnosis to determine all the
> > > >parameters for communication. What would happen if we bring an IB router
> > in
> > > >between? One cannot obtain a path record to a node in another subnet.
> > One
> > > >can obtain the path record to the IB Router. Should we be talking about
> > the
> > > >behaviour of the IB router in case of mismatch of MTUs or should we say
> > that
> > > >we will always use the smallest mtu for communicating across an IB
> > router.
> > > >Or should we say that this could fall on the plate of the Infiniband
> > WGs.
> > >
> > > The IB router protocol will define how one routes across IB subnets - an
> > > IBTA issue.  An IP router may be implemented along side an IB router or
> > it
> > > may be an edge device such as a multi-protocol router.  The fact that IP
> > > subnets may span multiple IB subnets isn't really that big of an issue
> > per
> > > se.  By keeping the LID not part of the link address structure, the ARP
> > > should be able to transparently flow across multiple IB subnets.
> > >
> > > Mike
> > >
> > > _______________________________________________
> > > IPoverIB mailing list
> > > IPoverIB@ietf.org
> > > http://www.ietf.org/mailman/listinfo/ipoverib
> >
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> >
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 15:33:13 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA12501
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 15:33:13 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id PAA21610
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 15:33:14 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA21172;
	Tue, 15 Jan 2002 15:29:33 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA21142
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 15:29:32 -0500 (EST)
Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA12415
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 15:29:22 -0500 (EST)
Received: from sunmail1.Sun.COM ([129.145.1.2])
	by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id MAA25364;
	Tue, 15 Jan 2002 12:29:19 -0800 (PST)
Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.87.31])
	by sunmail1.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1-Sun.COM.mod.2) with ESMTP id MAA08019;
	Tue, 15 Jan 2002 12:30:30 -0800 (PST)
Received: from taipei (taipei.Eng.Sun.COM [129.146.85.178])
	by jurassic.eng.sun.com (8.12.2+Sun/8.12.2) with SMTP id g0FKTIqT306973;
	Tue, 15 Jan 2002 12:29:18 -0800 (PST)
Message-Id: <200201152029.g0FKTIqT306973@jurassic.eng.sun.com>
Date: Tue, 15 Jan 2002 12:26:48 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
To: yaronh@voltaire.com
Cc: ipoverib@ietf.org
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: chF1v36XpmwdO3DuCJkrDA==
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.6_06 SunOS 5.8 sun4u sparc 
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Please see comments below.

<snip>

>Vivek,
>
>I agree with your assumptions
>We need them all: LID + capability flags + GID + QPN
>Mikes remark that you could extract the LID from the incoming packet however
>looks interesting (as long as we make sure that people don't play with it,
>e.g. redirect traffic etc')
>
>As far as I understand the only open issues currently are (if someone thinks
>there are more please respond):
>LID: Yes/No
>Ethertype
>Capability flags
>
>As for the why LID:
>I think that going to the SM is not so efficient given the fact that it is a
>central resource and all the nodes in the subnet go to if we could avoid it
>why not

First the benefit is dubious. In modern TCP/IP stacks the frequency of ARP
requests is capped by the ARP cache flush internal, which is in minutes
range. (E.g. in Solaris it's five minutes.) Assuming you have 100 nodes
on a subnet, you're telling me saving the SM lookup call, which at max
rate (if a local node is activetly talking to every other node) of one
every 3 sec. is worth noting?

I'm still not sure if a trip to SM can be saved at all (need to catch
up with all the mail this morning). Even if in some cases it could, do
we believe that not doing any pathrecord lookup is going to be the norm 
of IPoIB implementation that warrents an extr field in the link addr?
How about all the issues I've pointed out before? I have another one
to add - simplicility. Unless there is sufficient justification, the more
fields you add to a protocol the more complexity (in areas you may not
even forsee now) it's going to add.

>When IP will span IB the problem becomes more serious: Unlike IP routers
>which has a default route entry, currently IB has explicit entries for each
>path this might require doing too many PathRecord operations, which are also
>currently very slow when going outside an IB subnet
>(I had posted a proposal to the MSO team trying to resolve those issues
>somehow however it is under discussion)
>I would tend to send this info anyway since the extra 16 bits wouldn't harm
>anyone,

But it does. What would you define the LID for a node on a non-local IB
subnet? It can't be a LID to a router because this is an endnode's "MAC
addr" we're defining here.

>one might choose not to use them if he wants to go to the SM, this
>way everybody will be happy and over time people will use the best solution

No we can't do handwaving here. You either define a field with clear
semantics so implementations have a chance to interoperate, or take it
out. There is no iffy here (unless you add a "LID included" flag to the
link addr).

Jerry

>
>As for the Ethertype:
>After giving Vivek a hard time on this issue I accept his proposal
>Not because of the performance issues raised (since I don't think they are
>real)
>But because it is not really needed, and if it simplifies things, why not
>
>As for the capability bits:
>Avoiding the SIDR is a smart move
>I'm not sure if the UC option will be used, if using connection why not
>reliable connection
>I just wanted to add SDP (Socket Direct Protocol) bit, Again ones might
>choose not to use it, for the one that do it really simplifies the process a
>lot
>
>I'm hoping we get a quick convergence on those issues so we would move
>forward
>The issues are not serious and I think if an alternative enable someone to
>do something he wants without violating the other's ability to do what they
>want just ignoring the extra info, we should go for it
>
>Yaron
>
>-----Original Message-----
>From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
>Vivek Kashyap
>Sent: Tuesday, January 15, 2002 12:46 AM
>To: Jerry.Chu@eng.sun.com
>Cc: ipoverib@ietf.org
>Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>
>Folks,
>
>       It is close to the last call. In this regard I would like to
>       state that we have been discussing this issue for a long time
>       now. The most recent discussion is what I presented at the SLC
>       last december. (I certainly felt that the room was in agreement
>       - no queries, objection etc.)
>
>       I believe the following link-description provides the best
>       assimilation of all the choices that have been put forth. If we
>       step back and restart the whole discussion again we are likely
>       to come back to the following - so here is a brief compilation
>       of issues.
>
>
>        Suggested Link description to be used in address resolution:
>
>                LID + capability flags + GID + QPN
>
>
>
>       Why the LID?
>
>             People said they don't want to go to the SM if possible
>             (LID allows for that. As noted here and LWG the SL can be
>             determined based on QoS or other parameters. The SL may
>             also be determined from the SA/SM)
>
>             People said they want the process to end in small number
>             of messages (LID allows a query/response to settle the
>             issue. As noted one doesn't necessarily have to go to the
>             SM to get the SL).
>
>             LID is always needed. (you can't escape it).
>
>             BTW, people said they want the 'link-address' to be 48 bits.
>             (LID + QPN + cap_flags is 48 bits).
>
>
>       Why the GID ?
>
>                People want IP to span across IB subnets. We are
>                stretching stretching here since such implementations
>                are not possible yet.
>                However, we can define it such that it will work in
>                the future. Therefore include the GID.
>
>        GUID
>
>                Can be made part of the GID.
>
>
>        Why the QPN:
>
>                1) There is only one other way to get the QPN. That is
>                the SIDR protocol. However, this has a drawback.
>
>                SIDR is NOT REQUIRED by IBTA.
>
>                I had raised this point last August 'Feasibility of
>                SIDR':
>
>                ***** Section 12.11 essentially states that SIDR is
>                optional. As the specification states:
>
>               << GSAs shall support this protocol if non-management
>               services are pro-vided on the Channel Adapter at other
>               than fixed QPNs. If this protocol is *not supported*,
>               the status "The method/attribute combination is not
>               sup-ported" shall be returned, as described in section
>               Table 101 MAD Common Status Field Bit Values >>
>
>                *****
>
>                That still holds. SIDR is NOT required by IBTA. A
>                device can always choose to provide all non-management
>                services at fixed QPNs. It may just choose to provide
>                IP and nothing else at a particular QP and hence won't
>                do SIDR.
>
>                That is why I propose that the QPN be always included
>                for the default case. SIDR usage is gated through the
>                QPN flag and is in the realm of the advanced
>                capabilities draft as has been suggested by many on
>                this mailing list. And I agree.
>
>
>                2) Address resolution that returns the QPN is done in
>                one request/reply exchange. Any other method, while
>                talking to the same endpoint, requires 6 messages. 2
>                for ARP/ND and 2 each for each of the lower layer
>                request/reply. IT is wasteful, more complicated,
>                more time consuming wrt fast failover etc.
>
>
>        Capability flags :
>
>                Multiple people have opined that they would prefer in
>                certain cases to map the protocol/ULP endpoints to
>                specific QPs. People also want to use IP over RC mode
>                (and others maybe) due to the advantages of larger MTU
>                and Automatic path failover etc.
>
>                The capability flags allow for interoperation between
>                the default/dumb devices and the more sophisticated
>                devices. IT also allows the endpoints to avoid
>                unnecessary trial and error/to and fro when trying to
>                utilise these features. The flag indicates the
>                advanced capabilities supported and the receiver can
>                choose to utilies them.
>
>
>Result:
>
>        We cannot depend on SIDR. We need to include the default QPN.
>        We should not force implementations to go to the SA/SM (IBTA
>        does not) and so allow LID/GID to be determined from the
>        remote host. The default case works with :
>
>        LID+Capability flags(all set to 0)+QPN+GID
>
>
>Vivek
>
>
>
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>http://www.ietf.org/mailman/listinfo/ipoverib
>


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 16:00:45 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA13205
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 16:00:44 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id QAA22443
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 16:00:47 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA22052;
	Tue, 15 Jan 2002 15:58:12 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA22001
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 15:58:09 -0500 (EST)
Received: from sentinel.sanera.net (ns1.sanera.net [208.253.254.10])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA13122
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 15:58:05 -0500 (EST)
Received: from icarus.sanera.net (icarus [192.168.254.11])
	by sentinel.sanera.net (8.11.2/8.11.2) with ESMTP id g0FKvbu21619
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 12:57:38 -0800
Received: from newton.sanera.net (newton.sanera.net [172.16.2.11])
	by icarus.sanera.net (8.11.6/8.11.6) with ESMTP id g0FKvWn20243
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 12:57:32 -0800
Received: from sanera.net (dhcp-pc-100-161.sanera.net [192.168.100.161])
	by newton.sanera.net (8.11.4/8.11.4) with ESMTP id g0FKvWM24668
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 12:57:32 -0800 (PST)
Message-ID: <3C4497BB.587C3AD1@sanera.net>
Date: Tue, 15 Jan 2002 12:57:31 -0800
From: Joe Pelissier <joep@Sanera.net>
X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U)
X-Accept-Language: en
MIME-Version: 1.0
To: ipoverib@ietf.org
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
References: <200201151958.g0FJwO408903@eng4.beaverton.ibm.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Sumarizing, yes I think we are in agreement.  Stating that the IB spec does not specify how one gets SL or static rate is a
simple statement of fact - I have no objection to this.  As you state in the architecture draft, "Every IPoIB implementation
will determine the relevant SL value based on its own policy. No method or process for choosing the SL will be defined by
the IPoIB standards."  I completely agree with this philosophy.  I think this makes any further discussion on whether the IB
specification or its creators intended one mode or the other moot.

Regards,
Joe Pelissier


Vivek Kashyap wrote:

> >
> > >
> > >       There are other reasons for avoiding the SM/SA that I  have mentioned
> > > previously. However, nowhere has it been suggested that SM/SA cannot be
> > > consulted. The counter-suggestion from you though says that one must choose
> > > to go to  SM/SA - that is something that IBTA itself doesn't agree with.
> > > Those that want to consult the SM/SA can continue to do so but those that
> > > don't want to because they certainly see advantages in avoiding SM/SA when
> > > possible should be allowed to do so; exactly as IBTA does.
> > >
> >
> > <JEP>  I agree that no one is suggesting prohibiting consultation with the SM/SA.  What I object to is the
> > recommendation in the draft against such consultation.  If this recommendation was simply removed and the draft was
> > silent on the topic, then I would have no objection.  I think I am correct in stating the you believe it works and
> > interoperates either way (which, btw, I do not believe), so why push so hard for the recommendation?  It seems clear
> > the consensus is unlikely, so lets compromise by not promoting nor prohibiting either approach in the draft and future
> > RFC.
>
> <VK> Joe, you have not been looking at the drafts latest drafts. There
> are two working group drafts out - IPoIB architecture and on
> multicast. Things have moved forward. Skip the old draft. It is no
> longer in consideration.
> <VK>
>
> >
> > >
> > >       Regarding your comments on SL - I am a member of IBTA. IBTA rules do
> > > not allow to one to give out the statements and the voting results  to
> > > non-IBTA forums, so I'll only state that IBTA did not change the
> > > specification - it is available to anyone on www.infinibandta.org  and did
> > > not modify the rules regarding SL determination as Mike Krause in the mail
> > > before yours has clarified. Mike is also a member(chair) of LWG.  Others
> > > who are members of the IBTA can verify for themselves the final decision of
> > > LWG. It did not preclude determining SL from sources other than the SA.
> > >
> >
> > <JEP>  You are correct about the rules, my apologies.  Yes, the specification does not require that one obtain the SL
> > from the SA and no public statement has been made by the IBTA regarding whether or not this is likely to change in the
> > future.
> >
> > >
> > >       You are right in stating that protection from a random SL is required
> > > but no where is it being suggested that the SL used by the endnodes will be
> > > random one. It will be a well thought of one and a correct one except it
> > > will be determined without consulting the SA/SM. Nothing in IBTA precludes
> > > it. Rather IBA specification promotes it.  BTW, the use SL from multicast
> > > or determination of it  in any other way has been left to the choice of the
> > > implementor. Not part of IPoIB architecture. Also, again, IBTA does not
> > > specify a particular method either.
> > >
> >
> > <JEP>  Hmmm, I'm not sure that the fact that the spec is silent on the topic means that it is promoting it.
> >
>
> <VK> It refers to service levels under QoS and talks a lot about that. It hardly mentions deadlocks for example.
> <VK>
>
> > >
> > >       All nodes if working greater than 1x are required to support static
> > > rate iff any destination in the fabric works at < the node's rate.  It
> > > merely says that the values needs to be determined but does not make the
> > > method for determining it mandatory. Consulting the SA to get the path
> > > record is one of the methods. Additionally IBTA is defining a congestion
> > > control mechanism disjoint from injection control.
> >
> > <JEP> A minor nit - all nodes that are operating at a rate greater than 1x are required to support static rate.  I
> > have not checked the spec in detail, but my guess is your probably right - it probably does not specify that one gets
> > the static rate from the SA.  However, as in the SL case, a mechanism is provided to get this information from the
> > SA.  So again, I grant you that you are not technically violating the specification if you use a static rate other
> > than that provided by the SA.  My only objection would be any language in a draft or RFC that specifically required or
> > recommended that one not consult the SA for this information.  Likewise, I am not proposing that the the draft or RFC
> > specifically prohibit or recommend against it.
>
> <VK> Nothing is suggested to be added to the draft. All I am trying to
> point out is that other solutions are possible. So let us not put it
> into the draft that it must be through a particular method i.e.
> consult the SA. I think we are in agreement to say that the method of
> determinig the values is upto the implementations.
>
> I am also suggesting that due to this reason and the reason that this
> information may be received at a different 'time period' one must not
> make it a reason to say that the LID must be got from the SA. That is
> my main point.
>
> <VK>
>
> > >       The suggestion here that some implementations will avoid the SA/SM in
> > > a particular subnet if they think it is right in no way precludes
> > > consulting the SA/SM or any other entity when required. That is not the
> > > contention at all.  It is not being mandated on everyone. Similarly there
> > > is no need to require every implementation to throw a look toward the SA/SM
> > > for SL/path rate or any other information.
> >
> > <JEP>  I am not asking that we require or recommend that every implementation throw a look the the SA/SM.  I am asking
> > that we do not require or recommend that they do not.
>
> <VK> We are in agreement then.
>
>         Determining such values is upto the implementation.
>         Determining the LID is a separate disjoint issue.
>
> Vivek
>
> >
> > >
> > >
> > > Vivek
> > >
> > > --
> > > Vivek Kashyap
> > > Linux Technology Center, IBM
> > > vivk@us.ibm.com
> > > kashyapv@us.ibm.com
> > > Ph: 503 578 3422 T/L: 775 3422
> > >
> > >
> > >                       Joe Pelissier
> > >                       <joep@Sanera.net>        To:       ipoverib@ietf.org
> > >                       Sent by:                 cc:
> > >                       ipoverib-admin@ie        Subject:  Re: [Ipoverib] What's inside an IPoIB link layer address?
> > >                       tf.org
> > >
> > >
> > >                       01/14/2002 08:35
> > >                       PM
> > >
> > >
> > >
> > > Greetings:
> > > I unfortunately have had to reduce my participation in this effort for a
> > > while,
> > > but have now been studying the email threads attempting to get back up to
> > > speed.
> > > My apologies in advance if my comments below are missing some context from
> > > previous disucssions.
> > >
> > > It seems that there continues to be a great deal of discussion as to
> > > whether ARP
> > > can be accomplished without consulting the SA (or at least cached
> > > information
> > > from a previous consultation).  There seems to be two major issues: static
> > > rate
> > > and service level.
> > >
> > > All nodes that operate at greater than x1 speed are required to support
> > > static
> > > rate.  The static rate setting is a function of the path between two
> > > endpoints.
> > > It is important to note that the path multicast uses is very likely to be
> > > quite
> > > different than the path that unicast traffic uses.  Therefore, the static
> > > rate
> > > may well be different.  The only way to get this information is to consult
> > > the SA
> > > (actually, one could always set their static rate to x1 speed and avoid the
> > > consultation, but if you got a x12 link, it seems a shame not to use it
> > > just to
> > > avoid talking to your friendly SA ;-).
> > >
> > > The other is SL.  It has repeatedly been argued that the spec does not
> > > require
> > > getting the SL from the SA.  This is true.  This has been recognized by the
> > > IBTA
> > > and steps are being taken to correct it.  In order to support multiple
> > > paths
> > > through the IB fabric, it was a fundamental assumption that the SA controls
> > > which
> > > SLs a node may use based on path.  The SA may provide a set of SLs from
> > > which a
> > > node may choose, but the SA needs to control this.  If it does not,
> > > deadlock of
> > > the fabric may occur.  To prevent deadlock, the SA can load certain IB
> > > tables
> > > that would cause the offending packets to be discarded.  This protects the
> > > fabric, but would cause communication to fail between the offending nodes.
> > >
> > > So we could argue all this forever, or maybe there is another solution.
> > > The IB
> > > spec already talks about how to determine both static rate and SL.
> > > Therefore, we
> > > don't need to rehash it in the RFC (other than maybe a note that says these
> > > are
> > > determined IAW the InfiniBand standard).
> > >
> > > wrt/ SL, this is pretty much what section 6.0 of
> > > draft-kashyap-ipoib-ipv4-and-arp-01.txt says, except for the last paragraph
> > > (which recommends using the SL from the broadcast group).  Unfortunately,
> > > the
> > > recommendation is directly contrary to the some intended usage of IB and is
> > > likely to either be prohibited or at least strongly discouraged by a future
> > > version of the specification.
> > >
> > > The LWG voted unanimously to recommend the addition of language that
> > > strongly
> > > discourages the use of parameters that were not obtained by an applicable
> > > path
> > > record and stating that the parameters from multicast and unicast should
> > > not be
> > > mixed (see the LWG meeting minutes dated 8/16/01).
> > >
> > > If we just remove the recommendation from the draft, then we can move
> > > forward
> > > without further discussion.  It just seems bazaar to continue to push
> > > something
> > > as "recommended" that causes great concern to many of the architects that
> > > created
> > > IB.  All we need to do is leave out the recommendation and we are done with
> > > this
> > > issue.  Those that wish to ignore the SL assignment from the SA and use the
> > > multicast one can still do so without technically violating the current
> > > IBTA
> > > spec.  There seems to be little downside to this.
> > >
> > > Besides, you need to get static rate anyway....
> > >
> > > Another $0.02 worth...
> > > Regards,
> > > Joe Pelissier
> > >
> > > Michael Krause wrote:
> > >
> > > > At 05:10 PM 1/14/2002 -0600, Ashwin Kamath wrote:
> > > > >There is a general thought that ARP is a one step solution for obtaining
> > > the
> > > > >parameters required for establishing IPoIB over UD. But I think that
> > > even
> > > > >for using ARP, one will require to get the Path record information. I do
> > > not
> > > > >think that ARP is a one stop solution. What happens if both nodes
> > > determine
> > > > >that their MTU is 2K without considering that a switch component in
> > > between
> > > > >might restrict the MTU to 256 bytes?
> > > >
> > > > The minimum supported MTU for UD traffic is 2KB so a path with a 256
> > > bytes
> > > > MTU should not be used by the IP over IB service.  The SM working with
> > > the
> > > > partition manager will establish a set of paths that can be used for IP
> > > > over IB traffic which will include the setting up of an all nodes
> > > multicast
> > > > group per partition that support IP traffic.
> > > >
> > > > >Such information can be obtained only
> > > > >from the central SM via a path record. Am I missing something here? If
> > > > >address resolution is all that ARP is required to do, without trying to
> > > > >establish parameters for communication one may not want to know the MTU,
> > > but
> > > > >from the discussions I see that the Address resolution is being used for
> > > > >establishing communication parameters (like the QPN)
> > > >
> > > > QP identification is an optimization that allows one to eliminate the
> > > > Ethertype field.  It also allows an implementation to map different IP
> > > > addresses to different QP thereby providing an implementation
> > > optimization
> > > > through more fine grain hardware-based de-multiplexing.  An example use
> > > > might be to assign an IP address per OS instance which shares a HCA port
> > > > and then allow all traffic to target a corresponding QP without having to
> > > > flow through an intermediate hypervisor that would have to de-multiplex
> > > the
> > > > traffic.  This can provide both performance and scalability improvements
> > > > for vendor offerings.
> > > >
> > > > >So the only real problem with having the LID in the ARP response, that I
> > > see
> > > > >is when the ARP response needs to cross IB Routers. Can I assume that
> > > > >changes in LID assignments are not a big problem because such changes
> > > are
> > > > >not intended to be frequent and current ARP cache implementations
> > > already
> > > > >factor in the possibility of a link level address change. Changing LIDs
> > > > >would have a bigger impact on shared devices using RC than on IPoIB
> > > using
> > > > >UD. Is it ok if I consider a change in LID assignment to be equivalent
> > > of a
> > > > >server reboot?
> > > >
> > > > No.  Applications should not be impacted by a change in LID assignment as
> > > > there are defined methods to migrate applications from one LID or GID or
> > > > port to another that are completely transparent to the application.  For
> > > > UD-based services, one could simply re-ARP the endnode via the all-nodes
> > > > multicast and provide a simple update of the LID mapping as this can be
> > > > derived from the IB headers if desired.
> > > >
> > > > >If I understood what Jerry, one will have a problem if the ARP responder
> > > > >lies in another IB Subnet, in which case the LID in the response will
> > > not be
> > > > >relavant to the originator of the ARP request, in-fact the LID will have
> > > to
> > > > >be the LID of the router. Was there any resolution reached for this
> > > problem?
> > > > >
> > > > >In short you do need an ARP + Path record diagnosis to determine all the
> > > > >parameters for communication. What would happen if we bring an IB router
> > > in
> > > > >between? One cannot obtain a path record to a node in another subnet.
> > > One
> > > > >can obtain the path record to the IB Router. Should we be talking about
> > > the
> > > > >behaviour of the IB router in case of mismatch of MTUs or should we say
> > > that
> > > > >we will always use the smallest mtu for communicating across an IB
> > > router.
> > > > >Or should we say that this could fall on the plate of the Infiniband
> > > WGs.
> > > >
> > > > The IB router protocol will define how one routes across IB subnets - an
> > > > IBTA issue.  An IP router may be implemented along side an IB router or
> > > it
> > > > may be an edge device such as a multi-protocol router.  The fact that IP
> > > > subnets may span multiple IB subnets isn't really that big of an issue
> > > per
> > > > se.  By keeping the LID not part of the link address structure, the ARP
> > > > should be able to transparently flow across multiple IB subnets.
> > > >
> > > > Mike
> > > >
> > > > _______________________________________________
> > > > IPoverIB mailing list
> > > > IPoverIB@ietf.org
> > > > http://www.ietf.org/mailman/listinfo/ipoverib
> > >
> > > _______________________________________________
> > > IPoverIB mailing list
> > > IPoverIB@ietf.org
> > > http://www.ietf.org/mailman/listinfo/ipoverib
> > >
> > > _______________________________________________
> > > IPoverIB mailing list
> > > IPoverIB@ietf.org
> > > http://www.ietf.org/mailman/listinfo/ipoverib
> >
> >
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> >
>
> --
> Vivek Kashyap
> Linux Technology Center, IBM
> kashyapv@us.ibm.com
> vivk@us.ibm.com
> 503 578 3422 (o)
>
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 16:04:21 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA13332
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 16:04:21 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id QAA22573
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 16:04:24 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA22525;
	Tue, 15 Jan 2002 16:03:23 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA22493
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 16:03:22 -0500 (EST)
Received: from e32.co.us.ibm.com (e32.co.us.ibm.com [32.97.110.130])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA13296
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 16:03:17 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e32.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id QAA166874;
	Tue, 15 Jan 2002 16:00:17 -0500
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0FL3Hr106320;
	Tue, 15 Jan 2002 14:03:17 -0700
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0FL3FV21374;
	Tue, 15 Jan 2002 13:03:15 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0FL3ET12642;
	Tue, 15 Jan 2002 13:03:14 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201152103.g0FL3ET12642@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
To: joep@Sanera.net (Joe Pelissier)
Date: Tue, 15 Jan 2002 13:03:13 -0800 (PST)
Cc: ipoverib@ietf.org
In-Reply-To: <3C4497BB.587C3AD1@sanera.net> from "Joe Pelissier" at Jan 15, 2002 11:57:31 AM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

I would like to add to the summary that since the mode of determinig
these values is upto the implementation there is no link between
stating that these values have to be got from SA and therefore LID can
be got from there too. The two are disjoint issues.

Vivek


> 
> Sumarizing, yes I think we are in agreement.  Stating that the IB spec does not specify how one gets SL or static rate is a
> simple statement of fact - I have no objection to this.  As you state in the architecture draft, "Every IPoIB implementation
> will determine the relevant SL value based on its own policy. No method or process for choosing the SL will be defined by
> the IPoIB standards."  I completely agree with this philosophy.  I think this makes any further discussion on whether the IB
> specification or its creators intended one mode or the other moot.
> 
> Regards,
> Joe Pelissier
> 
> 
> Vivek Kashyap wrote:
> 
> > >
> > > >
> > > >       There are other reasons for avoiding the SM/SA that I  have mentioned
> > > > previously. However, nowhere has it been suggested that SM/SA cannot be
> > > > consulted. The counter-suggestion from you though says that one must choose
> > > > to go to  SM/SA - that is something that IBTA itself doesn't agree with.
> > > > Those that want to consult the SM/SA can continue to do so but those that
> > > > don't want to because they certainly see advantages in avoiding SM/SA when
> > > > possible should be allowed to do so; exactly as IBTA does.
> > > >
> > >
> > > <JEP>  I agree that no one is suggesting prohibiting consultation with the SM/SA.  What I object to is the
> > > recommendation in the draft against such consultation.  If this recommendation was simply removed and the draft was
> > > silent on the topic, then I would have no objection.  I think I am correct in stating the you believe it works and
> > > interoperates either way (which, btw, I do not believe), so why push so hard for the recommendation?  It seems clear
> > > the consensus is unlikely, so lets compromise by not promoting nor prohibiting either approach in the draft and future
> > > RFC.
> >
> > <VK> Joe, you have not been looking at the drafts latest drafts. There
> > are two working group drafts out - IPoIB architecture and on
> > multicast. Things have moved forward. Skip the old draft. It is no
> > longer in consideration.
> > <VK>
> >
> > >
> > > >
> > > >       Regarding your comments on SL - I am a member of IBTA. IBTA rules do
> > > > not allow to one to give out the statements and the voting results  to
> > > > non-IBTA forums, so I'll only state that IBTA did not change the
> > > > specification - it is available to anyone on www.infinibandta.org  and did
> > > > not modify the rules regarding SL determination as Mike Krause in the mail
> > > > before yours has clarified. Mike is also a member(chair) of LWG.  Others
> > > > who are members of the IBTA can verify for themselves the final decision of
> > > > LWG. It did not preclude determining SL from sources other than the SA.
> > > >
> > >
> > > <JEP>  You are correct about the rules, my apologies.  Yes, the specification does not require that one obtain the SL
> > > from the SA and no public statement has been made by the IBTA regarding whether or not this is likely to change in the
> > > future.
> > >
> > > >
> > > >       You are right in stating that protection from a random SL is required
> > > > but no where is it being suggested that the SL used by the endnodes will be
> > > > random one. It will be a well thought of one and a correct one except it
> > > > will be determined without consulting the SA/SM. Nothing in IBTA precludes
> > > > it. Rather IBA specification promotes it.  BTW, the use SL from multicast
> > > > or determination of it  in any other way has been left to the choice of the
> > > > implementor. Not part of IPoIB architecture. Also, again, IBTA does not
> > > > specify a particular method either.
> > > >
> > >
> > > <JEP>  Hmmm, I'm not sure that the fact that the spec is silent on the topic means that it is promoting it.
> > >
> >
> > <VK> It refers to service levels under QoS and talks a lot about that. It hardly mentions deadlocks for example.
> > <VK>
> >
> > > >
> > > >       All nodes if working greater than 1x are required to support static
> > > > rate iff any destination in the fabric works at < the node's rate.  It
> > > > merely says that the values needs to be determined but does not make the
> > > > method for determining it mandatory. Consulting the SA to get the path
> > > > record is one of the methods. Additionally IBTA is defining a congestion
> > > > control mechanism disjoint from injection control.
> > >
> > > <JEP> A minor nit - all nodes that are operating at a rate greater than 1x are required to support static rate.  I
> > > have not checked the spec in detail, but my guess is your probably right - it probably does not specify that one gets
> > > the static rate from the SA.  However, as in the SL case, a mechanism is provided to get this information from the
> > > SA.  So again, I grant you that you are not technically violating the specification if you use a static rate other
> > > than that provided by the SA.  My only objection would be any language in a draft or RFC that specifically required or
> > > recommended that one not consult the SA for this information.  Likewise, I am not proposing that the the draft or RFC
> > > specifically prohibit or recommend against it.
> >
> > <VK> Nothing is suggested to be added to the draft. All I am trying to
> > point out is that other solutions are possible. So let us not put it
> > into the draft that it must be through a particular method i.e.
> > consult the SA. I think we are in agreement to say that the method of
> > determinig the values is upto the implementations.
> >
> > I am also suggesting that due to this reason and the reason that this
> > information may be received at a different 'time period' one must not
> > make it a reason to say that the LID must be got from the SA. That is
> > my main point.
> >
> > <VK>
> >
> > > >       The suggestion here that some implementations will avoid the SA/SM in
> > > > a particular subnet if they think it is right in no way precludes
> > > > consulting the SA/SM or any other entity when required. That is not the
> > > > contention at all.  It is not being mandated on everyone. Similarly there
> > > > is no need to require every implementation to throw a look toward the SA/SM
> > > > for SL/path rate or any other information.
> > >
> > > <JEP>  I am not asking that we require or recommend that every implementation throw a look the the SA/SM.  I am asking
> > > that we do not require or recommend that they do not.
> >
> > <VK> We are in agreement then.
> >
> >         Determining such values is upto the implementation.
> >         Determining the LID is a separate disjoint issue.
> >
> > Vivek
> >
> > >
> > > >
> > > >
> > > > Vivek
> > > >
> > > > --
> > > > Vivek Kashyap
> > > > Linux Technology Center, IBM
> > > > vivk@us.ibm.com
> > > > kashyapv@us.ibm.com
> > > > Ph: 503 578 3422 T/L: 775 3422
> > > >
> > > >
> > > >                       Joe Pelissier
> > > >                       <joep@Sanera.net>        To:       ipoverib@ietf.org
> > > >                       Sent by:                 cc:
> > > >                       ipoverib-admin@ie        Subject:  Re: [Ipoverib] What's inside an IPoIB link layer address?
> > > >                       tf.org
> > > >
> > > >
> > > >                       01/14/2002 08:35
> > > >                       PM
> > > >
> > > >
> > > >
> > > > Greetings:
> > > > I unfortunately have had to reduce my participation in this effort for a
> > > > while,
> > > > but have now been studying the email threads attempting to get back up to
> > > > speed.
> > > > My apologies in advance if my comments below are missing some context from
> > > > previous disucssions.
> > > >
> > > > It seems that there continues to be a great deal of discussion as to
> > > > whether ARP
> > > > can be accomplished without consulting the SA (or at least cached
> > > > information
> > > > from a previous consultation).  There seems to be two major issues: static
> > > > rate
> > > > and service level.
> > > >
> > > > All nodes that operate at greater than x1 speed are required to support
> > > > static
> > > > rate.  The static rate setting is a function of the path between two
> > > > endpoints.
> > > > It is important to note that the path multicast uses is very likely to be
> > > > quite
> > > > different than the path that unicast traffic uses.  Therefore, the static
> > > > rate
> > > > may well be different.  The only way to get this information is to consult
> > > > the SA
> > > > (actually, one could always set their static rate to x1 speed and avoid the
> > > > consultation, but if you got a x12 link, it seems a shame not to use it
> > > > just to
> > > > avoid talking to your friendly SA ;-).
> > > >
> > > > The other is SL.  It has repeatedly been argued that the spec does not
> > > > require
> > > > getting the SL from the SA.  This is true.  This has been recognized by the
> > > > IBTA
> > > > and steps are being taken to correct it.  In order to support multiple
> > > > paths
> > > > through the IB fabric, it was a fundamental assumption that the SA controls
> > > > which
> > > > SLs a node may use based on path.  The SA may provide a set of SLs from
> > > > which a
> > > > node may choose, but the SA needs to control this.  If it does not,
> > > > deadlock of
> > > > the fabric may occur.  To prevent deadlock, the SA can load certain IB
> > > > tables
> > > > that would cause the offending packets to be discarded.  This protects the
> > > > fabric, but would cause communication to fail between the offending nodes.
> > > >
> > > > So we could argue all this forever, or maybe there is another solution.
> > > > The IB
> > > > spec already talks about how to determine both static rate and SL.
> > > > Therefore, we
> > > > don't need to rehash it in the RFC (other than maybe a note that says these
> > > > are
> > > > determined IAW the InfiniBand standard).
> > > >
> > > > wrt/ SL, this is pretty much what section 6.0 of
> > > > draft-kashyap-ipoib-ipv4-and-arp-01.txt says, except for the last paragraph
> > > > (which recommends using the SL from the broadcast group).  Unfortunately,
> > > > the
> > > > recommendation is directly contrary to the some intended usage of IB and is
> > > > likely to either be prohibited or at least strongly discouraged by a future
> > > > version of the specification.
> > > >
> > > > The LWG voted unanimously to recommend the addition of language that
> > > > strongly
> > > > discourages the use of parameters that were not obtained by an applicable
> > > > path
> > > > record and stating that the parameters from multicast and unicast should
> > > > not be
> > > > mixed (see the LWG meeting minutes dated 8/16/01).
> > > >
> > > > If we just remove the recommendation from the draft, then we can move
> > > > forward
> > > > without further discussion.  It just seems bazaar to continue to push
> > > > something
> > > > as "recommended" that causes great concern to many of the architects that
> > > > created
> > > > IB.  All we need to do is leave out the recommendation and we are done with
> > > > this
> > > > issue.  Those that wish to ignore the SL assignment from the SA and use the
> > > > multicast one can still do so without technically violating the current
> > > > IBTA
> > > > spec.  There seems to be little downside to this.
> > > >
> > > > Besides, you need to get static rate anyway....
> > > >
> > > > Another $0.02 worth...
> > > > Regards,
> > > > Joe Pelissier
> > > >
> > > > Michael Krause wrote:
> > > >
> > > > > At 05:10 PM 1/14/2002 -0600, Ashwin Kamath wrote:
> > > > > >There is a general thought that ARP is a one step solution for obtaining
> > > > the
> > > > > >parameters required for establishing IPoIB over UD. But I think that
> > > > even
> > > > > >for using ARP, one will require to get the Path record information. I do
> > > > not
> > > > > >think that ARP is a one stop solution. What happens if both nodes
> > > > determine
> > > > > >that their MTU is 2K without considering that a switch component in
> > > > between
> > > > > >might restrict the MTU to 256 bytes?
> > > > >
> > > > > The minimum supported MTU for UD traffic is 2KB so a path with a 256
> > > > bytes
> > > > > MTU should not be used by the IP over IB service.  The SM working with
> > > > the
> > > > > partition manager will establish a set of paths that can be used for IP
> > > > > over IB traffic which will include the setting up of an all nodes
> > > > multicast
> > > > > group per partition that support IP traffic.
> > > > >
> > > > > >Such information can be obtained only
> > > > > >from the central SM via a path record. Am I missing something here? If
> > > > > >address resolution is all that ARP is required to do, without trying to
> > > > > >establish parameters for communication one may not want to know the MTU,
> > > > but
> > > > > >from the discussions I see that the Address resolution is being used for
> > > > > >establishing communication parameters (like the QPN)
> > > > >
> > > > > QP identification is an optimization that allows one to eliminate the
> > > > > Ethertype field.  It also allows an implementation to map different IP
> > > > > addresses to different QP thereby providing an implementation
> > > > optimization
> > > > > through more fine grain hardware-based de-multiplexing.  An example use
> > > > > might be to assign an IP address per OS instance which shares a HCA port
> > > > > and then allow all traffic to target a corresponding QP without having to
> > > > > flow through an intermediate hypervisor that would have to de-multiplex
> > > > the
> > > > > traffic.  This can provide both performance and scalability improvements
> > > > > for vendor offerings.
> > > > >
> > > > > >So the only real problem with having the LID in the ARP response, that I
> > > > see
> > > > > >is when the ARP response needs to cross IB Routers. Can I assume that
> > > > > >changes in LID assignments are not a big problem because such changes
> > > > are
> > > > > >not intended to be frequent and current ARP cache implementations
> > > > already
> > > > > >factor in the possibility of a link level address change. Changing LIDs
> > > > > >would have a bigger impact on shared devices using RC than on IPoIB
> > > > using
> > > > > >UD. Is it ok if I consider a change in LID assignment to be equivalent
> > > > of a
> > > > > >server reboot?
> > > > >
> > > > > No.  Applications should not be impacted by a change in LID assignment as
> > > > > there are defined methods to migrate applications from one LID or GID or
> > > > > port to another that are completely transparent to the application.  For
> > > > > UD-based services, one could simply re-ARP the endnode via the all-nodes
> > > > > multicast and provide a simple update of the LID mapping as this can be
> > > > > derived from the IB headers if desired.
> > > > >
> > > > > >If I understood what Jerry, one will have a problem if the ARP responder
> > > > > >lies in another IB Subnet, in which case the LID in the response will
> > > > not be
> > > > > >relavant to the originator of the ARP request, in-fact the LID will have
> > > > to
> > > > > >be the LID of the router. Was there any resolution reached for this
> > > > problem?
> > > > > >
> > > > > >In short you do need an ARP + Path record diagnosis to determine all the
> > > > > >parameters for communication. What would happen if we bring an IB router
> > > > in
> > > > > >between? One cannot obtain a path record to a node in another subnet.
> > > > One
> > > > > >can obtain the path record to the IB Router. Should we be talking about
> > > > the
> > > > > >behaviour of the IB router in case of mismatch of MTUs or should we say
> > > > that
> > > > > >we will always use the smallest mtu for communicating across an IB
> > > > router.
> > > > > >Or should we say that this could fall on the plate of the Infiniband
> > > > WGs.
> > > > >
> > > > > The IB router protocol will define how one routes across IB subnets - an
> > > > > IBTA issue.  An IP router may be implemented along side an IB router or
> > > > it
> > > > > may be an edge device such as a multi-protocol router.  The fact that IP
> > > > > subnets may span multiple IB subnets isn't really that big of an issue
> > > > per
> > > > > se.  By keeping the LID not part of the link address structure, the ARP
> > > > > should be able to transparently flow across multiple IB subnets.
> > > > >
> > > > > Mike
> > > > >
> > > > > _______________________________________________
> > > > > IPoverIB mailing list
> > > > > IPoverIB@ietf.org
> > > > > http://www.ietf.org/mailman/listinfo/ipoverib
> > > >
> > > > _______________________________________________
> > > > IPoverIB mailing list
> > > > IPoverIB@ietf.org
> > > > http://www.ietf.org/mailman/listinfo/ipoverib
> > > >
> > > > _______________________________________________
> > > > IPoverIB mailing list
> > > > IPoverIB@ietf.org
> > > > http://www.ietf.org/mailman/listinfo/ipoverib
> > >
> > >
> > > _______________________________________________
> > > IPoverIB mailing list
> > > IPoverIB@ietf.org
> > > http://www.ietf.org/mailman/listinfo/ipoverib
> > >
> >
> > --
> > Vivek Kashyap
> > Linux Technology Center, IBM
> > kashyapv@us.ibm.com
> > vivk@us.ibm.com
> > 503 578 3422 (o)
> >
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 16:09:12 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA13421
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 16:09:12 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id QAA22673
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 16:09:15 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA22652;
	Tue, 15 Jan 2002 16:07:50 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA22618
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 16:07:48 -0500 (EST)
Received: from e23.esmtp.ibm.com (e23.nc.us.ibm.com [32.97.136.229])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA13406
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 16:07:44 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e23.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id PAA60868;
	Tue, 15 Jan 2002 15:04:44 -0600
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0FL7he140364;
	Tue, 15 Jan 2002 16:07:43 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id g0FL7hg23046;
	Tue, 15 Jan 2002 13:07:43 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0FL7gw13021;
	Tue, 15 Jan 2002 13:07:42 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201152107.g0FL7gw13021@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
To: Jerry.Chu@eng.sun.com
Date: Tue, 15 Jan 2002 13:07:41 -0800 (PST)
Cc: ipoverib@ietf.org
In-Reply-To: <200201130528.g0D5S9R8812864@jurassic.eng.sun.com> from "H.K. Jerry Chu" at Jan 12, 2002 08:28:29 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit


> 
> What I don't like to see included is LID.
> 
> Discussions:
> 
> There has been a consensus of using GID, not LID as the basic

I don't think there has been any consensus on this issue.

> IB address identifying the port hosting an IP address. GID is
> sufficiently stable, not subject to change across a node reboot
> or subnet merge (IB level) like LID is. Also people don't want
> to exclude the possibility of IPoIB link spanning more than
> one IB subnet. GID supports scope beyond local link so it's
> suitable for that purpose.
> 
> Multiple GIDs with different scopes may have been assigned to a CA.
> It is required that a CA returns the GID with a scope no less than
> the one for the IPoIB link. E.g. if an IPoIB link covers multiple
> IB subnets, but a CA uses a link-local GID as part of its IPoIB
> link layer address, it's not going to work well.

        Let us look into this issue in detail first. It has not been
        pointed out what one gains by spanning IB subnets. Let us
        first make or break the case for IP subnets spanning IB
        subnets. What is the gain ?

        As for things being stable - GID scopes, GID prefix changes,
        subnet merges etc. effect GIDs as much as LIDs. Once the node
        is up it will have a constant LID. SMs are also likely to keep
        the LID constant across reboots. But one point to note is that
        LIDs are always necessary. Whether they change or not you need
        to determine them. GIDs are not necessary. So this leaves us
        neither here nor there. They are just link addresses with the
        LID always required and GID an optional value.

> 
> I don't see any value of including LID besides GID as proposed
> by Vivek. I believes a lookup of PathRecord is always needed
> in order for a node to retrieve complete path info before sending

        As noted elsewhere there is no need to to always access the
        pathrecord. Even if the information is required it doesn't
        mean that one needs to determine the information from the path
        record when LID is to be determiend. Determining the LID is
        not tied to that. They are disjoint processes. Also, one can
        look up the path record using the LID/GID or GUID.


> unicast pkts to another node. The destination LID is part of
> the PathRecord so including it in the link layer address returned
> from ARP/ND doesn't save work. But you have all the drawbacks that
> swayed people away from using LID - it adds volatility to the link
> address. LID changing due to subnet merge or other reasons requires
> potentially new mechanism being added to IP/driver implementation
> to trigger ARP/ND flush. Lastly, what should be the LID if the IP
> node is in a different IB subnet? Obviously it can't be the LID of
> an intermediate IB router.

LID changing or GID changing (in scope in case of merger or
renumbering) etc. are issues that will have other effects and are
equally connected to any flushing or more drastic effects that will
have on the SM and the subnet. IBTA has not solved SM merges, P_Key
merges and management and various such issues. Such a disruption
applies all over IB components and is not limited to LIDs/GIDs.

Let us first answer the question of why IPoIB subnets must span IB
subnets ? We don't know why we need such a setup ? We don't know when
such a setup will be possible ? We do know that IPoIB subnets within
an IB subnets can be setup easily with the current specification.


Secondly, if you step back you'll realise that LID is hardly an issue
when resolving across subnets. Unlike what has been postulated the
process is not the same when resolving within the IB subnet and when
resolving across the IB subnet. 

	Assumption: at IP level- a multicast request; receive a unicast reply

        Now, when sending the broadcast the sender doesn't know where
        the responder is. So he includes his LID/GID. The receiver
        gets it across multiple subnets. It gets sees a different
        GID-prefix and hence replaces the LID with its default router
        when recording the address. The same happens on the response.

        It works fine for both situations - IPoIB subnets
        within/across IB subnets. You can easily have it that if the
        GID is local the routing lookup will return the local LID.
        Single process.

        In the case of not including the LID you do the same thing. If
        the GID prefix is local you go to the SM/SA to get the LID. If
        the GID prefix is different you record your default router's
        LID. So the process for across the subnet is the same but you
        shortchange the implementation that is more likely - the one
        whithin the subnet.

        For within the subnet you use a different method - going to
        The SA always. Not the same as going to the routing
        infrastructure. 

However, I would still like to know what is the use of IPoIB spanning
multiple subnets. Why do you think that is a requirement ?


Vivek
-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)
-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 16:36:40 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA14008
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 16:36:35 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id QAA23527
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 16:36:39 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA23503;
	Tue, 15 Jan 2002 16:34:58 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA23452
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 16:34:56 -0500 (EST)
Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA13972
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 16:34:52 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id PAA90474;
	Tue, 15 Jan 2002 15:30:42 -0600
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0FLYre10988;
	Tue, 15 Jan 2002 16:34:53 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id g0FLYqg23610;
	Tue, 15 Jan 2002 13:34:52 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0FLYma14366;
	Tue, 15 Jan 2002 13:34:48 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201152134.g0FLYma14366@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] IPoIB architecture/encapsulation (EtherType or
To: krause@cup.hp.com (Michael Krause)
Date: Tue, 15 Jan 2002 13:34:47 -0800 (PST)
Cc: ipoverib@ietf.org
In-Reply-To: <5.1.0.14.2.20020114142357.029b58f0@esalpha2.cup.hp.com> from "Michael Krause" at Jan 14, 2002 03:37:08 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit


> 
> Link address fields:
> 
> P_Key - For each partition that an endnode participates and requires IP 
> over IB services, it will send an ARP request to identify other nodes on 
> the IP subnet.  Only those endnodes that share a common P_Key will respond 
> and thus be discovered.  Hence, all discussion of P_Key is irrelevant from 
> the IP over IB perspective and should not be included within the IP over IB 
> specification.

        Agreed except to ensure unique MGIDs within partitions we have
        had to include the P_Key into the IP multicast to MGID
        mapping. 

> 
> Q_Key - Q_Key provides no value to IP over IB solutions and should be 
> treated as a well-defined value by the specification.  Recommend a value of 
> 0xFFFFFFFE be used.

        This is left to the admin. The value set in the broadcast-GID
        is used for all communication.

> 
> GUID - The value of a GUID is primarily being ascertain how things are 
> co-located on a given CA.  At first glance this seems to have some value 
> but most people no longer think it is required to establish a communication 
> channel between endnodes, in other words, use the GID to define the port 
> and let software figure out any co-location using local means if 
> required.  Hence, this field should not be included for IP over IB operations.
> 
> GID - In reality there are two ways of creating GIDs within IB (the 
> original IP over IB architecture discussion isn't quite accurate). The 
> first (called GID 0) and is created by concatenating the port's EUI64 with 
> the GID prefix to create a 128-bit value.  The second by concatenating a 
> SM-assigned 64-bit value with the GID prefix.  This second method provides 
> a way for endnodes to have a variety of link addresses (e.g. one per OS 
> instance, per set of IP addresses per endnode, etc.).  

        You can create any form of interface within your machine and
        tie it to any number of link addresses; these can be LIDs from
        different ports. Whatever GID is chosen it is finally tied to
        a port and the LID needs to be determined. Having different
        link addresses don't provide any gain in themselves if they ar
        tied to the same port. You can differentiate more easily by
        using QPs. If the ports are different then use the LIDs
        anyway. 


> In general, given an 
> IP subnet may span multiple IB subnets, this is the mandatory value to 
> exchange to uniquely identify an endnode for address resolution purposes.

        Why do we need to span multiple IB subnets? The fabric is
        broken into IB subnets for isolation. Why should we want to
        rejoin them when they are separated by design ? 

> 
> QP - There is some benefit to exchanging the QP as part of address 
> resolution.  For example, an endnode may have different policies in terms 
> of how a QP is mapped per IP subnet.  One might have one QP per IP address 
> or may have one QP per multiple IP addresses (potentially for all IP 
> communications).  As such, the ARP process should exchange the QP where it 
> is known and should use QP 0xFFFFFF when it is unknown, e.g. an IP router 
> at the edge of an IB fabric instance.

        Any endpoint that will talk IP will have a default QP. It
        returns that. I am not sure how it cannot know its own QP.

> 
> LID - There has been general discussion that IP subnets should be able to 
> span multiple IB subnets.  As such, the LID is invisible to the address 
> resolution process since this is an IB subnet only issue (whether it refers 
> to an IB router or an IB endnode is irrelevant).  The counter arguments 
> that one could optimize the address resolution process slightly by avoiding 
> a trip to the SA / SM by including the LID as a visible exchanged item (it 
> is always available since this can be derived from the IB packet in the 
> first place) really don't provide much substance.  The resolution process 
> should be able to resolve an address independent of the underlying IB 
> fabric using a single algorithm.  Implementations can take appropriate 
> steps to optimize their communications through smart caching and 
> examination of the provided IB headers.

        As discussed in the earlier mails these issues are important.
        The argument is that the method to determine any other
        information is upto the end node's implementation. Determining
        LID is disjoint from that. Those that don't want to use the
        LID in the adress resolution packets can always ignore the
        value on input and consult the SA.

        The process when determining the LID when across a subnet *is*
        different by definition. In the case of resolving the LID
        within the subnet the LID is received from the peer/SA but
        when communicating across IB subnets the LID of the first hop
        router is received from the routing infrastructure. A common
        method is not possible.

> 
> GRH - There has been some discussion of whether the GRH should be included 
> for all ARP operations or not.  For an ARP request, one injects a multicast 
> packet per partition.  The receiving endnode will receive the packet and 
> can determine whether the source is on the same subnet or not and respond 
> with either a GRH or a LRH-only packet.   As such, it seems GRH usage is 
> implicit and the current IB specifications on usage do not require any 
> additional work within the IP over IB specification.

	Agreed.

> 
> Capability bits - If an IP router is allowed to exist between two endnodes, 
> then the specification must include a method to exchange capabilities (e.g. 
> additional IB transport services that may be used to for IP over IB 
> communications).   Since this topology is allowed, the specification must 
> provide a method to exchange capabilities through some form of standalone 
> operation, i.e. an endnode must be able to resolve an address to an IP 
> router. The endnode will then inject a capabilities enumeration message to 
> the destination endnode.   Given the endnode must be able to perform this 
> operation to guarantee interoperability separate from the address 
> resolution process, it would be preferred if the capability discovery (only 
> required if an endnode provides additional services beyond IP over IB which 
> is not necessarily universal) be kept a separate operation.  Once the 
> capabilities are determined, these can be cached for a fairly substantial 
> amount of time such that there is no performance loss for the majority of 
> the operations that may occur between endnodes.  In addition, the 
> capabilities exchange may be used to provide a target QP to be used for 
> each capability thus allowing one to optimize the communications to 
> initiate / implementations of these capabilities.

        Please see the
        draft-kashyap-ipoib-advanced-capabilities-00.txt. I think I
        have taken good care of interoperability. The default case
        which doesn't support RC/QPN modes work seamlessly with nodes
        that do.

> 
> Ethertype - Quite a bit of discussion has occurred but it may be fair to 
> state that the address resolution provides a first level method to obtain 
> the associated QP for a given IP address and that the capabilities exchange 
> may be used to provide additional QP for a given capability (e.g. perform a 
> RC establishment using QP X), it seems the Ethertype is no longer necessary 
> and can be removed from the specification.  
> 

	Agreed.

Vivek

-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)
-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 16:39:10 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA14054
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 16:39:01 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id QAA23618
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 16:39:05 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA23588;
	Tue, 15 Jan 2002 16:38:07 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA23554
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 16:38:01 -0500 (EST)
Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA14031
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 16:37:57 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id PAA21420;
	Tue, 15 Jan 2002 15:33:47 -0600
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0FLbte60760;
	Tue, 15 Jan 2002 16:37:56 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id g0FLbsg23690;
	Tue, 15 Jan 2002 13:37:54 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0FLbod14559;
	Tue, 15 Jan 2002 13:37:50 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201152137.g0FLbod14559@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
To: kamath@omegaband.com (Ashwin Kamath)
Date: Tue, 15 Jan 2002 13:37:49 -0800 (PST)
Cc: ipoverib@ietf.org
In-Reply-To: <IFEKJBDLPFOEGGPIIGMFOECKCAAA.kamath@omegaband.com> from "Ashwin Kamath" at Jan 14, 2002 05:45:51 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Actually I'll ask you to read the ipoib-architecture and multicast
drafts that hve been posted. They should answer your questions. If not
then ask them and we will update the drafts accordingly.

Vivek

> 
> 
> Vivek,
> 
> Is the assumption true that the MTUs established for switching multicast
> packets will be the same as that used for Unicast switching? ie will
> multicast packets from one end node to another traverse the same path as a
> unicast packet? I personally would not try to configure a path which uses a
> higher MTU for a multicast group vs a lower MTU for other path records, but
> in general the SM is not restricted by this requirement. ( It is possible
> that the SM chooses 4x 1K MTU links for normal data and 1x 4K mtu links for
> Multicast data, if by some chance, the topology forces it to do so. If some
> switches do not support multicast and atleast one path per node exists such
> that multicast can be supported for the fabric as a whole, the end-to-end
> paths for the unicast packets may differ from that of multicast packets for
> any given pair of nodes and hence the mtus can be different, more
> specifically, the mtus can be larger for multicast than for unicast. ) If it
> is true that multicast and unicast packets traverse the same paths
> end-to-end or if it is reasonable to assume the MTU to be that of the
> Multicast group because that is the best possible configuration, I do not
> see any issue, except that such a statement of restriction be made on the
> subnet management for use with ipoib. I think such a restriction is
> reasonable for operation of IPoIB though I do not know if it is essential,
> neither can I offer an alternative at this point. Do you think that such a
> statement of restriction is required?
> 
>  What about the maximum static rate of injection which is part of the path
> record? This parameter is not part of the Multicast Group record and is a
> function of the DLID for any node injecting packets into the fabric. The
> compliance statments for static rate injection are called optional. But from
> the compliance statements I could not determine whether they were optional
> contingent of the CA not supporting rates higher than 2.5 GBPs or they are
> truly optional.

You might have already got the answer based on the recent discussions
between me, Joe and others.


> 
>  PS: You can ask me to shut up if it is kind of late for such questions? :)




> 
> -thanks,
> Ashwin
> 
> -----Original Message-----
> From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
> Of Vivek Kashyap
> Sent: None
> To: Ashwin Kamath
> Cc: ipoverib@ietf.org
> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> 
> 
> >
> > There is a general thought that ARP is a one step solution for obtaining
> the
> > parameters required for establishing IPoIB over UD. But I think that even
> > for using ARP, one will require to get the Path record information. I do
> not
> > think that ARP is a one stop solution. What happens if both nodes
> determine
> > that their MTU is 2K without considering that a switch component in
> between
> > might restrict the MTU to 256 bytes? Such information can be obtained only
> > from the central SM via a path record. Am I missing something here? If
> 
>         The all-nodes multicast(IPv6)/broadcast(IPv4) addresses are
>         mapped to a multicast GID. This multicast GID is setup with
>         the MTU for the subnet/link. All nodes always IB_join this
>         MGID. Thus the MTU is always determined from the multicast
>         GID. So is the Q_Key to be used. See the ipoib architecture
>         and multicast drafts that have been posted. You can get them
>         from the ipoverib pages at IETF site.
> 
>         Other parameters LID/GID/QPN are received via ARP. SL is
>         determined based on the implementation. So, ARP is and
>         certainly can be a one-step solution like you mention in the
>         begining of your note.
> 
> 
> Vivek
> 
> > address resolution is all that ARP is required to do, without trying to
> > establish parameters for communication one may not want to know the MTU,
> but
> > from the discussions I see that the Address resolution is being used for
> > establishing communication parameters (like the QPN)
> >
> > So the only real problem with having the LID in the ARP response, that I
> see
> > is when the ARP response needs to cross IB Routers. Can I assume that
> > changes in LID assignments are not a big problem because such changes are
> > not intended to be frequent and current ARP cache implementations already
> > factor in the possibility of a link level address change. Changing LIDs
> > would have a bigger impact on shared devices using RC than on IPoIB using
> > UD. Is it ok if I consider a change in LID assignment to be equivalent of
> a
> > server reboot?
> >
> > If I understood what Jerry, one will have a problem if the ARP responder
> > lies in another IB Subnet, in which case the LID in the response will not
> be
> > relavant to the originator of the ARP request, in-fact the LID will have
> to
> > be the LID of the router. Was there any resolution reached for this
> problem?
> >
> > In short you do need an ARP + Path record diagnosis to determine all the
> > parameters for communication. What would happen if we bring an IB router
> in
> > between? One cannot obtain a path record to a node in another subnet. One
> > can obtain the path record to the IB Router. Should we be talking about
> the
> > behaviour of the IB router in case of mismatch of MTUs or should we say
> that
> > we will always use the smallest mtu for communicating across an IB router.
> > Or should we say that this could fall on the plate of the Infiniband WGs.
> >
> > -Ashwin.
> >
> > -----Original Message-----
> > From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
> > Sent: None
> > To: kamath@omegaband.com
> > Cc: ipoverib@ietf.org
> > Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
> >
> >
> > My comments are embeded in the following.
> >
> > >I agree with using GID both for longevity and scope. What about using
> > >SIDR(Private Data) for communicating all the capability bits and any
> other
> > >extras? If it is not too late for this, I have some new feedback.
> > >
> > >For ARP on an IB subet one would require that
> > >a. IB Switches support multicast. (This is true in most cases today.)
> > >b. IP addresses used in an infiniband subnet are unique for use by IPoIB.
> > >(This is an important constraint which gets rid of the ethertype field)
> > >c. Some issues related to Multicast over IB need to be resolved (like
> > >multiple consumers using multicast resources.)
> >
> > This one is really simple. A MGID contains 128 bits. Even IPv6 recognizes
> > that it only needs the lower 32bits for the forseeable future. So the
> > probability of MGID collisions is negligible. If one really wants
> > something bullet-proof, we can ask IBTA to reserve a chunk (or endorse
> > the IPoIB signature covered in the link and multicast draft) for IPoIB.
> >
> > >
> > >Requirement (b) was addressed by defining a multicast group and the IP
> > >address format. One would still have to deal with the GID, LID and the IP
> > >address. Including the LID in the ARP Response will be difficult to deal
> > >with because
> > > a. LID Assignments change.
> > > b. ARP might have to traverse a router, for IP Subnets spanning IB
> Subnets
> > >(a) might not really be an issue because ARP entries are aged. In
> ethernet,
> > >the ethernet topology might change and MAC Addresses can change on server
> > >reboots (by way of using a different MAC or by assigning a MAC Address.)
> > The
> > >current ARP implementations do take these scenarios into account.
> Moreover
> > >Path record lookups alone would not help resolve the issue of changing
> > LIDs.
> > >The locally cached path record will not be valid if the LID changes. For
> a
> > >true solution, there has to be some way of using IB Traps to get
> > information
> > >about topology changes as and when it happens. An end node can subscribe
> > for
> >
> > This is a generic, not IPoIB specific issue. So the work belongs to IBTA
> > to ensure some mechanism exists for flushing stale path records.
> >
> > >a trap, but the end node will have to do something like,
> > > * Tell me when all the switches on the path to this node report a state
> > >change on such and such a port.
> > > This will not work very well right now because one will have to
> subscribe
> > >for traps on all switches leading to the end node, which implies that an
> > end
> > >node interested in the existence of another end node will need to have
> > >knowledge of the complete IB topology. This is probably something worth
> > >considering in the Infiniband WGs.
> >
> > Ok, we're saying the same thing.
> >
> > >
> > >On the same token, if you use the globally unique GID, information about
> > the
> > >GID will not be maintained by the local subnet's subnet administration
> and
> > >hence a path record lookup based on the GID would fail.
> >
> > In this case you follow the IBA rule. The rule says that you don't even
> > attempt to lookup a pathrecord for a remote GID in a different IB subnet.
> > You need to find an IB route to it first. This is similar to IP.
> >
> > > So if the GID lookup
> > >fails, one would send the ipoib packet to the designated IB router (as is
> > >done with ethernet today, albiet based on netmasks). But what if you have
> > an
> > >IP subnet spanning IB Subnets. The designated router exists for an IP
> > Subnet
> > >and the local subnet IB administration will not have information about
> all
> > >the IP nodes, because they could exist on another IB Subnet.
> >
> > Hmmm, you seem to confuse routing at the IB level with that at the IP
> > level. (Or am I the one who is confused?)
> >
> > >
> > > The following IB transactions are required before sending any IPoIB data
> > >over the subnet (assuming the multicast join is already done and we use
> > GIDs
> > >as link layer addresses)
> > > 1. IpoIB ARP.
> > > 2. Path record lookup.
> > > 3. SIDR
> > >
> > >In all this, the existence of the ARP is questionable. The same
> > >functionality can be achieved using 2 and 3 above.
> > >If you use GIDs as the IP addresses ,you can use a Path record lookup and
> > >then use SIDR to get all the information to use IPoIB. The advantages are
> > >evident, for eg,
> >
> > I think similar idea has been brought up before. One major obstacle here
> > is that a given interface may host many (e.g. a few thousand) IP addresses
> > simultaneously (e.g. virtual hosting). You just don't have space for that
> > many GIDs on a HCA.
> >
> > Jerry
> >
> > > a. Traversing a router. Assuming that the path record lookup fails, one
> > >will redirect all the related ipoib packets to the Gateway/router. Since
> > ARP
> > >does not exist, one does not have to worry about the issues involved by
> ARP
> > >traversing a router, or the router proxying for end nodes on the same IP
> > >Subnet but different IB Subnets. IP and IB Subnets will be one and the
> > same.
> > > b. Debate on using the ether-type field vanishes. The Service ID will be
> > >unique to IPoIB and will be assigned by some WG. You could potentially
> use
> > >two Service IDs (might be an overkill), one for IPv6 packets and another
> > for
> > >IPv4.
> > > c. No special format for IP addresses is required (I have not thought
> this
> > >through very well.)
> > > d. Switches are not required to support Multicast.
> > > e. There are no issues with multiple consumers using the same multicast
> > >resources.
> > > f. Multicast could be used by an IB Gateways only to avoid
> multi-unicast.
> > >
> > >In this scenario the IP subnet is the same as an IB Subnet, since the
> same
> > >network addresses are used.
> > >
> > >-Ashwin
> > >
> > >
> > >
> > >-----Original Message-----
> > >From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
> > >Of H.K. Jerry Chu
> > >Sent: None
> > >To: ipoverib@ietf.org
> > >Subject: [Ipoverib] What's inside an IPoIB link layer address?
> > >
> > >
> > >While proposals beyond the basic IPoIB over UD started showing up,
> > >I still don't think we've reached a rough consensus on some basic
> > >questions such as what the IPoIB link layer address should contain.
> > >Again I'd like to apologize for not being there at the SLC meeting
> > >last month. Perhaps a consensus was reached there but i've missed
> > >the boat. In that case please let me know what it is and I'll shut
> > >up.
> > >
> > >What I'd like to see included in the IPoIB link layer address is
> > >
> > >GID
> > >QPN
> > >
> > >and I'd like to allow QPN field to be 0.
> > >
> > >What I don't like to see included is LID.
> > >
> > >Discussions:
> > >
> > >There has been a consensus of using GID, not LID as the basic
> > >IB address identifying the port hosting an IP address. GID is
> > >sufficiently stable, not subject to change across a node reboot
> > >or subnet merge (IB level) like LID is. Also people don't want
> > >to exclude the possibility of IPoIB link spanning more than
> > >one IB subnet. GID supports scope beyond local link so it's
> > >suitable for that purpose.
> > >
> > >Multiple GIDs with different scopes may have been assigned to a CA.
> > >It is required that a CA returns the GID with a scope no less than
> > >the one for the IPoIB link. E.g. if an IPoIB link covers multiple
> > >IB subnets, but a CA uses a link-local GID as part of its IPoIB
> > >link layer address, it's not going to work well.
> > >
> > >I don't see any value of including LID besides GID as proposed
> > >by Vivek. I believes a lookup of PathRecord is always needed
> > >in order for a node to retrieve complete path info before sending
> > >unicast pkts to another node. The destination LID is part of
> > >the PathRecord so including it in the link layer address returned
> > >from ARP/ND doesn't save work. But you have all the drawbacks that
> > >swayed people away from using LID - it adds volatility to the link
> > >address. LID changing due to subnet merge or other reasons requires
> > >potentially new mechanism being added to IP/driver implementation
> > >to trigger ARP/ND flush. Lastly, what should be the LID if the IP
> > >node is in a different IB subnet? Obviously it can't be the LID of
> > >an intermediate IB router.
> > >
> > >For QPN I'd like to suggest one thing. Since SIDR is a required
> > >feature by IBA, the QPN field is allowed to be zero. What that
> > >means is that the requester must use SIDR to retrieve the remote
> > >QPN, given the GID of a remote IP address.
> > >
> > >This feature allows the support of static/persistent "MAC addr"
> > >wrt node reboot by removing the volatile part of the link layer
> > >address - the QPN. In many implementations the QPN allocated for
> > >a CA may change across node reboot. Support of static "MAC addr"
> > >has been considered desirable by a number of people.
> > >
> > >I don't like to add additional requirement such as SIDR to IPoIB,
> > >and was trying to make the above feature "optional". Ideally nodes
> > >can express whether they'd like QPN to be returned by embedding
> > >some flags, such as "I don't talk SIDR so please return QPN" in
> > >their IPoIB link layer addresses. But the broadcast nature of
> > >IPv4/ARP forbids the use of different "MAC addrs" for the same IP
> > >address, so my attempt failed.
> > >
> > >Comment?
> > >
> > >Jerry
> > >
> > >
> > >_______________________________________________
> > >IPoverIB mailing list
> > >IPoverIB@ietf.org
> > >http://www.ietf.org/mailman/listinfo/ipoverib
> > >
> > >
> > >_______________________________________________
> > >IPoverIB mailing list
> > >IPoverIB@ietf.org
> > >http://www.ietf.org/mailman/listinfo/ipoverib
> >
> >
> >
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> >
> 
> 
> --
> Vivek Kashyap
> Linux Technology Center, IBM
> kashyapv@us.ibm.com
> vivk@us.ibm.com
> 503 578 3422 (o)
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 16:40:38 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA14122
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 16:40:38 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id QAA23690
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 16:40:41 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA23667;
	Tue, 15 Jan 2002 16:39:34 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA23631
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 16:39:32 -0500 (EST)
Received: from vieo.com (vieo.com [216.30.79.131])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA14071
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 16:39:27 -0500 (EST)
Received: (from root@localhost)
	by vieo.com (8.11.2/8.11.2) id g0FLd0R81059
	for ipoverib@ietf.org; Tue, 15 Jan 2002 15:39:00 -0600 (CST)
	(envelope-from rronald@VIEO.com)
Received: from Fishhead (dhcp-10-2-15-151.vieo.com [10.2.15.151])
	by vieo.com (8.11.2/8.11.2) with SMTP id g0FLcxK80977;
	Tue, 15 Jan 2002 15:38:59 -0600 (CST)
	(envelope-from rronald@VIEO.com)
Message-ID: <00a601c19e0d$02924b70$970f020a@Fishhead>
From: "Roger Ronald" <rronald@VIEO.com>
To: "Vivek Kashyap" <kashyapv@us.ibm.com>
Cc: <ipoverib@ietf.org>
References: <200201152107.g0FL7gw13021@eng4.beaverton.ibm.com>
Date: Tue, 15 Jan 2002 15:38:45 -0600
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 5.50.4133.2400
X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400
X-scanner: scanned by Inflex 0.1.5c+ on vieo.com
Content-Transfer-Encoding: 7bit
Subject: [Ipoverib] Summary of the IPoIB link layer address debate
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

It seems like the general debate that's raging is trying to answer 
a single question:

"Is it wise to include the LID in order to enable a
shortcut that allows bypassing a trip to the SM/SA?"

My read is that second party dissemination of LIDs is not 
something that is prohibited (per the IBTA rules), but it is a
shortcut that the IBTA architects weren't expecting
and are uncomfortable seeing. Various objections
about PathRecords, rate control, SL, and MTU have
been mentioned, but don't seem to make for a conclusive
decision point.

In the end, the tradeoff seems to be simplicity versus
flexibility. Leaving out the LID makes the header
smaller and the processing is always done in one
way. Putting the LID into the header allows
skipping a step under (at least) some circumstances.

I'm willing to live with either approach. I am
not as ambivalent about Vivek's last email that
questions the inclusion of GID. While I'm not too
concerned about supporting a single IP subnet over
multiple IB subnets, I do think that the GID is
required to support any sort of bridging devices. If only a 
16-bit LID is used, one would never be able to build
a useful bridge to important fabrics like Ethernet.

RR
----- Original Message ----- 
From: "Vivek Kashyap" <kashyapv@us.ibm.com>
To: <Jerry.Chu@eng.sun.com>
Cc: <ipoverib@ietf.org>
Sent: Tuesday, January 15, 2002 3:07 PM
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?


> 
> > 
> > What I don't like to see included is LID.
> > 
> > Discussions:
> > 
> > There has been a consensus of using GID, not LID as the basic
> 
> I don't think there has been any consensus on this issue.
> 
> > IB address identifying the port hosting an IP address. GID is
> > sufficiently stable, not subject to change across a node reboot
> > or subnet merge (IB level) like LID is. Also people don't want
> > to exclude the possibility of IPoIB link spanning more than
> > one IB subnet. GID supports scope beyond local link so it's
> > suitable for that purpose.
> > 
> > Multiple GIDs with different scopes may have been assigned to a CA.
> > It is required that a CA returns the GID with a scope no less than
> > the one for the IPoIB link. E.g. if an IPoIB link covers multiple
> > IB subnets, but a CA uses a link-local GID as part of its IPoIB
> > link layer address, it's not going to work well.
> 
>         Let us look into this issue in detail first. It has not been
>         pointed out what one gains by spanning IB subnets. Let us
>         first make or break the case for IP subnets spanning IB
>         subnets. What is the gain ?
> 
>         As for things being stable - GID scopes, GID prefix changes,
>         subnet merges etc. effect GIDs as much as LIDs. Once the node
>         is up it will have a constant LID. SMs are also likely to keep
>         the LID constant across reboots. But one point to note is that
>         LIDs are always necessary. Whether they change or not you need
>         to determine them. GIDs are not necessary. So this leaves us
>         neither here nor there. They are just link addresses with the
>         LID always required and GID an optional value.
> 
> > 
> > I don't see any value of including LID besides GID as proposed
> > by Vivek. I believes a lookup of PathRecord is always needed
> > in order for a node to retrieve complete path info before sending
> 
>         As noted elsewhere there is no need to to always access the
>         pathrecord. Even if the information is required it doesn't
>         mean that one needs to determine the information from the path
>         record when LID is to be determiend. Determining the LID is
>         not tied to that. They are disjoint processes. Also, one can
>         look up the path record using the LID/GID or GUID.
> 
> 
> > unicast pkts to another node. The destination LID is part of
> > the PathRecord so including it in the link layer address returned
> > from ARP/ND doesn't save work. But you have all the drawbacks that
> > swayed people away from using LID - it adds volatility to the link
> > address. LID changing due to subnet merge or other reasons requires
> > potentially new mechanism being added to IP/driver implementation
> > to trigger ARP/ND flush. Lastly, what should be the LID if the IP
> > node is in a different IB subnet? Obviously it can't be the LID of
> > an intermediate IB router.
> 
> LID changing or GID changing (in scope in case of merger or
> renumbering) etc. are issues that will have other effects and are
> equally connected to any flushing or more drastic effects that will
> have on the SM and the subnet. IBTA has not solved SM merges, P_Key
> merges and management and various such issues. Such a disruption
> applies all over IB components and is not limited to LIDs/GIDs.
> 
> Let us first answer the question of why IPoIB subnets must span IB
> subnets ? We don't know why we need such a setup ? We don't know when
> such a setup will be possible ? We do know that IPoIB subnets within
> an IB subnets can be setup easily with the current specification.
> 
> 
> Secondly, if you step back you'll realise that LID is hardly an issue
> when resolving across subnets. Unlike what has been postulated the
> process is not the same when resolving within the IB subnet and when
> resolving across the IB subnet. 
> 
> Assumption: at IP level- a multicast request; receive a unicast reply
> 
>         Now, when sending the broadcast the sender doesn't know where
>         the responder is. So he includes his LID/GID. The receiver
>         gets it across multiple subnets. It gets sees a different
>         GID-prefix and hence replaces the LID with its default router
>         when recording the address. The same happens on the response.
> 
>         It works fine for both situations - IPoIB subnets
>         within/across IB subnets. You can easily have it that if the
>         GID is local the routing lookup will return the local LID.
>         Single process.
> 
>         In the case of not including the LID you do the same thing. If
>         the GID prefix is local you go to the SM/SA to get the LID. If
>         the GID prefix is different you record your default router's
>         LID. So the process for across the subnet is the same but you
>         shortchange the implementation that is more likely - the one
>         whithin the subnet.
> 
>         For within the subnet you use a different method - going to
>         The SA always. Not the same as going to the routing
>         infrastructure. 
> 
> However, I would still like to know what is the use of IPoIB spanning
> multiple subnets. Why do you think that is a requirement ?
> 
> 
> Vivek
> -- 
> Vivek Kashyap
> Linux Technology Center, IBM
> kashyapv@us.ibm.com 
> vivk@us.ibm.com 
> 503 578 3422 (o)
> -- 
> Vivek Kashyap
> Linux Technology Center, IBM
> kashyapv@us.ibm.com 
> vivk@us.ibm.com 
> 503 578 3422 (o)
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 
> 


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 16:58:51 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA14561
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 16:58:51 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id QAA24014
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 16:58:55 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA23993;
	Tue, 15 Jan 2002 16:57:46 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA23942
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 16:57:43 -0500 (EST)
Received: from e34.co.us.ibm.com (e34.co.us.ibm.com [32.97.110.132])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA14538
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 16:57:39 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e34.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id QAA239048;
	Tue, 15 Jan 2002 16:54:40 -0500
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0FLver63388;
	Tue, 15 Jan 2002 14:57:40 -0700
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0FLvbV23010;
	Tue, 15 Jan 2002 13:57:37 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0FLvad15693;
	Tue, 15 Jan 2002 13:57:36 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201152157.g0FLvad15693@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
To: kashyapv@us.ibm.com (Vivek Kashyap)
Date: Tue, 15 Jan 2002 13:57:35 -0800 (PST)
Cc: Jerry.Chu@eng.sun.com, ipoverib@ietf.org
In-Reply-To: <200201152107.g0FL7gw13021@eng4.beaverton.ibm.com> from "Vivek Kashyap" at Jan 15, 2002 12:07:41 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

> >
> >
> >	Suggested Link description to be used in address resolution:
> >
> >		LID + capability flags + GID + QPN
> >
> >
> > 
> >       Why the LID?
> >
> >             People said they don't want to go to the SM if possible
> >             (LID allows for that. As noted here and LWG the SL can be
> >             determined based on QoS or other parameters. The SL may
> >             also be determined from the SA/SM)
> 
> I don't think pathrecord lookup can be avoided or should be skipped.
> You're proposing something that IMHO is beyond my understanding of
> how things work in IBA. Can you get an experienced IBTA member
> to substantiate your statements here?

        There has been some discussion on this issue. As noted SL
        certainly doesn't require going to the SM/SA. The path-rate
        support is mandatory if sending > 2.5Gbps but not determining
        it from SM/SA. To my knowledge IBTA is actively pursuing a
        dynamic congestion management solution that would not depend
        on static path rate. So this is not an argument to tie down
        LID determination to the SA.

> 
> And what's the real benefit of saving just a lookup out of an already
> infrequent operation like ARP due to cacheing?

        I would like to know what is the drawback that you see that
        you don't want one to go ahead with it? ARP is repeated every
        so often and happens quite often. It also happens on failover
        - it is preferable that these operations be simple, fast and
        error-free. 

        Secondly, things are more complicated the way you propose.
        Note that the first level address would be first determined.
        This completes ARP/ND. However, address resolution is not
        complete. The packet is sent as far as IP is concerned and now
        the SA/SM is consulted to get the LID etc. This has very
        strong shades of ATM VCs. IP thought it had sent the packet
        since the ARP/ND was complete - it took a while before the
        lower layer completed the determination. The driver of course
        keeps a GID to LID table too since the LID is a separate
        entity. More complication.

	Include LID in the resolution - fast/simple/error-free.


> 
> >
> >             People said they want the process to end in small number
> >             of messages (LID allows a query/response to settle the
> >             issue. As noted one doesn't necessarily have to go to the
> >             SM to get the SL).
> >
> >             LID is always needed. (you can't escape it).
> 
> There are a number of issues of including LID as I've pointed out in
> another mail.

	I have just responded to an earlier mail of yours on these points.


> 
> >
> >             BTW, people said they want the 'link-address' to be 48 bits.
> >             (LID + QPN + cap_flags is 48 bits).
> 
> I'm confused. Aren't you proposing to include GID as well. Where does
> this 48 bits come from?

        I was pointing out another issue that many have raised.
        In general I am ok with dropping GID since it is for the
        future. IT buys us nothing as of now. So if we dropped GID we
        can work with these 48 bits. When IPoIB in reality starts
        spanning multiple IPoIB subnets (not for a long time) we can
        include GID. Then the hardware length will become 22 bytes as
        against 6 bytes. One can still interoperate quite easily.

	Wouldn't that work for all ? 

        I included the GID because many want to put in the hooks for
        the future case of IPoIB spanning IB subnets. I would though
        like a valid reason specified for why do we need IP subnets to
        span IB subnets. Isn't IP subnetting enough?



> 
> >
> >
> >       Why the GID ?
> >       
> >                People want IP to span across IB subnets. We are
> >                stretching stretching here since such implementations
> >                are not possible yet.
> >                However, we can define it such that it will work in
> >                the future. Therefore include the GID.
> 
> You're ignoring all the other good reasons of using GID as I and many
> have pointed out before (recently in another mail).

	I have covered those in another mail I sent some time back.


> >	Why the QPN: 
> >
> >                1) There is only one other way to get the QPN. That is
> >                the SIDR protocol. However, this has a drawback.
> >
> >		SIDR is NOT REQUIRED by IBTA.
> >
> >                I had raised this point last August 'Feasibility of
> >                SIDR': 
> >
> >                ***** Section 12.11 essentially states that SIDR is
> >                optional. As the specification states:
> >
> >               << GSAs shall support this protocol if non-management
> >               services are pro-vided on the Channel Adapter at other
> >               than fixed QPNs. If this protocol is *not supported*,
> >               the status "The method/attribute combination is not
> >               sup-ported" shall be returned, as described in section
> >               Table 101 MAD Common Status Field Bit Values >>
> >
> >		*****
> >
> >                That still holds. SIDR is NOT required by IBTA. A
> >                device can always choose to provide all non-management
> >                services at fixed QPNs. It may just choose to provide
> >                IP and nothing else at a particular QP and hence won't
> >                do SIDR.
> >
> >                That is why I propose that the QPN be always included
> >                for the default case. SIDR usage is gated through the
> >                QPN flag and is in the realm of the advanced
> >                capabilities draft as has been suggested by many on
> >                this mailing list. And I agree.
> 
> I agree in general that SIDR should not be required by IPoIB.
> I'd second what Arvind said, before we can fully define what
> these capability are meant to work, let's mark these bits as
> simply reserved.

        The advanced capabilities draft describes how these bits are
        supposed to work. I have also described it in the IETF slides
        and on this mailing list in item 9 of the thread 'ipoib
        architecture/encapsulation'. 

        The default case is IPoIB-UD with QPN included in the ARP/ND
        packets. If the QPN/RC flag is set then the sender is
        advertising its support for SIDR-QPN and RC. A base
        implementation ignores these flags and always sets them to
        zero on send.


> 
> Jerry
>  
> >
> >
> >                2) Address resolution that returns the QPN is done in
> >                one request/reply exchange. Any other method, while
> >                talking to the same endpoint, requires 6 messages. 2
> >                for ARP/ND and 2 each for each of the lower layer
> >                request/reply. IT is wasteful, more complicated,
> >                more time consuming wrt fast failover etc.
> >
> >
> >	Capability flags :
> >
> >                Multiple people have opined that they would prefer in
> >                certain cases to map the protocol/ULP endpoints to
> >                specific QPs. People also want to use IP over RC mode
> >                (and others maybe) due to the advantages of larger MTU
> >                and Automatic path failover etc.
> >
> >                The capability flags allow for interoperation between
> >                the default/dumb devices and the more sophisticated
> >                devices. IT also allows the endpoints to avoid
> >                unnecessary trial and error/to and fro when trying to
> >                utilise these features. The flag indicates the
> >                advanced capabilities supported and the receiver can
> >                choose to utilies them.
> 
> I'd second what Arvind said, before we have a consensus that
> including these capability flags in the link addr is a good thing
> and have their meaning fully defined, let's mark these bits as
> simply reserved.

        I urge you to review the draft I have submitted. A few WG
        members have already written that they like the idea. It
        interoperates with the default case seamlessly and allows
        optimised solutions to be supported within the same subnet.

> >
> >Result:
> >
> >        We cannot depend on SIDR. We need to include the default QPN.
> >        We should not force implementations to go to the SA/SM (IBTA
> >        does not) and so allow LID/GID to be determined from the
> >        remote host. The default case works with :
> >
> >	LID+Capability flags(all set to 0)+QPN+GID
> >


Vivek

-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 17:16:07 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA14874
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 17:16:07 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id RAA24907
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 17:16:09 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA24843;
	Tue, 15 Jan 2002 17:12:11 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA24812
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 17:12:10 -0500 (EST)
Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA14798
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 17:12:07 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id QAA37196
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 16:07:58 -0600
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0FMC9e200388
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 17:12:09 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id g0FMC8g24407
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 14:12:08 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0FMC8816612
	for ipoverib@ietf.org; Tue, 15 Jan 2002 14:12:08 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201152212.g0FMC8816612@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
To: ipoverib@ietf.org
Date: Tue, 15 Jan 2002 14:12:07 -0800 (PST)
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Jerry,


> >As for the why LID:
> >I think that going to the SM is not so efficient given the fact that it is a
> >central resource and all the nodes in the subnet go to if we could avoid it
> >why not
> 
> First the benefit is dubious. In modern TCP/IP stacks the frequency of ARP
> requests is capped by the ARP cache flush internal, which is in minutes
> range. (E.g. in Solaris it's five minutes.) Assuming you have 100 nodes
> on a subnet, you're telling me saving the SM lookup call, which at max
> rate (if a local node is activetly talking to every other node) of one
> every 3 sec. is worth noting?

        What if there are 500/1000 or 3000 nodes ? Besides, as I have
        noted in the last mail the process is more complicated not
        only internally but also requires 6 messages where only 2
        would have sufficed. More complication means more maintenance,
        slower failover, more bugs etc. We also know that centralised
        systems don't scale well. That is why we are using all-nodes
        multicast and reply rather than go to some central ARP
        server. 



> 
> I'm still not sure if a trip to SM can be saved at all (need to catch
> up with all the mail this morning). Even if in some cases it could, do
> we believe that not doing any pathrecord lookup is going to be the norm 
> of IPoIB implementation that warrents an extr field in the link addr?
> How about all the issues I've pointed out before? I have another one
> to add - simplicility. Unless there is sufficient justification, the more
> fields you add to a protocol the more complexity (in areas you may not
> even forsee now) it's going to add.

        See my note I sent you describing the process. Not including
        the LID is more complicated. Including it allows you to be
        complicated or simple as you wish. Using the LID your ARP
        module can be coded in a common way - as explained in the
        previous mail - without it you need two disjoint proceses with
        timeouts, multiple messages interspersed within the whole
        process. 

> 
> >When IP will span IB the problem becomes more serious: Unlike IP routers
> >which has a default route entry, currently IB has explicit entries for each
> >path this might require doing too many PathRecord operations, which are also
> >currently very slow when going outside an IB subnet
> >(I had posted a proposal to the MSO team trying to resolve those issues
> >somehow however it is under discussion)
> >I would tend to send this info anyway since the extra 16 bits wouldn't harm
> >anyone,
> 
> But it does. What would you define the LID for a node on a non-local IB
> subnet? It can't be a LID to a router because this is an endnode's "MAC
> addr" we're defining here.

	See my previous mail.

> 
> >one might choose not to use them if he wants to go to the SM, this
> >way everybody will be happy and over time people will use the best solution
> 
> No we can't do handwaving here. You either define a field with clear
> semantics so implementations have a chance to interoperate, or take it
> out. There is no iffy here (unless you add a "LID included" flag to the
> link addr).
> 
> >
> >As for the Ethertype:
> >After giving Vivek a hard time on this issue I accept his proposal
> >Not because of the performance issues raised (since I don't think they are
> >real)
> >But because it is not really needed, and if it simplifies things, why not
> >
> >As for the capability bits:
> >Avoiding the SIDR is a smart move
> >I'm not sure if the UC option will be used, if using connection why not
> >reliable connection
> >I just wanted to add SDP (Socket Direct Protocol) bit, Again ones might
> >choose not to use it, for the one that do it really simplifies the process a
> >lot
> >
> >I'm hoping we get a quick convergence on those issues so we would move
> >forward
> >The issues are not serious and I think if an alternative enable someone to
> >do something he wants without violating the other's ability to do what they
> >want just ignoring the extra info, we should go for it
> >
> >Yaron
> >
> >-----Original Message-----
> >From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
> >Vivek Kashyap
> >Sent: Tuesday, January 15, 2002 12:46 AM
> >To: Jerry.Chu@eng.sun.com
> >Cc: ipoverib@ietf.org
> >Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> >
> >Folks,
> >
> >       It is close to the last call. In this regard I would like to
> >       state that we have been discussing this issue for a long time
> >       now. The most recent discussion is what I presented at the SLC
> >       last december. (I certainly felt that the room was in agreement
> >       - no queries, objection etc.)
> >
> >       I believe the following link-description provides the best
> >       assimilation of all the choices that have been put forth. If we
> >       step back and restart the whole discussion again we are likely
> >       to come back to the following - so here is a brief compilation
> >       of issues.
> >
> >
> >        Suggested Link description to be used in address resolution:
> >
> >                LID + capability flags + GID + QPN
> >
> >
> >
> >       Why the LID?
> >
> >             People said they don't want to go to the SM if possible
> >             (LID allows for that. As noted here and LWG the SL can be
> >             determined based on QoS or other parameters. The SL may
> >             also be determined from the SA/SM)
> >
> >             People said they want the process to end in small number
> >             of messages (LID allows a query/response to settle the
> >             issue. As noted one doesn't necessarily have to go to the
> >             SM to get the SL).
> >
> >             LID is always needed. (you can't escape it).
> >
> >             BTW, people said they want the 'link-address' to be 48 bits.
> >             (LID + QPN + cap_flags is 48 bits).
> >
> >
> >       Why the GID ?
> >
> >                People want IP to span across IB subnets. We are
> >                stretching stretching here since such implementations
> >                are not possible yet.
> >                However, we can define it such that it will work in
> >                the future. Therefore include the GID.
> >
> >        GUID
> >
> >                Can be made part of the GID.
> >
> >
> >        Why the QPN:
> >
> >                1) There is only one other way to get the QPN. That is
> >                the SIDR protocol. However, this has a drawback.
> >
> >                SIDR is NOT REQUIRED by IBTA.
> >
> >                I had raised this point last August 'Feasibility of
> >                SIDR':
> >
> >                ***** Section 12.11 essentially states that SIDR is
> >                optional. As the specification states:
> >
> >               << GSAs shall support this protocol if non-management
> >               services are pro-vided on the Channel Adapter at other
> >               than fixed QPNs. If this protocol is *not supported*,
> >               the status "The method/attribute combination is not
> >               sup-ported" shall be returned, as described in section
> >               Table 101 MAD Common Status Field Bit Values >>
> >
> >                *****
> >
> >                That still holds. SIDR is NOT required by IBTA. A
> >                device can always choose to provide all non-management
> >                services at fixed QPNs. It may just choose to provide
> >                IP and nothing else at a particular QP and hence won't
> >                do SIDR.
> >
> >                That is why I propose that the QPN be always included
> >                for the default case. SIDR usage is gated through the
> >                QPN flag and is in the realm of the advanced
> >                capabilities draft as has been suggested by many on
> >                this mailing list. And I agree.
> >
> >
> >                2) Address resolution that returns the QPN is done in
> >                one request/reply exchange. Any other method, while
> >                talking to the same endpoint, requires 6 messages. 2
> >                for ARP/ND and 2 each for each of the lower layer
> >                request/reply. IT is wasteful, more complicated,
> >                more time consuming wrt fast failover etc.
> >
> >
> >        Capability flags :
> >
> >                Multiple people have opined that they would prefer in
> >                certain cases to map the protocol/ULP endpoints to
> >                specific QPs. People also want to use IP over RC mode
> >                (and others maybe) due to the advantages of larger MTU
> >                and Automatic path failover etc.
> >
> >                The capability flags allow for interoperation between
> >                the default/dumb devices and the more sophisticated
> >                devices. IT also allows the endpoints to avoid
> >                unnecessary trial and error/to and fro when trying to
> >                utilise these features. The flag indicates the
> >                advanced capabilities supported and the receiver can
> >                choose to utilies them.
> >
> >
> >Result:
> >
> >        We cannot depend on SIDR. We need to include the default QPN.
> >        We should not force implementations to go to the SA/SM (IBTA
> >        does not) and so allow LID/GID to be determined from the
> >        remote host. The default case works with :
> >
> >        LID+Capability flags(all set to 0)+QPN+GID
> >
> >
> >Vivek
> >
> >
> >
> >
> >_______________________________________________
> >IPoverIB mailing list
> >IPoverIB@ietf.org
> >http://www.ietf.org/mailman/listinfo/ipoverib
> >
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)
-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 17:26:53 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA15110
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 17:26:53 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id RAA25114
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 17:26:55 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA25060;
	Tue, 15 Jan 2002 17:25:07 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA25025
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 17:25:05 -0500 (EST)
Received: from mail6.microsoft.com (mail6.microsoft.com [131.107.3.126])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA15089
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 17:25:02 -0500 (EST)
Received: from inet-vrs-06.redmond.corp.microsoft.com ([157.54.6.201]) by mail6.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Tue, 15 Jan 2002 14:24:32 -0800
Received: from 157.54.8.155 by inet-vrs-06.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Tue, 15 Jan 2002 14:24:32 -0800
Received: from red-imc-02.redmond.corp.microsoft.com ([157.54.9.107]) by inet-hub-04.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Tue, 15 Jan 2002 14:24:32 -0800
Received: from win-imc-02.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.84]) by red-imc-02.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Tue, 15 Jan 2002 14:24:20 -0800
Received: from win-msg-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.52]) by win-imc-02.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3588.0);
	 Tue, 15 Jan 2002 14:22:15 -0800
X-MimeOLE: Produced By Microsoft Exchange V6.0.6132.0
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Subject: RE: [Ipoverib] Summary of the IPoIB link layer address debate
Date: Tue, 15 Jan 2002 14:22:15 -0800
Message-ID: <2E33960095B58E40A4D3345AB9F65EC10353E216@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
Thread-Topic: [Ipoverib] Summary of the IPoIB link layer address debate
Thread-Index: AcGeDNcQSo/7zlr/TgKmgR10f/HoDgABgvZw
From: "Arvind Murching" <arvindm@windows.microsoft.com>
To: "Roger Ronald" <rronald@VIEO.com>, "Vivek Kashyap" <kashyapv@us.ibm.com>
Cc: <ipoverib@ietf.org>
X-OriginalArrivalTime: 15 Jan 2002 22:22:15.0397 (UTC) FILETIME=[16035950:01C19E13]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id RAA25026
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

Implementations that wish to avoid the trip to SM may optionally use the
technique Michael Krause proposed, i.e., use the source LID in the ARP
response packet. There is no need to repeat the LID in the link layer
address.

- Arvind

-----Original Message-----
From: Roger Ronald [mailto:rronald@VIEO.com] 
Sent: Tuesday, January 15, 2002 1:39 PM
To: Vivek Kashyap
Cc: ipoverib@ietf.org
Subject: [Ipoverib] Summary of the IPoIB link layer address debate

It seems like the general debate that's raging is trying to answer 
a single question:

"Is it wise to include the LID in order to enable a
shortcut that allows bypassing a trip to the SM/SA?"

My read is that second party dissemination of LIDs is not 
something that is prohibited (per the IBTA rules), but it is a
shortcut that the IBTA architects weren't expecting
and are uncomfortable seeing. Various objections
about PathRecords, rate control, SL, and MTU have
been mentioned, but don't seem to make for a conclusive
decision point.

In the end, the tradeoff seems to be simplicity versus
flexibility. Leaving out the LID makes the header
smaller and the processing is always done in one
way. Putting the LID into the header allows
skipping a step under (at least) some circumstances.

I'm willing to live with either approach. I am
not as ambivalent about Vivek's last email that
questions the inclusion of GID. While I'm not too
concerned about supporting a single IP subnet over
multiple IB subnets, I do think that the GID is
required to support any sort of bridging devices. If only a 
16-bit LID is used, one would never be able to build
a useful bridge to important fabrics like Ethernet.

RR
----- Original Message ----- 
<snip>

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 17:33:01 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA15226
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 17:32:56 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id RAA25648
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 17:32:58 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA25381;
	Tue, 15 Jan 2002 17:30:26 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA25337
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 17:30:24 -0500 (EST)
Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA15183
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 17:30:21 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id QAA29834;
	Tue, 15 Jan 2002 16:26:12 -0600
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0FMUEe184474;
	Tue, 15 Jan 2002 17:30:16 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0FMUCV23916;
	Tue, 15 Jan 2002 14:30:12 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0FMUBs17509;
	Tue, 15 Jan 2002 14:30:11 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201152230.g0FMUBs17509@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] Summary of the IPoIB link layer address debate
To: rronald@VIEO.com (Roger Ronald)
Date: Tue, 15 Jan 2002 14:30:11 -0800 (PST)
Cc: kashyapv@us.ibm.com (Vivek Kashyap), ipoverib@ietf.org
In-Reply-To: <00a601c19e0d$02924b70$970f020a@Fishhead> from "Roger Ronald" at Jan 15, 2002 02:38:45 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

> 
> It seems like the general debate that's raging is trying to answer 
> a single question:
> 
> "Is it wise to include the LID in order to enable a
> shortcut that allows bypassing a trip to the SM/SA?"
> 
> My read is that second party dissemination of LIDs is not 
> something that is prohibited (per the IBTA rules), but it is a
> shortcut that the IBTA architects weren't expecting
> and are uncomfortable seeing. Various objections
> about PathRecords, rate control, SL, and MTU have
> been mentioned, but don't seem to make for a conclusive
> decision point.
> 
> In the end, the tradeoff seems to be simplicity versus
> flexibility. Leaving out the LID makes the header
> smaller and the processing is always done in one
> way. Putting the LID into the header allows
> skipping a step under (at least) some circumstances.

Leaving out the GID makes the header a lot less smaller. I am not sure
what process is done in one way ? If you are talking about spanning IB
subnets with a single IPoIB subnet the process is definitely not done
in a single way. It is more easily a single way if the LID is used.
This is noted in my note below.


> > applies all over IB components and is not limited to LIDs/GIDs.
> > 
> > Let us first answer the question of why IPoIB subnets must span IB
> > subnets ? We don't know why we need such a setup ? We don't know when
> > such a setup will be possible ? We do know that IPoIB subnets within
> > an IB subnets can be setup easily with the current specification.
> > 
> > 
> > Secondly, if you step back you'll realise that LID is hardly an issue
> > when resolving across subnets. Unlike what has been postulated the
> > process is not the same when resolving within the IB subnet and when
> > resolving across the IB subnet. 
> > 
> > Assumption: at IP level- a multicast request; receive a unicast reply
> > 
> >         Now, when sending the broadcast the sender doesn't know where
> >         the responder is. So he includes his LID/GID. The receiver
> >         gets it across multiple subnets. It gets sees a different
> >         GID-prefix and hence replaces the LID with its default router
> >         when recording the address. The same happens on the response.
> > 
> >         It works fine for both situations - IPoIB subnets
> >         within/across IB subnets. You can easily have it that if the
> >         GID is local the routing lookup will return the local LID.
> >         Single process.
> > 
> >         In the case of not including the LID you do the same thing. If
> >         the GID prefix is local you go to the SM/SA to get the LID. If
> >         the GID prefix is different you record your default router's
> >         LID. So the process for across the subnet is the same but you
> >         shortchange the implementation that is more likely - the one
> >         whithin the subnet.
> > 
> >         For within the subnet you use a different method - going to
> >         The SA always. Not the same as going to the routing
> >         infrastructure. 
> > 
> > However, I would still like to know what is the use of IPoIB spanning
> > multiple subnets. Why do you think that is a requirement ?
> > 
> > 
> > Vivek
> > -- 
> > Vivek Kashyap
> > Linux Technology Center, IBM
> > kashyapv@us.ibm.com 
> > vivk@us.ibm.com 
> > 503 578 3422 (o)
> > -- 
> > Vivek Kashyap
> > Linux Technology Center, IBM
> > kashyapv@us.ibm.com 
> > vivk@us.ibm.com 
> > 503 578 3422 (o)
> > 
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> > 
> > 
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 17:58:09 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA15615
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 17:58:09 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id RAA25936
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 17:58:11 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA25913;
	Tue, 15 Jan 2002 17:56:47 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA25881
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 17:56:46 -0500 (EST)
Received: from ztxmail05.ztx.compaq.com (ztxmail05.ztx.compaq.com [161.114.1.209])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA15575
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 17:56:43 -0500 (EST)
Received: by ztxmail05.ztx.compaq.com (Postfix, from userid 12345)
	id 6DC8D41FF; Tue, 15 Jan 2002 16:56:15 -0600 (CST)
Received: from cceexg11.americas.cpqcorp.net (cceexg11.americas.cpqcorp.net [16.110.250.125])
	by ztxmail05.ztx.compaq.com (Postfix) with ESMTP
	id 5F7E84326; Tue, 15 Jan 2002 16:56:15 -0600 (CST)
Received: from cceexc18.americas.cpqcorp.net ([16.110.250.64]) by cceexg11.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966);
	 Tue, 15 Jan 2002 16:56:11 -0600
X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="US-ASCII"
Subject: RE: [Ipoverib] Summary of the IPoIB link layer address debate
Date: Tue, 15 Jan 2002 16:56:11 -0600
Message-ID: <4D027986353D1341ADA4F2A4F8A170760E316F@cceexc18.americas.cpqcorp.net>
Thread-Topic: [Ipoverib] Summary of the IPoIB link layer address debate
Thread-Index: AcGeFEy7RxRO9ngaTL+K42PwECvOFAAAPQJQ
From: "Culley, Paul" <Paul.Culley@compaq.com>
To: "Vivek Kashyap" <kashyapv@us.ibm.com>, "Roger Ronald" <rronald@VIEO.com>
Cc: <ipoverib@ietf.org>
X-OriginalArrivalTime: 15 Jan 2002 22:56:11.0888 (UTC) FILETIME=[D3DB3700:01C19E17]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id RAA25882
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

As has been pointed out several times in the past, a unicast ARP
response packet already includes the LID and an SL, both of which are
guaranteed (by IBA Spec) to be valid for sending (small) packets back to
the source of the ARP response packet.

Since this is already present in the IB Datagram header (LRH:SLID,
LRH:SL) and can be easily retrieved by HCAs built to the IB
Specification, I cannot see any point in adding a LID to the unicast
response.

Paul R. Culley
Compaq Sr. Fellow
281-514-5543


-----Original Message-----
From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
Sent: Tuesday, January 15, 2002 4:30 PM
To: Roger Ronald
Cc: Vivek Kashyap; ipoverib@ietf.org
Subject: Re: [Ipoverib] Summary of the IPoIB link layer address debate


> 
> It seems like the general debate that's raging is trying to answer 
> a single question:
> 
> "Is it wise to include the LID in order to enable a
> shortcut that allows bypassing a trip to the SM/SA?"
> 
> My read is that second party dissemination of LIDs is not 
> something that is prohibited (per the IBTA rules), but it is a
> shortcut that the IBTA architects weren't expecting
> and are uncomfortable seeing. Various objections
> about PathRecords, rate control, SL, and MTU have
> been mentioned, but don't seem to make for a conclusive
> decision point.
> 
> In the end, the tradeoff seems to be simplicity versus
> flexibility. Leaving out the LID makes the header
> smaller and the processing is always done in one
> way. Putting the LID into the header allows
> skipping a step under (at least) some circumstances.

Leaving out the GID makes the header a lot less smaller. I am not sure
what process is done in one way ? If you are talking about spanning IB
subnets with a single IPoIB subnet the process is definitely not done
in a single way. It is more easily a single way if the LID is used.
This is noted in my note below.


> > applies all over IB components and is not limited to LIDs/GIDs.
> > 
> > Let us first answer the question of why IPoIB subnets must span IB
> > subnets ? We don't know why we need such a setup ? We don't know
when
> > such a setup will be possible ? We do know that IPoIB subnets within
> > an IB subnets can be setup easily with the current specification.
> > 
> > 
> > Secondly, if you step back you'll realise that LID is hardly an
issue
> > when resolving across subnets. Unlike what has been postulated the
> > process is not the same when resolving within the IB subnet and when
> > resolving across the IB subnet. 
> > 
> > Assumption: at IP level- a multicast request; receive a unicast
reply
> > 
> >         Now, when sending the broadcast the sender doesn't know
where
> >         the responder is. So he includes his LID/GID. The receiver
> >         gets it across multiple subnets. It gets sees a different
> >         GID-prefix and hence replaces the LID with its default
router
> >         when recording the address. The same happens on the
response.
> > 
> >         It works fine for both situations - IPoIB subnets
> >         within/across IB subnets. You can easily have it that if the
> >         GID is local the routing lookup will return the local LID.
> >         Single process.
> > 
> >         In the case of not including the LID you do the same thing.
If
> >         the GID prefix is local you go to the SM/SA to get the LID.
If
> >         the GID prefix is different you record your default router's
> >         LID. So the process for across the subnet is the same but
you
> >         shortchange the implementation that is more likely - the one
> >         whithin the subnet.
> > 
> >         For within the subnet you use a different method - going to
> >         The SA always. Not the same as going to the routing
> >         infrastructure. 
> > 
> > However, I would still like to know what is the use of IPoIB
spanning
> > multiple subnets. Why do you think that is a requirement ?
> > 
> > 
> > Vivek
> > -- 
> > Vivek Kashyap
> > Linux Technology Center, IBM
> > kashyapv@us.ibm.com 
> > vivk@us.ibm.com 
> > 503 578 3422 (o)
> > -- 
> > Vivek Kashyap
> > Linux Technology Center, IBM
> > kashyapv@us.ibm.com 
> > vivk@us.ibm.com 
> > 503 578 3422 (o)
> > 
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> > 
> > 
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 18:06:15 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA15792
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 18:06:15 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id SAA26485
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 18:06:17 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA26472;
	Tue, 15 Jan 2002 18:05:31 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA26420
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 18:05:28 -0500 (EST)
Received: from omegaband.com (faust.seagullsemi.com [206.196.68.34])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA15764
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 18:05:25 -0500 (EST)
Received: from seahorsepc [207.8.52.226] by omegaband.com with ESMTP
  (SMTPD32-7.04) id A41E3F10068; Tue, 15 Jan 2002 16:58:38 -0600
From: "Ashwin Kamath" <kamath@omegaband.com>
To: <ipoverib@ietf.org>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
Date: Tue, 15 Jan 2002 17:03:40 -0600
Message-ID: <IFEKJBDLPFOEGGPIIGMFAEDGCAAA.kamath@omegaband.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0)
In-Reply-To: <200201152137.g0FLbod14559@eng4.beaverton.ibm.com>
X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400
Importance: Normal
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Vivek,

I am agreeable to the results of the discussion you had on the reflector. ie
end to end communication parameters are implementation specific, and the ARP
mechanism tries its best to get all information which may or may not be
enough for end-to-end communication. It is upto the implementation to use
the ARP information in an IB compliant fashion. As such, all end-to-end
communication parameters which cannot be determined from the configuration
cached on any given end-node, within the perimeter of the IB Specifications'
definition for end nodes, are omitted from ARP. This is to keep the
implementations free to choose from any method to obtain such information as
they deem fit. The method could be static configuration or consulting the
SA, or anything else. More specifically the definition does not exclude the
possibility of static configuration or othwerwise, of parameters like MTU,
SL, static rate. The intention of such possibilities could be simplicity,
conformance or anything else.
  I agree that this is reasonable.


-Ashwin.

-----Original Message-----
From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
Sent: None
To: Ashwin Kamath
Cc: ipoverib@ietf.org
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?


Actually I'll ask you to read the ipoib-architecture and multicast
drafts that hve been posted. They should answer your questions. If not
then ask them and we will update the drafts accordingly.

Vivek

>
>
> Vivek,
>
> Is the assumption true that the MTUs established for switching multicast
> packets will be the same as that used for Unicast switching? ie will
> multicast packets from one end node to another traverse the same path as a
> unicast packet? I personally would not try to configure a path which uses
a
> higher MTU for a multicast group vs a lower MTU for other path records,
but
> in general the SM is not restricted by this requirement. ( It is possible
> that the SM chooses 4x 1K MTU links for normal data and 1x 4K mtu links
for
> Multicast data, if by some chance, the topology forces it to do so. If
some
> switches do not support multicast and atleast one path per node exists
such
> that multicast can be supported for the fabric as a whole, the end-to-end
> paths for the unicast packets may differ from that of multicast packets
for
> any given pair of nodes and hence the mtus can be different, more
> specifically, the mtus can be larger for multicast than for unicast. ) If
it
> is true that multicast and unicast packets traverse the same paths
> end-to-end or if it is reasonable to assume the MTU to be that of the
> Multicast group because that is the best possible configuration, I do not
> see any issue, except that such a statement of restriction be made on the
> subnet management for use with ipoib. I think such a restriction is
> reasonable for operation of IPoIB though I do not know if it is essential,
> neither can I offer an alternative at this point. Do you think that such a
> statement of restriction is required?
>
>  What about the maximum static rate of injection which is part of the path
> record? This parameter is not part of the Multicast Group record and is a
> function of the DLID for any node injecting packets into the fabric. The
> compliance statments for static rate injection are called optional. But
from
> the compliance statements I could not determine whether they were optional
> contingent of the CA not supporting rates higher than 2.5 GBPs or they are
> truly optional.

You might have already got the answer based on the recent discussions
between me, Joe and others.


>
>  PS: You can ask me to shut up if it is kind of late for such questions?
:)




>
> -thanks,
> Ashwin
>
> -----Original Message-----
> From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
> Of Vivek Kashyap
> Sent: None
> To: Ashwin Kamath
> Cc: ipoverib@ietf.org
> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>
>
> >
> > There is a general thought that ARP is a one step solution for obtaining
> the
> > parameters required for establishing IPoIB over UD. But I think that
even
> > for using ARP, one will require to get the Path record information. I do
> not
> > think that ARP is a one stop solution. What happens if both nodes
> determine
> > that their MTU is 2K without considering that a switch component in
> between
> > might restrict the MTU to 256 bytes? Such information can be obtained
only
> > from the central SM via a path record. Am I missing something here? If
>
>         The all-nodes multicast(IPv6)/broadcast(IPv4) addresses are
>         mapped to a multicast GID. This multicast GID is setup with
>         the MTU for the subnet/link. All nodes always IB_join this
>         MGID. Thus the MTU is always determined from the multicast
>         GID. So is the Q_Key to be used. See the ipoib architecture
>         and multicast drafts that have been posted. You can get them
>         from the ipoverib pages at IETF site.
>
>         Other parameters LID/GID/QPN are received via ARP. SL is
>         determined based on the implementation. So, ARP is and
>         certainly can be a one-step solution like you mention in the
>         begining of your note.
>
>
> Vivek
>
> > address resolution is all that ARP is required to do, without trying to
> > establish parameters for communication one may not want to know the MTU,
> but
> > from the discussions I see that the Address resolution is being used for
> > establishing communication parameters (like the QPN)
> >
> > So the only real problem with having the LID in the ARP response, that I
> see
> > is when the ARP response needs to cross IB Routers. Can I assume that
> > changes in LID assignments are not a big problem because such changes
are
> > not intended to be frequent and current ARP cache implementations
already
> > factor in the possibility of a link level address change. Changing LIDs
> > would have a bigger impact on shared devices using RC than on IPoIB
using
> > UD. Is it ok if I consider a change in LID assignment to be equivalent
of
> a
> > server reboot?
> >
> > If I understood what Jerry, one will have a problem if the ARP responder
> > lies in another IB Subnet, in which case the LID in the response will
not
> be
> > relavant to the originator of the ARP request, in-fact the LID will have
> to
> > be the LID of the router. Was there any resolution reached for this
> problem?
> >
> > In short you do need an ARP + Path record diagnosis to determine all the
> > parameters for communication. What would happen if we bring an IB router
> in
> > between? One cannot obtain a path record to a node in another subnet.
One
> > can obtain the path record to the IB Router. Should we be talking about
> the
> > behaviour of the IB router in case of mismatch of MTUs or should we say
> that
> > we will always use the smallest mtu for communicating across an IB
router.
> > Or should we say that this could fall on the plate of the Infiniband
WGs.
> >
> > -Ashwin.
> >
> > -----Original Message-----
> > From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
> > Sent: None
> > To: kamath@omegaband.com
> > Cc: ipoverib@ietf.org
> > Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
> >
> >
> > My comments are embeded in the following.
> >
> > >I agree with using GID both for longevity and scope. What about using
> > >SIDR(Private Data) for communicating all the capability bits and any
> other
> > >extras? If it is not too late for this, I have some new feedback.
> > >
> > >For ARP on an IB subet one would require that
> > >a. IB Switches support multicast. (This is true in most cases today.)
> > >b. IP addresses used in an infiniband subnet are unique for use by
IPoIB.
> > >(This is an important constraint which gets rid of the ethertype field)
> > >c. Some issues related to Multicast over IB need to be resolved (like
> > >multiple consumers using multicast resources.)
> >
> > This one is really simple. A MGID contains 128 bits. Even IPv6
recognizes
> > that it only needs the lower 32bits for the forseeable future. So the
> > probability of MGID collisions is negligible. If one really wants
> > something bullet-proof, we can ask IBTA to reserve a chunk (or endorse
> > the IPoIB signature covered in the link and multicast draft) for IPoIB.
> >
> > >
> > >Requirement (b) was addressed by defining a multicast group and the IP
> > >address format. One would still have to deal with the GID, LID and the
IP
> > >address. Including the LID in the ARP Response will be difficult to
deal
> > >with because
> > > a. LID Assignments change.
> > > b. ARP might have to traverse a router, for IP Subnets spanning IB
> Subnets
> > >(a) might not really be an issue because ARP entries are aged. In
> ethernet,
> > >the ethernet topology might change and MAC Addresses can change on
server
> > >reboots (by way of using a different MAC or by assigning a MAC
Address.)
> > The
> > >current ARP implementations do take these scenarios into account.
> Moreover
> > >Path record lookups alone would not help resolve the issue of changing
> > LIDs.
> > >The locally cached path record will not be valid if the LID changes.
For
> a
> > >true solution, there has to be some way of using IB Traps to get
> > information
> > >about topology changes as and when it happens. An end node can
subscribe
> > for
> >
> > This is a generic, not IPoIB specific issue. So the work belongs to IBTA
> > to ensure some mechanism exists for flushing stale path records.
> >
> > >a trap, but the end node will have to do something like,
> > > * Tell me when all the switches on the path to this node report a
state
> > >change on such and such a port.
> > > This will not work very well right now because one will have to
> subscribe
> > >for traps on all switches leading to the end node, which implies that
an
> > end
> > >node interested in the existence of another end node will need to have
> > >knowledge of the complete IB topology. This is probably something worth
> > >considering in the Infiniband WGs.
> >
> > Ok, we're saying the same thing.
> >
> > >
> > >On the same token, if you use the globally unique GID, information
about
> > the
> > >GID will not be maintained by the local subnet's subnet administration
> and
> > >hence a path record lookup based on the GID would fail.
> >
> > In this case you follow the IBA rule. The rule says that you don't even
> > attempt to lookup a pathrecord for a remote GID in a different IB
subnet.
> > You need to find an IB route to it first. This is similar to IP.
> >
> > > So if the GID lookup
> > >fails, one would send the ipoib packet to the designated IB router (as
is
> > >done with ethernet today, albiet based on netmasks). But what if you
have
> > an
> > >IP subnet spanning IB Subnets. The designated router exists for an IP
> > Subnet
> > >and the local subnet IB administration will not have information about
> all
> > >the IP nodes, because they could exist on another IB Subnet.
> >
> > Hmmm, you seem to confuse routing at the IB level with that at the IP
> > level. (Or am I the one who is confused?)
> >
> > >
> > > The following IB transactions are required before sending any IPoIB
data
> > >over the subnet (assuming the multicast join is already done and we use
> > GIDs
> > >as link layer addresses)
> > > 1. IpoIB ARP.
> > > 2. Path record lookup.
> > > 3. SIDR
> > >
> > >In all this, the existence of the ARP is questionable. The same
> > >functionality can be achieved using 2 and 3 above.
> > >If you use GIDs as the IP addresses ,you can use a Path record lookup
and
> > >then use SIDR to get all the information to use IPoIB. The advantages
are
> > >evident, for eg,
> >
> > I think similar idea has been brought up before. One major obstacle here
> > is that a given interface may host many (e.g. a few thousand) IP
addresses
> > simultaneously (e.g. virtual hosting). You just don't have space for
that
> > many GIDs on a HCA.
> >
> > Jerry
> >
> > > a. Traversing a router. Assuming that the path record lookup fails,
one
> > >will redirect all the related ipoib packets to the Gateway/router.
Since
> > ARP
> > >does not exist, one does not have to worry about the issues involved by
> ARP
> > >traversing a router, or the router proxying for end nodes on the same
IP
> > >Subnet but different IB Subnets. IP and IB Subnets will be one and the
> > same.
> > > b. Debate on using the ether-type field vanishes. The Service ID will
be
> > >unique to IPoIB and will be assigned by some WG. You could potentially
> use
> > >two Service IDs (might be an overkill), one for IPv6 packets and
another
> > for
> > >IPv4.
> > > c. No special format for IP addresses is required (I have not thought
> this
> > >through very well.)
> > > d. Switches are not required to support Multicast.
> > > e. There are no issues with multiple consumers using the same
multicast
> > >resources.
> > > f. Multicast could be used by an IB Gateways only to avoid
> multi-unicast.
> > >
> > >In this scenario the IP subnet is the same as an IB Subnet, since the
> same
> > >network addresses are used.
> > >
> > >-Ashwin
> > >
> > >
> > >
> > >-----Original Message-----
> > >From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
> > >Of H.K. Jerry Chu
> > >Sent: None
> > >To: ipoverib@ietf.org
> > >Subject: [Ipoverib] What's inside an IPoIB link layer address?
> > >
> > >
> > >While proposals beyond the basic IPoIB over UD started showing up,
> > >I still don't think we've reached a rough consensus on some basic
> > >questions such as what the IPoIB link layer address should contain.
> > >Again I'd like to apologize for not being there at the SLC meeting
> > >last month. Perhaps a consensus was reached there but i've missed
> > >the boat. In that case please let me know what it is and I'll shut
> > >up.
> > >
> > >What I'd like to see included in the IPoIB link layer address is
> > >
> > >GID
> > >QPN
> > >
> > >and I'd like to allow QPN field to be 0.
> > >
> > >What I don't like to see included is LID.
> > >
> > >Discussions:
> > >
> > >There has been a consensus of using GID, not LID as the basic
> > >IB address identifying the port hosting an IP address. GID is
> > >sufficiently stable, not subject to change across a node reboot
> > >or subnet merge (IB level) like LID is. Also people don't want
> > >to exclude the possibility of IPoIB link spanning more than
> > >one IB subnet. GID supports scope beyond local link so it's
> > >suitable for that purpose.
> > >
> > >Multiple GIDs with different scopes may have been assigned to a CA.
> > >It is required that a CA returns the GID with a scope no less than
> > >the one for the IPoIB link. E.g. if an IPoIB link covers multiple
> > >IB subnets, but a CA uses a link-local GID as part of its IPoIB
> > >link layer address, it's not going to work well.
> > >
> > >I don't see any value of including LID besides GID as proposed
> > >by Vivek. I believes a lookup of PathRecord is always needed
> > >in order for a node to retrieve complete path info before sending
> > >unicast pkts to another node. The destination LID is part of
> > >the PathRecord so including it in the link layer address returned
> > >from ARP/ND doesn't save work. But you have all the drawbacks that
> > >swayed people away from using LID - it adds volatility to the link
> > >address. LID changing due to subnet merge or other reasons requires
> > >potentially new mechanism being added to IP/driver implementation
> > >to trigger ARP/ND flush. Lastly, what should be the LID if the IP
> > >node is in a different IB subnet? Obviously it can't be the LID of
> > >an intermediate IB router.
> > >
> > >For QPN I'd like to suggest one thing. Since SIDR is a required
> > >feature by IBA, the QPN field is allowed to be zero. What that
> > >means is that the requester must use SIDR to retrieve the remote
> > >QPN, given the GID of a remote IP address.
> > >
> > >This feature allows the support of static/persistent "MAC addr"
> > >wrt node reboot by removing the volatile part of the link layer
> > >address - the QPN. In many implementations the QPN allocated for
> > >a CA may change across node reboot. Support of static "MAC addr"
> > >has been considered desirable by a number of people.
> > >
> > >I don't like to add additional requirement such as SIDR to IPoIB,
> > >and was trying to make the above feature "optional". Ideally nodes
> > >can express whether they'd like QPN to be returned by embedding
> > >some flags, such as "I don't talk SIDR so please return QPN" in
> > >their IPoIB link layer addresses. But the broadcast nature of
> > >IPv4/ARP forbids the use of different "MAC addrs" for the same IP
> > >address, so my attempt failed.
> > >
> > >Comment?
> > >
> > >Jerry
> > >
> > >
> > >_______________________________________________
> > >IPoverIB mailing list
> > >IPoverIB@ietf.org
> > >http://www.ietf.org/mailman/listinfo/ipoverib
> > >
> > >
> > >_______________________________________________
> > >IPoverIB mailing list
> > >IPoverIB@ietf.org
> > >http://www.ietf.org/mailman/listinfo/ipoverib
> >
> >
> >
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> >
>
>
> --
> Vivek Kashyap
> Linux Technology Center, IBM
> kashyapv@us.ibm.com
> vivk@us.ibm.com
> 503 578 3422 (o)
>
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
>
>
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
>


--
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com
vivk@us.ibm.com
503 578 3422 (o)


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 18:57:16 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA16700
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 18:57:16 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id SAA27518
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 18:57:19 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA27432;
	Tue, 15 Jan 2002 18:56:17 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA27336
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 18:56:12 -0500 (EST)
Received: from palrel10.hp.com (palrel10.hp.com [156.153.255.245])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA16677
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 18:56:08 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel10.hp.com (Postfix) with ESMTP id DE07540073C
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 15:55:41 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id PAA10840
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 15:55:41 -0800 (PST)
Message-Id: <5.1.0.14.2.20020115150819.028a7420@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Tue, 15 Jan 2002 15:22:04 -0800
To: ipoverib@ietf.org
From: Michael Krause <krause@cup.hp.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
In-Reply-To: <200201152157.g0FLvad15693@eng4.beaverton.ibm.com>
References: <200201152107.g0FL7gw13021@eng4.beaverton.ibm.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

At 01:57 PM 1/15/2002 -0800, Vivek Kashyap wrote:

> > I don't think pathrecord lookup can be avoided or should be skipped.
> > You're proposing something that IMHO is beyond my understanding of
> > how things work in IBA. Can you get an experienced IBTA member
> > to substantiate your statements here?
>
>         There has been some discussion on this issue. As noted SL
>         certainly doesn't require going to the SM/SA. The path-rate
>         support is mandatory if sending > 2.5Gbps but not determining
>         it from SM/SA. To my knowledge IBTA is actively pursuing a
>         dynamic congestion management solution that would not depend
>         on static path rate. So this is not an argument to tie down
>         LID determination to the SA.

Congestion management would be optional functionality given the V1.0 
specification has been public for over a year now.  As such, it cannot be 
relied upon to solve any problems only to improve a solution over 
time.  Further discussion into the details is not permitted due to the IBTA 
confidentiality rules but it is fair to state that injection rate control 
is one lever used by congestion management but congestion management does 
not lead to injection rate control which may be driven more by QoS 
requirements which go beyond the scope of congestion management.

The IP over IB spec should define the architectural minimums needed to 
implement a credible IP over IB solution and leave optimizations such as 
what have been debated these last few days to vendors to solve.  If the 
optimizations prove worthwhile after experience has been gained in real 
customer deployment situations such that a subsequent spec should be 
generated, then the workgroup can provide a new spec that provides an 
interoperable solution using these new techniques.

> > And what's the real benefit of saving just a lookup out of an already
> > infrequent operation like ARP due to cacheing?
>
>         I would like to know what is the drawback that you see that
>         you don't want one to go ahead with it? ARP is repeated every
>         so often and happens quite often. It also happens on failover
>         - it is preferable that these operations be simple, fast and
>         error-free.

No one is arguing with any of these assertions.   The issue is strictly 
what is the architectural minimum the spec should define and what should be 
left for future spec or vendor implementation to solve.


>         Secondly, things are more complicated the way you propose.
>         Note that the first level address would be first determined.
>         This completes ARP/ND. However, address resolution is not
>         complete. The packet is sent as far as IP is concerned and now
>         the SA/SM is consulted to get the LID etc. This has very
>         strong shades of ATM VCs. IP thought it had sent the packet
>         since the ARP/ND was complete - it took a while before the
>         lower layer completed the determination. The driver of course
>         keeps a GID to LID table too since the LID is a separate
>         entity. More complication.

This is one implementation choice and while it may be used by some, it does 
not require the spec to make this a requirement.


>         Include LID in the resolution - fast/simple/error-free.

It isn't that simple either

>         I was pointing out another issue that many have raised.
>         In general I am ok with dropping GID since it is for the
>         future. IT buys us nothing as of now. So if we dropped GID we
>         can work with these 48 bits. When IPoIB in reality starts
>         spanning multiple IPoIB subnets (not for a long time) we can
>         include GID. Then the hardware length will become 22 bytes as
>         against 6 bytes. One can still interoperate quite easily.

GID is required because it is the primary way and application determines 
where a service resides on a given port as such not including it makes life 
quite difficult to implement.  GIDs are also used by applications within 
the address vector to identify a given vector to use for a given service 
such as IP.  GIDs cannot be dropped and have value.

LIDs are strictly a local issue that will be learned for all CA / router 
ports participating within a given partition that support IP services.  As 
such, their assignment, management, transitory nature, etc. make them 
unsuited as an architectural minimum to include within the spec.  I see no 
problem with people ascertaining the value and using it for optimizing 
their implementation but this is outside the scope of the specification.

>         Wouldn't that work for all ?
>
>         I included the GID because many want to put in the hooks for
>         the future case of IPoIB spanning IB subnets. I would though
>         like a valid reason specified for why do we need IP subnets to
>         span IB subnets. Isn't IP subnetting enough?

IB is viewed as a layer 2 fabric w.r.t. IP so the fact that IB has multiple 
subnets is independent of IP operation.  I see no point in dealing with the 
IB subnet issues within this specification other than to state that a LID 
is not an end-to-end address between two endnode ports and thus should not 
be mandated within the IP over IB spec for address resolution.

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 18:57:21 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA16715
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 18:57:17 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id SAA27532
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 18:57:19 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA27383;
	Tue, 15 Jan 2002 18:56:14 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA27335
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 18:56:12 -0500 (EST)
Received: from palrel10.hp.com (palrel10.hp.com [156.153.255.245])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA16678
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 18:56:08 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel10.hp.com (Postfix) with ESMTP id 21F56400204
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 15:55:42 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id PAA10843
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 15:55:41 -0800 (PST)
Message-Id: <5.1.0.14.2.20020115152217.029fd5a0@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Tue, 15 Jan 2002 15:29:17 -0800
To: ipoverib@ietf.org
From: Michael Krause <krause@cup.hp.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
In-Reply-To: <200201152212.g0FMC8816612@eng4.beaverton.ibm.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

At 02:12 PM 1/15/2002 -0800, Vivek Kashyap wrote:
>Jerry,
>
>
> > >As for the why LID:
> > >I think that going to the SM is not so efficient given the fact that 
> it is a
> > >central resource and all the nodes in the subnet go to if we could 
> avoid it
> > >why not
> >
> > First the benefit is dubious. In modern TCP/IP stacks the frequency of ARP
> > requests is capped by the ARP cache flush internal, which is in minutes
> > range. (E.g. in Solaris it's five minutes.) Assuming you have 100 nodes
> > on a subnet, you're telling me saving the SM lookup call, which at max
> > rate (if a local node is activetly talking to every other node) of one
> > every 3 sec. is worth noting?
>
>         What if there are 500/1000 or 3000 nodes ?

Let's see, we are all complaining about an operation that may occur some N 
operations unit of time where N may be somewhat large but the end result is 
still something measured in 10's if not 100's of milliseconds in terms of 
interval.  Given the type of platforms delivered in such a time frame, it 
seems inconceivable that such a workload would impose any performance hit 
on either the fabric or the endnode.

>         Besides, as I have
>         noted in the last mail the process is more complicated not
>         only internally but also requires 6 messages where only 2
>         would have sufficed. More complication means more maintenance,
>         slower failover, more bugs etc. We also know that centralised
>         systems don't scale well. That is why we are using all-nodes
>         multicast and reply rather than go to some central ARP
>         server.

I don't agree.  It is an implementation issue as to whether one implements 
the additional messages (which may already be required for other purposes 
between endnodes such that there is no real net savings).   One can also 
not flush the ARP cache so often given the relative stability of the IB 
fabric elements thus making the workload potentially much larger.

As for whether something central scales well or not that is really a 
function of the type of service being delivered.  A very complicated IB 
fabric will use various techniques to simplify and create scalability for 
management operations.  I can easily see a value-add management service 
that replicates the SA database across a set of nodes thus allowing the 
workload to be horizontally scaled.  One can create various policies as to 
how this data is accessed (there is nothing that precludes a solution 
provider from delivering the data through a multicast-based request / 
unicast response given these are primarily on a LID-based communications QP 
which would be available given the requirement to have IP run over a 
multicast switch environment).

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 18:57:29 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA16734
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 18:57:28 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id SAA27548
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 18:57:31 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA27452;
	Tue, 15 Jan 2002 18:56:18 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA27347
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 18:56:13 -0500 (EST)
Received: from palrel12.hp.com (palrel12.hp.com [156.153.255.237])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA16681
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 18:56:09 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel12.hp.com (Postfix) with ESMTP id 920B1E0017E
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 15:55:42 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id PAA10846
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 15:55:42 -0800 (PST)
Message-Id: <5.1.0.14.2.20020115153047.09da3a18@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Tue, 15 Jan 2002 15:39:10 -0800
To: ipoverib@ietf.org
From: Michael Krause <krause@cup.hp.com>
Subject: Re: [Ipoverib] IPoIB architecture/encapsulation (EtherType or
In-Reply-To: <200201152134.g0FLYma14366@eng4.beaverton.ibm.com>
References: <5.1.0.14.2.20020114142357.029b58f0@esalpha2.cup.hp.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

At 01:34 PM 1/15/2002 -0800, Vivek Kashyap wrote:

> >
> > Link address fields:
> >
> > P_Key - For each partition that an endnode participates and requires IP
> > over IB services, it will send an ARP request to identify other nodes on
> > the IP subnet.  Only those endnodes that share a common P_Key will respond
> > and thus be discovered.  Hence, all discussion of P_Key is irrelevant from
> > the IP over IB perspective and should not be included within the IP 
> over IB
> > specification.
>
>         Agreed except to ensure unique MGIDs within partitions we have
>         had to include the P_Key into the IP multicast to MGID
>         mapping.

MGID is an IB MGID that is obtained by an endnode joining an IB multicast 
group.  When it joins such a group, it be definition is provided the P_Key, 
etc. parameters.  As such, there is no need to have P_Key used for the IP 
multicast to MGID mapping - the identification of the IP service implicitly 
will provide the requisite values.

> > Q_Key - Q_Key provides no value to IP over IB solutions and should be
> > treated as a well-defined value by the specification.  Recommend a 
> value of
> > 0xFFFFFFFE be used.
>
>         This is left to the admin. The value set in the broadcast-GID
>         is used for all communication.

I believe the simplification to a well-known value will provide better 
interoperability as Q_Key in this context provides no benefit and having it 
administered seems counter productive.


> >
> > GUID - The value of a GUID is primarily being ascertain how things are
> > co-located on a given CA.  At first glance this seems to have some value
> > but most people no longer think it is required to establish a 
> communication
> > channel between endnodes, in other words, use the GID to define the port
> > and let software figure out any co-location using local means if
> > required.  Hence, this field should not be included for IP over IB 
> operations.
> >
> > GID - In reality there are two ways of creating GIDs within IB (the
> > original IP over IB architecture discussion isn't quite accurate). The
> > first (called GID 0) and is created by concatenating the port's EUI64 with
> > the GID prefix to create a 128-bit value.  The second by concatenating a
> > SM-assigned 64-bit value with the GID prefix.  This second method provides
> > a way for endnodes to have a variety of link addresses (e.g. one per OS
> > instance, per set of IP addresses per endnode, etc.).
>
>         You can create any form of interface within your machine and
>         tie it to any number of link addresses; these can be LIDs from
>         different ports. Whatever GID is chosen it is finally tied to
>         a port and the LID needs to be determined. Having different
>         link addresses don't provide any gain in themselves if they ar
>         tied to the same port. You can differentiate more easily by
>         using QPs. If the ports are different then use the LIDs
>         anyway.

The example provided illustrates why multiple GIDs and the use of GID as 
part of address resolution has value.  As such, different link address has 
value and can be used as part of a QoS strategy for different IP workloads 
as well as multi-OS instances per port.

> > In general, given an
> > IP subnet may span multiple IB subnets, this is the mandatory value to
> > exchange to uniquely identify an endnode for address resolution purposes.
>
>         Why do we need to span multiple IB subnets? The fabric is
>         broken into IB subnets for isolation. Why should we want to
>         rejoin them when they are separated by design ?

See another response but the essence is IB is a layer 2 from an IP 
perspective and there is no value in not supporting a transparent IP subnet 
across multiple IB subnets.  People are already managing IP subnets and 
their associated infrastructures so why not allow this to be supported from 
day one.

> > QP - There is some benefit to exchanging the QP as part of address
> > resolution.  For example, an endnode may have different policies in terms
> > of how a QP is mapped per IP subnet.  One might have one QP per IP address
> > or may have one QP per multiple IP addresses (potentially for all IP
> > communications).  As such, the ARP process should exchange the QP where it
> > is known and should use QP 0xFFFFFF when it is unknown, e.g. an IP router
> > at the edge of an IB fabric instance.
>
>         Any endpoint that will talk IP will have a default QP. It
>         returns that. I am not sure how it cannot know its own QP.

Of course it can know the individual QPs - the method is implementation 
specific as to how.  The spec need only state that a QP will be 
returned.  The local meaning of that QP within the endnode is outside the 
scope of the spec.

> > LID - There has been general discussion that IP subnets should be able to
> > span multiple IB subnets.  As such, the LID is invisible to the address
> > resolution process since this is an IB subnet only issue (whether it 
> refers
> > to an IB router or an IB endnode is irrelevant).  The counter arguments
> > that one could optimize the address resolution process slightly by 
> avoiding
> > a trip to the SA / SM by including the LID as a visible exchanged item (it
> > is always available since this can be derived from the IB packet in the
> > first place) really don't provide much substance.  The resolution process
> > should be able to resolve an address independent of the underlying IB
> > fabric using a single algorithm.  Implementations can take appropriate
> > steps to optimize their communications through smart caching and
> > examination of the provided IB headers.
>
>         As discussed in the earlier mails these issues are important.
>         The argument is that the method to determine any other
>         information is upto the end node's implementation. Determining
>         LID is disjoint from that. Those that don't want to use the
>         LID in the adress resolution packets can always ignore the
>         value on input and consult the SA.
>
>         The process when determining the LID when across a subnet *is*
>         different by definition. In the case of resolving the LID
>         within the subnet the LID is received from the peer/SA but
>         when communicating across IB subnets the LID of the first hop
>         router is received from the routing infrastructure. A common
>         method is not possible.

One can create a common method by simply asking the SA.  Anything beyond is 
an optimization and implementation specific.  Focusing on the architectural 
minimums will lead to a more interoperable solution from the start.

> > GRH - There has been some discussion of whether the GRH should be included
> > for all ARP operations or not.  For an ARP request, one injects a 
> multicast
> > packet per partition.  The receiving endnode will receive the packet and
> > can determine whether the source is on the same subnet or not and respond
> > with either a GRH or a LRH-only packet.   As such, it seems GRH usage is
> > implicit and the current IB specifications on usage do not require any
> > additional work within the IP over IB specification.
>
>         Agreed.
>
> >
> > Capability bits - If an IP router is allowed to exist between two 
> endnodes,
> > then the specification must include a method to exchange capabilities 
> (e.g.
> > additional IB transport services that may be used to for IP over IB
> > communications).   Since this topology is allowed, the specification must
> > provide a method to exchange capabilities through some form of standalone
> > operation, i.e. an endnode must be able to resolve an address to an IP
> > router. The endnode will then inject a capabilities enumeration message to
> > the destination endnode.   Given the endnode must be able to perform this
> > operation to guarantee interoperability separate from the address
> > resolution process, it would be preferred if the capability discovery 
> (only
> > required if an endnode provides additional services beyond IP over IB 
> which
> > is not necessarily universal) be kept a separate operation.  Once the
> > capabilities are determined, these can be cached for a fairly substantial
> > amount of time such that there is no performance loss for the majority of
> > the operations that may occur between endnodes.  In addition, the
> > capabilities exchange may be used to provide a target QP to be used for
> > each capability thus allowing one to optimize the communications to
> > initiate / implementations of these capabilities.
>
>         Please see the
>         draft-kashyap-ipoib-advanced-capabilities-00.txt. I think I
>         have taken good care of interoperability. The default case
>         which doesn't support RC/QPN modes work seamlessly with nodes
>         that do.
>
> >
> > Ethertype - Quite a bit of discussion has occurred but it may be fair to
> > state that the address resolution provides a first level method to obtain
> > the associated QP for a given IP address and that the capabilities 
> exchange
> > may be used to provide additional QP for a given capability (e.g. 
> perform a
> > RC establishment using QP X), it seems the Ethertype is no longer 
> necessary
> > and can be removed from the specification.
> >
>
>         Agreed.

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 19:37:53 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA17147
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 19:37:52 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id TAA28839
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 19:37:53 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA28821;
	Tue, 15 Jan 2002 19:37:19 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA28786
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 19:37:18 -0500 (EST)
Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA17141
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 19:37:13 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e31.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id TAA114984
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 19:34:13 -0500
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0G0bEr85214
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 17:37:14 -0700
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id g0G0bCg27169;
	Tue, 15 Jan 2002 16:37:12 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0G0bB424382;
	Tue, 15 Jan 2002 16:37:11 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201160037.g0G0bB424382@eng4.beaverton.ibm.com>
To: ipoverib@ietf.org
Date: Tue, 15 Jan 2002 16:37:10 -0800 (PST)
Cc: kashyapv@us.ibm.com
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Subject: [Ipoverib] link descriptor in address resolution packets
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit


        I feel the argument on including the LIDs vs. GIDs is
        revolving round two issues:

	GIDs : Why are they needed ?

                They cannot be justified by saying that IP subnets can
                span IB subnets because no benefit has been listed.
                Nor is the IB specification anywhere near providing
                such an infrastructure.

	LIDs : Get from SA

                It is wrong to link LID lookup to getting the
                information from the SA because other information is
                kept at the SA. This is because the other information
                can be got in implementation dependent way. The two
                are disjoint.


        How about agreeing to disagree and do the following in
        address-resolution packets:


	GID + QPN + Cap flags + LID


        The order is as specified above. Those that don't want to use
        the LID can ignore it in received packets and not record it in
        their information. All they are asked to do is record their
        own LID when sending the address-resolution request/reply.
        They ignore the LID in the request/reply they receive. Note
        that this format has nothing to do with the format in which
        the link address is recorded in the system.

	That should work for all implementations.

Vivek

-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 20:03:32 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA17488
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 20:03:32 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id UAA29921
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 20:03:33 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA29902;
	Tue, 15 Jan 2002 20:02:41 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA29870
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 20:02:40 -0500 (EST)
Received: from sentinel.sanera.net (ns1.sanera.net [208.253.254.10])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA17472
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 20:02:37 -0500 (EST)
Received: from icarus.sanera.net (icarus [192.168.254.11])
	by sentinel.sanera.net (8.11.2/8.11.2) with ESMTP id g0G129u23389
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 17:02:09 -0800
Received: from newton.sanera.net (newton.sanera.net [172.16.2.11])
	by icarus.sanera.net (8.11.6/8.11.6) with ESMTP id g0G124n28339
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 17:02:04 -0800
Received: from sanera.net (dhcp-pc-100-161.sanera.net [192.168.100.161])
	by newton.sanera.net (8.11.4/8.11.4) with ESMTP id g0G123M06122
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 17:02:03 -0800 (PST)
Message-ID: <3C44D10B.9708CAAE@sanera.net>
Date: Tue, 15 Jan 2002 17:02:03 -0800
From: Joe Pelissier <joep@Sanera.net>
X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U)
X-Accept-Language: en
MIME-Version: 1.0
To: ipoverib@ietf.org
Subject: Re: [Ipoverib] Summary of the IPoIB link layer address debate
References: <4D027986353D1341ADA4F2A4F8A170760E316F@cceexc18.americas.cpqcorp.net>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Hi Paul,

Here's an interesting question...
For sending the multicast ARP request, I know which SL and MLID to use since
I am a member of the multicast group.

When I respond to the ARP request, I'm sending unicast.  I know the DLID to
use, but how do I know what SL to use?  While the IB spec does not prohibit
me from using the SL in the ARP request packet, it also does not guarantee
that the SL in the ARP multicast request is valid for the ARP unicast
response.  If the SM is using VL to for deadlock avoidance (a perfectly
reasonable usage), then the ARP response could be lost.

As you stated, once the requester receives the response, it does have a
valid SL to use.  But I think the responder is in a bit of a quandary.

Thoughts?
Joe Pelissier

"Culley, Paul" wrote:

> As has been pointed out several times in the past, a unicast ARP
> response packet already includes the LID and an SL, both of which are
> guaranteed (by IBA Spec) to be valid for sending (small) packets back to
> the source of the ARP response packet.
>
> Since this is already present in the IB Datagram header (LRH:SLID,
> LRH:SL) and can be easily retrieved by HCAs built to the IB
> Specification, I cannot see any point in adding a LID to the unicast
> response.
>
> Paul R. Culley
> Compaq Sr. Fellow
> 281-514-5543
>
> -----Original Message-----
> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> Sent: Tuesday, January 15, 2002 4:30 PM
> To: Roger Ronald
> Cc: Vivek Kashyap; ipoverib@ietf.org
> Subject: Re: [Ipoverib] Summary of the IPoIB link layer address debate
>
> >
> > It seems like the general debate that's raging is trying to answer
> > a single question:
> >
> > "Is it wise to include the LID in order to enable a
> > shortcut that allows bypassing a trip to the SM/SA?"
> >
> > My read is that second party dissemination of LIDs is not
> > something that is prohibited (per the IBTA rules), but it is a
> > shortcut that the IBTA architects weren't expecting
> > and are uncomfortable seeing. Various objections
> > about PathRecords, rate control, SL, and MTU have
> > been mentioned, but don't seem to make for a conclusive
> > decision point.
> >
> > In the end, the tradeoff seems to be simplicity versus
> > flexibility. Leaving out the LID makes the header
> > smaller and the processing is always done in one
> > way. Putting the LID into the header allows
> > skipping a step under (at least) some circumstances.
>
> Leaving out the GID makes the header a lot less smaller. I am not sure
> what process is done in one way ? If you are talking about spanning IB
> subnets with a single IPoIB subnet the process is definitely not done
> in a single way. It is more easily a single way if the LID is used.
> This is noted in my note below.
>
> > > applies all over IB components and is not limited to LIDs/GIDs.
> > >
> > > Let us first answer the question of why IPoIB subnets must span IB
> > > subnets ? We don't know why we need such a setup ? We don't know
> when
> > > such a setup will be possible ? We do know that IPoIB subnets within
> > > an IB subnets can be setup easily with the current specification.
> > >
> > >
> > > Secondly, if you step back you'll realise that LID is hardly an
> issue
> > > when resolving across subnets. Unlike what has been postulated the
> > > process is not the same when resolving within the IB subnet and when
> > > resolving across the IB subnet.
> > >
> > > Assumption: at IP level- a multicast request; receive a unicast
> reply
> > >
> > >         Now, when sending the broadcast the sender doesn't know
> where
> > >         the responder is. So he includes his LID/GID. The receiver
> > >         gets it across multiple subnets. It gets sees a different
> > >         GID-prefix and hence replaces the LID with its default
> router
> > >         when recording the address. The same happens on the
> response.
> > >
> > >         It works fine for both situations - IPoIB subnets
> > >         within/across IB subnets. You can easily have it that if the
> > >         GID is local the routing lookup will return the local LID.
> > >         Single process.
> > >
> > >         In the case of not including the LID you do the same thing.
> If
> > >         the GID prefix is local you go to the SM/SA to get the LID.
> If
> > >         the GID prefix is different you record your default router's
> > >         LID. So the process for across the subnet is the same but
> you
> > >         shortchange the implementation that is more likely - the one
> > >         whithin the subnet.
> > >
> > >         For within the subnet you use a different method - going to
> > >         The SA always. Not the same as going to the routing
> > >         infrastructure.
> > >
> > > However, I would still like to know what is the use of IPoIB
> spanning
> > > multiple subnets. Why do you think that is a requirement ?
> > >
> > >
> > > Vivek
> > > --
> > > Vivek Kashyap
> > > Linux Technology Center, IBM
> > > kashyapv@us.ibm.com
> > > vivk@us.ibm.com
> > > 503 578 3422 (o)
> > > --
> > > Vivek Kashyap
> > > Linux Technology Center, IBM
> > > kashyapv@us.ibm.com
> > > vivk@us.ibm.com
> > > 503 578 3422 (o)
> > >
> > > _______________________________________________
> > > IPoverIB mailing list
> > > IPoverIB@ietf.org
> > > http://www.ietf.org/mailman/listinfo/ipoverib
> > >
> > >
> >
> >
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> >
>
> --
> Vivek Kashyap
> Linux Technology Center, IBM
> kashyapv@us.ibm.com
> vivk@us.ibm.com
> 503 578 3422 (o)
>
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
>
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 20:13:26 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA17582
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 20:13:26 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id UAA00272
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 20:13:27 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA00148;
	Tue, 15 Jan 2002 20:11:01 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA00112
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 20:11:00 -0500 (EST)
Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA17556
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 20:10:57 -0500 (EST)
Received: from sunmail1.Sun.COM ([129.145.1.2])
	by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id RAA13876;
	Tue, 15 Jan 2002 17:10:57 -0800 (PST)
Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.82.166])
	by sunmail1.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1-Sun.COM.mod.2) with ESMTP id RAA21409;
	Tue, 15 Jan 2002 17:12:08 -0800 (PST)
Received: from taipei (taipei.Eng.Sun.COM [129.146.85.178])
	by jurassic.eng.sun.com (8.12.2+Sun/8.12.2) with SMTP id g0G1AuqT366588;
	Tue, 15 Jan 2002 17:10:56 -0800 (PST)
Message-Id: <200201160110.g0G1AuqT366588@jurassic.eng.sun.com>
Date: Tue, 15 Jan 2002 17:08:27 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Subject: Re: [Ipoverib] link descriptor in address resolution packets
To: kashyapv@us.ibm.com
Cc: ipoverib@ietf.org
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: nkfQlA5OCcNf+tJdV85ThA==
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.6_06 SunOS 5.8 sun4u sparc 
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

>        I feel the argument on including the LIDs vs. GIDs is
>        revolving round two issues:
>
>	GIDs : Why are they needed ?
>
>                They cannot be justified by saying that IP subnets can
>                span IB subnets because no benefit has been listed.
>                Nor is the IB specification anywhere near providing
>                such an infrastructure.

No that's not the main reason. GIDs is highly persistent compared to
LIDs. There is a great deal of "hidden" benefit from having a persistent
component in a "MAC address" that have been pointed out many times before
(e.g. for diagnosis purpose, less dependency on gratuitous ARP getting
through...)

>
>	LIDs : Get from SA
>
>                It is wrong to link LID lookup to getting the
>                information from the SA because other information is
>                kept at the SA. This is because the other information
>                can be got in implementation dependent way. The two
>                are disjoint.

The linkage is there because you argued to include LID for performance
reason, but a high percentage of implementations that require a trip to
SA won't realize any performance on an already infrequent ARP operation.

>
>
>        How about agreeing to disagree and do the following in
>        address-resolution packets:
>
>
>	GID + QPN + Cap flags + LID
>
>
>        The order is as specified above. Those that don't want to use
>        the LID can ignore it in received packets and not record it in
>        their information. All they are asked to do is record their
>        own LID when sending the address-resolution request/reply.
>        They ignore the LID in the request/reply they receive. Note
>        that this format has nothing to do with the format in which
>        the link address is recorded in the system.
>
>	That should work for all implementations.

You can't do that without violating layering. ARP modules in most stacks
are independent of link types. Driver is not supposed to peek into ARP
packets.

Jerry

>
>Vivek
>
>-- 
>Vivek Kashyap
>Linux Technology Center, IBM
>kashyapv@us.ibm.com 
>vivk@us.ibm.com 
>503 578 3422 (o)
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 20:20:14 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA17655
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 20:20:09 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id UAA00499
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 20:20:11 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA00482;
	Tue, 15 Jan 2002 20:19:22 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA00430
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 20:19:20 -0500 (EST)
Received: from mxout1.netvision.net.il (mxout1.netvision.net.il [194.90.9.20])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA17644
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 20:19:16 -0500 (EST)
Received: from vaio505he ([32.103.135.44]) by mxout1.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GQ000GDMBMJ7B@mxout1.netvision.net.il> for ipoverib@ietf.org;
 Wed, 16 Jan 2002 03:18:47 +0200 (IST)
Date: Wed, 16 Jan 2002 01:14:38 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: [Ipoverib] Summarizing the open issues
In-reply-to: <5.1.0.14.2.20020115091655.00ae5668@esalpha2.cup.hp.com>
To: ipoverib@ietf.org
Reply-to: yaronh@voltaire.com
Message-id: <018401c19e2b$c45ac570$0100007f@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7BIT

Summarizing the points:

I don't understand why to block people that like having the LID from getting
it
We could deal with the issues around it by defining that a node must always
examine and compare the GID first
And not relay on the LID alone
If there are people who thinks avoiding the SM is important to them, and it
doesn't block the others from doing it, why not
I would like to here strong arguments as for what harm could be done by
adding the LID, and not why it is not needed
(Obviously there are people in this group who think it is needed)

No one responded on the Ethertype issue, I hope there is a consensus around
it

As for the capability issues
Since the working group has agreed to first solve UD.
For now we could specify reserved capability bits that
must-be-zero on transmit and not checked on receive, and then define
those bits in the follow-on RFC
this should satisfy everyone

as for the 1X issues, there is no problem any one could use how much he
needs
the problem as I understand is just around the multicast traffic
which all of you state is low performance
also this 1X could be a default value, and could be changed to support more
if desired
(I wonder way some of the people here are active on iWarp if they believe
plain TCP/IP is so efficient without HW offloading)

Yaron

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Michael Krause
Sent: Tuesday, January 15, 2002 7:25 PM
To: ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?

At 06:58 AM 1/15/2002 +0200, Yaron Haviv wrote:
>Vivek,
>
>I agree with your assumptions
>We need them all: LID + capability flags + GID + QPN
>Mikes remark that you could extract the LID from the incoming packet
however
>looks interesting (as long as we make sure that people don't play with it,
>e.g. redirect traffic etc')
>
>As far as I understand the only open issues currently are (if someone
thinks
>there are more please respond):
>LID: Yes/No
>Ethertype
>Capability flags
>
>As for the why LID:
>I think that going to the SM is not so efficient given the fact that it is
a
>central resource and all the nodes in the subnet go to if we could avoid it
>why not

How often do people really think they are going to the SM/SA for any of
this information?  This isn't the Internet; it is a data center fabric.  So
with the exception of a reboot of an individual node and fabric state
events, most implementations will be able to cache their respective IP to
IB mappings for very long periods of time.  This means that the actual
packet rate for IP implementations to the SM/SA will be quite low.

>When IP will span IB the problem becomes more serious: Unlike IP routers
>which has a default route entry, currently IB has explicit entries for each
>path this might require doing too many PathRecord operations, which are
also
>currently very slow when going outside an IB subnet
>(I had posted a proposal to the MSO team trying to resolve those issues
>somehow however it is under discussion)

IBTA proposals remain confidential within the IBTA and should not be
discussed within the IETF.  Currently, the IB router protocol is under
development and as such is not for open discussion.  As to whether the IB
router protocol will modified to support default route entries, uses
explicit route entries, etc. that has not been finalized and is subject to
member review, etc.

Again, I will argue that the rate of endnode change within the IP view is
rather slow and thus the inquiry rate to the SM/SA even for a global view
will not present scalability problems with an intelligent implementation.

>I would tend to send this info anyway since the extra 16 bits wouldn't harm
>anyone, one might choose not to use them if he wants to go to the SM, this
>way everybody will be happy and over time people will use the best solution

It is unnecessary data that could cause confusion and interoperability
problems.

>As for the Ethertype:
>After giving Vivek a hard time on this issue I accept his proposal
>Not because of the performance issues raised (since I don't think they are
>real) But because it is not really needed, and if it simplifies things,
>why not
>
>As for the capability bits:
>Avoiding the SIDR is a smart move
>I'm not sure if the UC option will be used, if using connection why not
>reliable connection
>I just wanted to add SDP (Socket Direct Protocol) bit, Again ones might
>choose not to use it, for the one that do it really simplifies the process
a
>lot

Capability discovery has to work as a separate inquiry method as noted in
my previous response.  Having two methods for a given task is a waste and
creates potential interoperability problems.  It is simpler for the spec to
only specify one method and let implementations do likewise.

>I'm hoping we get a quick convergence on those issues so we would move
>forward
>The issues are not serious and I think if an alternative enable someone to
>do something he wants without violating the other's ability to do what they
>want just ignoring the extra info, we should go for it

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 21:03:15 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA18095
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 21:03:10 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id VAA02041
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 21:03:12 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA02025;
	Tue, 15 Jan 2002 21:02:15 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA01993
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 21:02:13 -0500 (EST)
Received: from mxout2.netvision.net.il (mxout2.netvision.net.il [194.90.9.21])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA18073
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 21:02:10 -0500 (EST)
Received: from vaio505he ([32.103.135.44]) by mxout2.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GQ000E6XDMNMM@mxout2.netvision.net.il> for ipoverib@ietf.org;
 Wed, 16 Jan 2002 04:01:41 +0200 (IST)
Date: Wed, 16 Jan 2002 04:02:16 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: RE: [Ipoverib] link descriptor in address resolution packets
In-reply-to: <200201160037.g0G0bB424382@eng4.beaverton.ibm.com>
To: "'Vivek Kashyap'" <kashyapv@us.ibm.com>, ipoverib@ietf.org
Reply-to: yaronh@voltaire.com
Message-id: <019001c19e31$d3634910$0100007f@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7BIT

Vivek, as you could see I back you in the need for LID
However I'm sure we must have the GID as the primary address
There is a need for IP spanning IB this is what I hear from many of the data
centers I visit
As noted in few of the emails there are also few other reasons for GID

We need to maintain the tables by the GID, however I support taking the LID
to return the answer to the caller (avoiding the SM)
I'm not sure that we could take the LID from the packet since someone might
forward it somehow (like and ARP server or something like that)

No one mentioned in the arguments the additional latency caused by going to
the SM (for the LID)
As a central resource that probably has queuing it might take time
Not to mention the latency in IP span IB case (needing to resolve all the
path)

Give Vivek a break, lets put the LID in the header

Yaron

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Vivek Kashyap
Sent: Wednesday, January 16, 2002 2:37 AM
To: ipoverib@ietf.org
Cc: kashyapv@us.ibm.com
Subject: [Ipoverib] link descriptor in address resolution packets


        I feel the argument on including the LIDs vs. GIDs is
        revolving round two issues:

        GIDs : Why are they needed ?

                They cannot be justified by saying that IP subnets can
                span IB subnets because no benefit has been listed.
                Nor is the IB specification anywhere near providing
                such an infrastructure.

        LIDs : Get from SA

                It is wrong to link LID lookup to getting the
                information from the SA because other information is
                kept at the SA. This is because the other information
                can be got in implementation dependent way. The two
                are disjoint.


        How about agreeing to disagree and do the following in
        address-resolution packets:


        GID + QPN + Cap flags + LID


        The order is as specified above. Those that don't want to use
        the LID can ignore it in received packets and not record it in
        their information. All they are asked to do is record their
        own LID when sending the address-resolution request/reply.
        They ignore the LID in the request/reply they receive. Note
        that this format has nothing to do with the format in which
        the link address is recorded in the system.

        That should work for all implementations.

Vivek

--
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com
vivk@us.ibm.com
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 15 22:09:23 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA19625
	for <ipoverib-archive@odin.ietf.org>; Tue, 15 Jan 2002 22:09:23 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id WAA04364
	for ipoverib-archive@odin.ietf.org; Tue, 15 Jan 2002 22:09:26 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA04325;
	Tue, 15 Jan 2002 22:08:49 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA03552
	for <ipoverib@optimus.ietf.org>; Tue, 15 Jan 2002 21:55:07 -0500 (EST)
Received: from mail3.microsoft.com (mail3.microsoft.com [131.107.3.123])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA19536
	for <ipoverib@ietf.org>; Tue, 15 Jan 2002 21:55:04 -0500 (EST)
Received: from INET-VRS-03.redmond.corp.microsoft.com ([157.54.5.27]) by mail3.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Tue, 15 Jan 2002 18:54:35 -0800
Received: from 157.54.8.109 by INET-VRS-03.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Tue, 15 Jan 2002 18:54:35 -0800
Received: from red-imc-04.redmond.corp.microsoft.com ([157.54.2.168]) by inet-hub-02.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Tue, 15 Jan 2002 18:54:35 -0800
Received: from win-imc-02.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.84]) by red-imc-04.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Tue, 15 Jan 2002 18:54:35 -0800
Received: from win-msg-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.52]) by win-imc-02.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3588.0);
	 Tue, 15 Jan 2002 18:52:19 -0800
X-MimeOLE: Produced By Microsoft Exchange V6.0.6132.0
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Date: Tue, 15 Jan 2002 18:52:29 -0800
Message-ID: <2E33960095B58E40A4D3345AB9F65EC102450C80@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
Thread-Topic: MCGroup Attach Failure --- IPoIB link and multicast draft - 
Thread-Index: AcGdOPnbHZe99TUpRn6L/4IFwVhLHwA4mKnw
From: "Aditya Dube" <adube@windows.microsoft.com>
To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>, <ipoverib@ietf.org>
X-OriginalArrivalTime: 16 Jan 2002 02:52:19.0913 (UTC) FILETIME=[D0A84390:01C19E38]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id VAA03553
Subject: [Ipoverib] MCGroup Attach Failure --- IPoIB link and multicast draft -
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

Hello Jerry

I have a question from the Multicast draft that you posted a few days
ago.

What should a node that wishes to receive IP traffic on a MCast GID do
if the 'Attach' (SubnAdmSet MCMemberRecord) operation on that IB MCast
GID fails?


Aditya Dube

-----Original Message-----
From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com] 
Sent: Monday, January 14, 2002 12:20 PM
To: ipoverib@ietf.org
Subject: [Ipoverib] IPoIB link and multicast draft


...<deleted>


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 16 01:17:13 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA23226
	for <ipoverib-archive@odin.ietf.org>; Wed, 16 Jan 2002 01:17:08 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id BAA18007
	for ipoverib-archive@odin.ietf.org; Wed, 16 Jan 2002 01:17:09 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id BAA17989;
	Wed, 16 Jan 2002 01:16:31 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id BAA17957
	for <ipoverib@optimus.ietf.org>; Wed, 16 Jan 2002 01:16:30 -0500 (EST)
Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA23221
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 01:16:28 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e31.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id BAA108826;
	Wed, 16 Jan 2002 01:13:28 -0500
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0G6GSr98656;
	Tue, 15 Jan 2002 23:16:28 -0700
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id g0G6GSg09880;
	Tue, 15 Jan 2002 22:16:28 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0G6GQW10846;
	Tue, 15 Jan 2002 22:16:26 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201160616.g0G6GQW10846@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] MCGroup Attach Failure --- IPoIB link and multicast draft -
To: adube@windows.microsoft.com (Aditya Dube)
Date: Tue, 15 Jan 2002 22:16:25 -0800 (PST)
Cc: Jerry.Chu@eng.sun.com (H.K. Jerry Chu), ipoverib@ietf.org
In-Reply-To: <2E33960095B58E40A4D3345AB9F65EC102450C80@win-msg-01.wingroup.windeploy.ntdev.microsoft.com> from "Aditya Dube" at Jan 15, 2002 05:52:29 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Aditya,
        The failure cases at the IB level are covered in the
        architecture draft.

Vivek

> 
> Hello Jerry
> 
> I have a question from the Multicast draft that you posted a few days
> ago.
> 
> What should a node that wishes to receive IP traffic on a MCast GID do
> if the 'Attach' (SubnAdmSet MCMemberRecord) operation on that IB MCast
> GID fails?
> 
> 
> Aditya Dube
> 
> -----Original Message-----
> From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com] 
> Sent: Monday, January 14, 2002 12:20 PM
> To: ipoverib@ietf.org
> Subject: [Ipoverib] IPoIB link and multicast draft
> 
> 
> ...<deleted>
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 16 01:50:26 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA23527
	for <ipoverib-archive@odin.ietf.org>; Wed, 16 Jan 2002 01:50:26 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id BAA18752
	for ipoverib-archive@odin.ietf.org; Wed, 16 Jan 2002 01:50:26 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id BAA18735;
	Wed, 16 Jan 2002 01:49:35 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id BAA18701
	for <ipoverib@optimus.ietf.org>; Wed, 16 Jan 2002 01:49:33 -0500 (EST)
Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA23517
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 01:49:31 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id AAA24502;
	Wed, 16 Jan 2002 00:45:20 -0600
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0G6nVe80740;
	Wed, 16 Jan 2002 01:49:32 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id g0G6nVg10381;
	Tue, 15 Jan 2002 22:49:31 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0G6nUW12854;
	Tue, 15 Jan 2002 22:49:30 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201160649.g0G6nUW12854@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] link descriptor in address resolution packets
To: Jerry.Chu@eng.sun.com
Date: Tue, 15 Jan 2002 22:49:29 -0800 (PST)
Cc: kashyapv@us.ibm.com, ipoverib@ietf.org
In-Reply-To: <200201160110.g0G1AuqT366588@jurassic.eng.sun.com> from "H.K. Jerry Chu" at Jan 15, 2002 04:08:27 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

> 
> >        I feel the argument on including the LIDs vs. GIDs is
> >        revolving round two issues:
> >
> >	GIDs : Why are they needed ?
> >
> >                They cannot be justified by saying that IP subnets can
> >                span IB subnets because no benefit has been listed.
> >                Nor is the IB specification anywhere near providing
> >                such an infrastructure.
> 
> No that's not the main reason. GIDs is highly persistent compared to
> LIDs. There is a great deal of "hidden" benefit from having a persistent
> component in a "MAC address" that have been pointed out many times before
> (e.g. for diagnosis purpose, less dependency on gratuitous ARP getting
> through...)

        I am not sure what you mean by the examples. For persistency
        you need the GUID not GID. You talked about subnet mergers and
        scoped GIDs. Mergers will cause the GIDs to change as much as
        LIDs. 

> >
> >	LIDs : Get from SA
> >
> >                It is wrong to link LID lookup to getting the
> >                information from the SA because other information is
> >                kept at the SA. This is because the other information
> >                can be got in implementation dependent way. The two
> >                are disjoint.
> 
> The linkage is there because you argued to include LID for performance
> reason, but a high percentage of implementations that require a trip to
> SA won't realize any performance on an already infrequent ARP operation.

        No my argument has been that getting the LID in one
        request/response is highly desirable. I further argued that
        just because some may want to go to SA for path rate/SL that
        is no justification to say that LID also be got from the SA.

        Nothing stops those that want to consult the SA from
        consulting it. Similarly don't block the implementations that
        don't want to consult the SA.


> 
> >
> >
> >        How about agreeing to disagree and do the following in
> >        address-resolution packets:
> >
> >
> >	GID + QPN + Cap flags + LID
> >
> >
> >        The order is as specified above. Those that don't want to use
> >        the LID can ignore it in received packets and not record it in
> >        their information. All they are asked to do is record their
> >        own LID when sending the address-resolution request/reply.
> >        They ignore the LID in the request/reply they receive. Note
> >        that this format has nothing to do with the format in which
> >        the link address is recorded in the system.
> >
> >	That should work for all implementations.
> 
> You can't do that without violating layering. ARP modules in most stacks
> are independent of link types. Driver is not supposed to peek into ARP
> packets.

        Don't make your driver peek into anything. You have an ARP
        module that is going to record the link address. This can
        always mark the entry incomplete, in the process or zero out
        the LID part. You have thus ignored the LID. When your trip to
        the SA is done your ARP module will update the entry. No
        violation of layers at all.


Vivek

-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 16 01:56:13 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA23591
	for <ipoverib-archive@odin.ietf.org>; Wed, 16 Jan 2002 01:56:13 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id BAA18953
	for ipoverib-archive@odin.ietf.org; Wed, 16 Jan 2002 01:56:14 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id BAA18885;
	Wed, 16 Jan 2002 01:55:29 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id BAA18855
	for <ipoverib@optimus.ietf.org>; Wed, 16 Jan 2002 01:55:27 -0500 (EST)
Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA23577
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 01:55:25 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e31.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id BAA216768;
	Wed, 16 Jan 2002 01:52:25 -0500
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0G6tPr66470;
	Tue, 15 Jan 2002 23:55:25 -0700
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0G6tMV07350;
	Tue, 15 Jan 2002 22:55:22 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0G6tLf13244;
	Tue, 15 Jan 2002 22:55:21 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201160655.g0G6tLf13244@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] Summary of the IPoIB link layer address debate
To: Paul.Culley@compaq.com (Culley, Paul)
Date: Tue, 15 Jan 2002 22:55:20 -0800 (PST)
Cc: kashyapv@us.ibm.com (Vivek Kashyap), rronald@VIEO.com (Roger Ronald),
        ipoverib@ietf.org
In-Reply-To: <4D027986353D1341ADA4F2A4F8A170760E316F@cceexc18.americas.cpqcorp.net> from "Culley, Paul" at Jan 15, 2002 03:56:11 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

        Sure, the received SL can be used. It is allowed and
        guaranteed by the IBTA spec to work when using UD. The
        response should be going back to the sender though. The LID
        could also be picked. However, for ARP this won't work since
        it doesn't work with proxy ARP. The sender is fronting for
        someone else. 
	
	
	Vivek


> 
> As has been pointed out several times in the past, a unicast ARP
> response packet already includes the LID and an SL, both of which are
> guaranteed (by IBA Spec) to be valid for sending (small) packets back to
> the source of the ARP response packet.
> 
> Since this is already present in the IB Datagram header (LRH:SLID,
> LRH:SL) and can be easily retrieved by HCAs built to the IB
> Specification, I cannot see any point in adding a LID to the unicast
> response.
> 
> Paul R. Culley
> Compaq Sr. Fellow
> 281-514-5543
> 
> 
> -----Original Message-----
> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> Sent: Tuesday, January 15, 2002 4:30 PM
> To: Roger Ronald
> Cc: Vivek Kashyap; ipoverib@ietf.org
> Subject: Re: [Ipoverib] Summary of the IPoIB link layer address debate
> 
> 
> > 
> > It seems like the general debate that's raging is trying to answer 
> > a single question:
> > 
> > "Is it wise to include the LID in order to enable a
> > shortcut that allows bypassing a trip to the SM/SA?"
> > 
> > My read is that second party dissemination of LIDs is not 
> > something that is prohibited (per the IBTA rules), but it is a
> > shortcut that the IBTA architects weren't expecting
> > and are uncomfortable seeing. Various objections
> > about PathRecords, rate control, SL, and MTU have
> > been mentioned, but don't seem to make for a conclusive
> > decision point.
> > 
> > In the end, the tradeoff seems to be simplicity versus
> > flexibility. Leaving out the LID makes the header
> > smaller and the processing is always done in one
> > way. Putting the LID into the header allows
> > skipping a step under (at least) some circumstances.
> 
> Leaving out the GID makes the header a lot less smaller. I am not sure
> what process is done in one way ? If you are talking about spanning IB
> subnets with a single IPoIB subnet the process is definitely not done
> in a single way. It is more easily a single way if the LID is used.
> This is noted in my note below.
> 
> 
> > > applies all over IB components and is not limited to LIDs/GIDs.
> > > 
> > > Let us first answer the question of why IPoIB subnets must span IB
> > > subnets ? We don't know why we need such a setup ? We don't know
> when
> > > such a setup will be possible ? We do know that IPoIB subnets within
> > > an IB subnets can be setup easily with the current specification.
> > > 
> > > 
> > > Secondly, if you step back you'll realise that LID is hardly an
> issue
> > > when resolving across subnets. Unlike what has been postulated the
> > > process is not the same when resolving within the IB subnet and when
> > > resolving across the IB subnet. 
> > > 
> > > Assumption: at IP level- a multicast request; receive a unicast
> reply
> > > 
> > >         Now, when sending the broadcast the sender doesn't know
> where
> > >         the responder is. So he includes his LID/GID. The receiver
> > >         gets it across multiple subnets. It gets sees a different
> > >         GID-prefix and hence replaces the LID with its default
> router
> > >         when recording the address. The same happens on the
> response.
> > > 
> > >         It works fine for both situations - IPoIB subnets
> > >         within/across IB subnets. You can easily have it that if the
> > >         GID is local the routing lookup will return the local LID.
> > >         Single process.
> > > 
> > >         In the case of not including the LID you do the same thing.
> If
> > >         the GID prefix is local you go to the SM/SA to get the LID.
> If
> > >         the GID prefix is different you record your default router's
> > >         LID. So the process for across the subnet is the same but
> you
> > >         shortchange the implementation that is more likely - the one
> > >         whithin the subnet.
> > > 
> > >         For within the subnet you use a different method - going to
> > >         The SA always. Not the same as going to the routing
> > >         infrastructure. 
> > > 
> > > However, I would still like to know what is the use of IPoIB
> spanning
> > > multiple subnets. Why do you think that is a requirement ?
> > > 
> > > 
> > > Vivek
> > > -- 
> > > Vivek Kashyap
> > > Linux Technology Center, IBM
> > > kashyapv@us.ibm.com 
> > > vivk@us.ibm.com 
> > > 503 578 3422 (o)
> > > -- 
> > > Vivek Kashyap
> > > Linux Technology Center, IBM
> > > kashyapv@us.ibm.com 
> > > vivk@us.ibm.com 
> > > 503 578 3422 (o)
> > > 
> > > _______________________________________________
> > > IPoverIB mailing list
> > > IPoverIB@ietf.org
> > > http://www.ietf.org/mailman/listinfo/ipoverib
> > > 
> > > 
> > 
> > 
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> > 
> 
> 
> -- 
> Vivek Kashyap
> Linux Technology Center, IBM
> kashyapv@us.ibm.com 
> vivk@us.ibm.com 
> 503 578 3422 (o)
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 16 02:10:57 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA01972
	for <ipoverib-archive@odin.ietf.org>; Wed, 16 Jan 2002 02:10:57 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id CAA19712
	for ipoverib-archive@odin.ietf.org; Wed, 16 Jan 2002 02:10:58 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id CAA19688;
	Wed, 16 Jan 2002 02:10:25 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id CAA19664
	for <ipoverib@optimus.ietf.org>; Wed, 16 Jan 2002 02:10:24 -0500 (EST)
Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA01966
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 02:10:22 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id BAA82928;
	Wed, 16 Jan 2002 01:06:09 -0600
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0G7AJe64118;
	Wed, 16 Jan 2002 02:10:19 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0G7AHV07772;
	Tue, 15 Jan 2002 23:10:17 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0G7AGq14196;
	Tue, 15 Jan 2002 23:10:16 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201160710.g0G7AGq14196@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
To: krause@cup.hp.com (Michael Krause)
Date: Tue, 15 Jan 2002 23:10:15 -0800 (PST)
Cc: ipoverib@ietf.org
In-Reply-To: <5.1.0.14.2.20020115150819.028a7420@esalpha2.cup.hp.com> from "Michael Krause" at Jan 15, 2002 02:22:04 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit


> 
> Congestion management would be optional functionality given the V1.0 
> specification has been public for over a year now.  As such, it cannot be 
> relied upon to solve any problems only to improve a solution over 
> time.  Further discussion into the details is not permitted due to the IBTA 
> confidentiality rules but it is fair to state that injection rate control 
> is one lever used by congestion management but congestion management does 
> not lead to injection rate control which may be driven more by QoS 
> requirements which go beyond the scope of congestion management.
> 
> The IP over IB spec should define the architectural minimums needed to 
> implement a credible IP over IB solution and leave optimizations such as 
> what have been debated these last few days to vendors to solve.  If the 
> optimizations prove worthwhile after experience has been gained in real 
> customer deployment situations such that a subsequent spec should be 
> generated, then the workgroup can provide a new spec that provides an 
> interoperable solution using these new techniques.

        The only issue I am pointing out is that there are multiple
        ways to solve the problem - as you yourself point out. The
        architectural minimum should not be defined such that it ties
        down every body to the minimum leaving the optimisations out
        of scope in the future.

        Therefore include LID/GID in the address resolution packets
        and all sorts of implementations can coexist. Remove GID and
        only one set is possible, remove LID and only another set is
        possible. Include both, both are possible and there are no
        negatives for either.

> 
> > > And what's the real benefit of saving just a lookup out of an already
> > > infrequent operation like ARP due to cacheing?
> >
> >         I would like to know what is the drawback that you see that
> >         you don't want one to go ahead with it? ARP is repeated every
> >         so often and happens quite often. It also happens on failover
> >         - it is preferable that these operations be simple, fast and
> >         error-free.
> 
> No one is arguing with any of these assertions.   The issue is strictly 
> what is the architectural minimum the spec should define and what should be 
> left for future spec or vendor implementation to solve.
> 
> 
> >         Secondly, things are more complicated the way you propose.
> >         Note that the first level address would be first determined.
> >         This completes ARP/ND. However, address resolution is not
> >         complete. The packet is sent as far as IP is concerned and now
> >         the SA/SM is consulted to get the LID etc. This has very
> >         strong shades of ATM VCs. IP thought it had sent the packet
> >         since the ARP/ND was complete - it took a while before the
> >         lower layer completed the determination. The driver of course
> >         keeps a GID to LID table too since the LID is a separate
> >         entity. More complication.
> 
> This is one implementation choice and while it may be used by some, it does 
> not require the spec to make this a requirement.

        The assertion against LIDs was that they cause complication. I
        am showing that it is the other way down. This description is
        not part of the spec. but shows that the argument that
        inclusion of LIDs causes more complexity is incorrect. It is
        the GID to LID conversion that is more complicated. Including
        LIDs in address resolution allows for a simple, 2 message,
        fast failover solution.

        I have also noted that the process, in another mail, of
        resolving the IP address is simpler and fits in more with a
        'single method' when LIDs are included than does the process
        when the LID is not. I have not seen a counter example.


> 
> 
> >         Include LID in the resolution - fast/simple/error-free.
> 
> It isn't that simple either

        As shown above it is simpler than getting the GID and the
        answer. I request a corresponding scenario that shows that the
        GID to LID conversion is a better solution. I am not against
        anyone implementing it differently. Allow those that want ot
        implenet it in this way to do so.


> 
> >         I was pointing out another issue that many have raised.
> >         In general I am ok with dropping GID since it is for the
> >         future. IT buys us nothing as of now. So if we dropped GID we
> >         can work with these 48 bits. When IPoIB in reality starts
> >         spanning multiple IPoIB subnets (not for a long time) we can
> >         include GID. Then the hardware length will become 22 bytes as
> >         against 6 bytes. One can still interoperate quite easily.
> 
> GID is required because it is the primary way and application determines 
> where a service resides on a given port as such not including it makes life 
> quite difficult to implement.  GIDs are also used by applications within 
> the address vector to identify a given vector to use for a given service 
> such as IP.  GIDs cannot be dropped and have value.
> 
> LIDs are strictly a local issue that will be learned for all CA / router 
> ports participating within a given partition that support IP services.  As 
> such, their assignment, management, transitory nature, etc. make them 
> unsuited as an architectural minimum to include within the spec.  I see no 
> problem with people ascertaining the value and using it for optimizing 
> their implementation but this is outside the scope of the specification.

        I don't understand. If those that don't want this information
        just ignore it what is the loss? We can format the link
        description such that the LID is the last element. Those that
        don't want it can drop this piece of information and ignore
        it. Of course those that don't want to use GID can ignore it
        too. 

> 
> >         Wouldn't that work for all ?
> >
> >         I included the GID because many want to put in the hooks for
> >         the future case of IPoIB spanning IB subnets. I would though
> >         like a valid reason specified for why do we need IP subnets to
> >         span IB subnets. Isn't IP subnetting enough?
> 
> IB is viewed as a layer 2 fabric w.r.t. IP so the fact that IB has multiple 
> subnets is independent of IP operation.  I see no point in dealing with the 
> IB subnet issues within this specification other than to state that a LID 
> is not an end-to-end address between two endnode ports and thus should not 
> be mandated within the IP over IB spec for address resolution.
> 
> Mike
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)
-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 16 08:52:24 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA07066
	for <ipoverib-archive@odin.ietf.org>; Wed, 16 Jan 2002 08:52:24 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id IAA01295
	for ipoverib-archive@odin.ietf.org; Wed, 16 Jan 2002 08:52:28 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id IAA01220;
	Wed, 16 Jan 2002 08:49:58 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id IAA01172
	for <ipoverib@optimus.ietf.org>; Wed, 16 Jan 2002 08:49:55 -0500 (EST)
Received: from ztxmail03.ztx.compaq.com (ztxmail03.ztx.compaq.com [161.114.1.207])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA07009
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 08:49:51 -0500 (EST)
Received: by ztxmail03.ztx.compaq.com (Postfix, from userid 12345)
	id 95B5D2F0E; Wed, 16 Jan 2002 07:49:23 -0600 (CST)
Received: from cceexg12.americas.cpqcorp.net (cceexg12.americas.cpqcorp.net [16.110.250.124])
	by ztxmail03.ztx.compaq.com (Postfix) with ESMTP
	id 8C6702E5D; Wed, 16 Jan 2002 07:49:23 -0600 (CST)
Received: from cceexc18.americas.cpqcorp.net ([16.110.250.64]) by cceexg12.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966);
	 Wed, 16 Jan 2002 07:49:23 -0600
X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="US-ASCII"
Subject: RE: [Ipoverib] Summary of the IPoIB link layer address debate (using SLID/SL)
Date: Wed, 16 Jan 2002 07:49:22 -0600
Message-ID: <4D027986353D1341ADA4F2A4F8A170760E3170@cceexc18.americas.cpqcorp.net>
Thread-Topic: [Ipoverib] Summary of the IPoIB link layer address debate
Thread-Index: AcGeKYVMkvSoRfQSRo2GGVC2bM2Y+AAakkRg
From: "Culley, Paul" <Paul.Culley@compaq.com>
To: "Joe Pelissier" <joep@Sanera.net>, <ipoverib@ietf.org>
X-OriginalArrivalTime: 16 Jan 2002 13:49:23.0339 (UTC) FILETIME=[9AD9ADB0:01C19E94]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id IAA01173
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

When we get out the next version of the IB spec, I think the language
will state that using a multicast SL for unicast need not be supported
by all SMs.  As such, the ARP responder must do a path lookup to make
the unicast response (or somehow know what the SM is up to).  Given
that, the requestor could avoid its own path lookup by using the
SLID/SL.

Paul R. Culley
Compaq Sr. Fellow
281-514-5543


-----Original Message-----
From: Joe Pelissier [mailto:joep@Sanera.net]
Sent: Tuesday, January 15, 2002 7:02 PM
To: ipoverib@ietf.org
Subject: Re: [Ipoverib] Summary of the IPoIB link layer address debate


Hi Paul,

Here's an interesting question...
For sending the multicast ARP request, I know which SL and MLID to use
since
I am a member of the multicast group.

When I respond to the ARP request, I'm sending unicast.  I know the DLID
to
use, but how do I know what SL to use?  While the IB spec does not
prohibit
me from using the SL in the ARP request packet, it also does not
guarantee
that the SL in the ARP multicast request is valid for the ARP unicast
response.  If the SM is using VL to for deadlock avoidance (a perfectly
reasonable usage), then the ARP response could be lost.

As you stated, once the requester receives the response, it does have a
valid SL to use.  But I think the responder is in a bit of a quandary.

Thoughts?
Joe Pelissier

"Culley, Paul" wrote:

> As has been pointed out several times in the past, a unicast ARP
> response packet already includes the LID and an SL, both of which are
> guaranteed (by IBA Spec) to be valid for sending (small) packets back
to
> the source of the ARP response packet.
>
> Since this is already present in the IB Datagram header (LRH:SLID,
> LRH:SL) and can be easily retrieved by HCAs built to the IB
> Specification, I cannot see any point in adding a LID to the unicast
> response.
>
> Paul R. Culley
> Compaq Sr. Fellow
> 281-514-5543
>
> -----Original Message-----
> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> Sent: Tuesday, January 15, 2002 4:30 PM
> To: Roger Ronald
> Cc: Vivek Kashyap; ipoverib@ietf.org
> Subject: Re: [Ipoverib] Summary of the IPoIB link layer address debate
>
> >
> > It seems like the general debate that's raging is trying to answer
> > a single question:
> >
> > "Is it wise to include the LID in order to enable a
> > shortcut that allows bypassing a trip to the SM/SA?"
> >
> > My read is that second party dissemination of LIDs is not
> > something that is prohibited (per the IBTA rules), but it is a
> > shortcut that the IBTA architects weren't expecting
> > and are uncomfortable seeing. Various objections
> > about PathRecords, rate control, SL, and MTU have
> > been mentioned, but don't seem to make for a conclusive
> > decision point.
> >
> > In the end, the tradeoff seems to be simplicity versus
> > flexibility. Leaving out the LID makes the header
> > smaller and the processing is always done in one
> > way. Putting the LID into the header allows
> > skipping a step under (at least) some circumstances.
>
> Leaving out the GID makes the header a lot less smaller. I am not sure
> what process is done in one way ? If you are talking about spanning IB
> subnets with a single IPoIB subnet the process is definitely not done
> in a single way. It is more easily a single way if the LID is used.
> This is noted in my note below.
>
> > > applies all over IB components and is not limited to LIDs/GIDs.
> > >
> > > Let us first answer the question of why IPoIB subnets must span IB
> > > subnets ? We don't know why we need such a setup ? We don't know
> when
> > > such a setup will be possible ? We do know that IPoIB subnets
within
> > > an IB subnets can be setup easily with the current specification.
> > >
> > >
> > > Secondly, if you step back you'll realise that LID is hardly an
> issue
> > > when resolving across subnets. Unlike what has been postulated the
> > > process is not the same when resolving within the IB subnet and
when
> > > resolving across the IB subnet.
> > >
> > > Assumption: at IP level- a multicast request; receive a unicast
> reply
> > >
> > >         Now, when sending the broadcast the sender doesn't know
> where
> > >         the responder is. So he includes his LID/GID. The receiver
> > >         gets it across multiple subnets. It gets sees a different
> > >         GID-prefix and hence replaces the LID with its default
> router
> > >         when recording the address. The same happens on the
> response.
> > >
> > >         It works fine for both situations - IPoIB subnets
> > >         within/across IB subnets. You can easily have it that if
the
> > >         GID is local the routing lookup will return the local LID.
> > >         Single process.
> > >
> > >         In the case of not including the LID you do the same
thing.
> If
> > >         the GID prefix is local you go to the SM/SA to get the
LID.
> If
> > >         the GID prefix is different you record your default
router's
> > >         LID. So the process for across the subnet is the same but
> you
> > >         shortchange the implementation that is more likely - the
one
> > >         whithin the subnet.
> > >
> > >         For within the subnet you use a different method - going
to
> > >         The SA always. Not the same as going to the routing
> > >         infrastructure.
> > >
> > > However, I would still like to know what is the use of IPoIB
> spanning
> > > multiple subnets. Why do you think that is a requirement ?
> > >
> > >
> > > Vivek
> > > --
> > > Vivek Kashyap
> > > Linux Technology Center, IBM
> > > kashyapv@us.ibm.com
> > > vivk@us.ibm.com
> > > 503 578 3422 (o)
> > > --
> > > Vivek Kashyap
> > > Linux Technology Center, IBM
> > > kashyapv@us.ibm.com
> > > vivk@us.ibm.com
> > > 503 578 3422 (o)
> > >
> > > _______________________________________________
> > > IPoverIB mailing list
> > > IPoverIB@ietf.org
> > > http://www.ietf.org/mailman/listinfo/ipoverib
> > >
> > >
> >
> >
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> >
>
> --
> Vivek Kashyap
> Linux Technology Center, IBM
> kashyapv@us.ibm.com
> vivk@us.ibm.com
> 503 578 3422 (o)
>
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
>
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 16 09:16:03 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA08069
	for <ipoverib-archive@odin.ietf.org>; Wed, 16 Jan 2002 09:16:03 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id JAA02233
	for ipoverib-archive@odin.ietf.org; Wed, 16 Jan 2002 09:16:07 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id JAA02158;
	Wed, 16 Jan 2002 09:13:43 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id JAA02124
	for <ipoverib@optimus.ietf.org>; Wed, 16 Jan 2002 09:13:41 -0500 (EST)
Received: from ztxmail03.ztx.compaq.com (ztxmail03.ztx.compaq.com [161.114.1.207])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA07911
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 09:13:36 -0500 (EST)
Received: by ztxmail03.ztx.compaq.com (Postfix, from userid 12345)
	id CAC2A2EE6; Wed, 16 Jan 2002 08:13:01 -0600 (CST)
Received: from cceexg12.americas.cpqcorp.net (cceexg12.americas.cpqcorp.net [16.110.250.124])
	by ztxmail03.ztx.compaq.com (Postfix) with ESMTP
	id 8F9512C8F; Wed, 16 Jan 2002 08:13:01 -0600 (CST)
Received: from cceexc18.americas.cpqcorp.net ([16.110.250.64]) by cceexg12.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966);
	 Wed, 16 Jan 2002 08:12:55 -0600
X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="US-ASCII"
Subject: RE: [Ipoverib] Summary of the IPoIB link layer address debate (proxy Arp)
Date: Wed, 16 Jan 2002 08:12:54 -0600
Message-ID: <4D027986353D1341ADA4F2A4F8A170760E3171@cceexc18.americas.cpqcorp.net>
Thread-Topic: [Ipoverib] Summary of the IPoIB link layer address debate
Thread-Index: AcGeWsvTrdaTwc8xQJCLKVNP6/Iv+gAOdgJg
From: "Culley, Paul" <Paul.Culley@compaq.com>
To: "Vivek Kashyap" <kashyapv@us.ibm.com>
Cc: "Roger Ronald" <rronald@VIEO.com>, <ipoverib@ietf.org>
X-OriginalArrivalTime: 16 Jan 2002 14:12:55.0073 (UTC) FILETIME=[E44F3110:01C19E97]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id JAA02125
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

Forgive me for perhaps not understanding how proxy ARP works; but I was
under the impression that a proxy ARP device was two way.  That is, if
some device was "fronting" for the actual device, then it would provide
the ARP service, including ARP replies, and ALSO, pass the data comm
packets through that interface to another network.

If such a device were to be used for IB, it seems to me that it would
behave properly.  When replying to ARP, the unicast ARP response's
SLID/SL would point back to the proxy device, which is also responsible
for passing the data packet to some other interface.

On another subject, Use of any sort of LID in the ARP response; I have
these comments:

1) Unless the ARP Requestor has some (IB approved) way to find an
appropriate SL, it must still do the PathRecord lookup.  So providing
only LID seems inadequate.

2) While IB does not yet have routers finished, it will someday.  Unless
you prohibit multiple IB subnets per IP subnet, when that day arrives,
the ARP Responder will have NO IDEA on what an appropriate LID is to
place in the body of the ARP Response.  If the idea of using SLID/SL is
accepted, then the router will be modifying that field as the ARP
response passes through, so that it will point back to the router, and
thus be suitable for communication.

3) If you decide to include the LID in the ARP response body, please
point out to implementers when it may not be valid; for instance, it
should be considered invalid whenever the GID prefix in the ARP response
does not match the GID prefix of the ARP requestor.

Paul R. Culley
Compaq Sr. Fellow
281-514-5543


-----Original Message-----
From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
Sent: Wednesday, January 16, 2002 12:55 AM
To: Culley, Paul
Cc: Vivek Kashyap; Roger Ronald; ipoverib@ietf.org
Subject: Re: [Ipoverib] Summary of the IPoIB link layer address debate


        Sure, the received SL can be used. It is allowed and
        guaranteed by the IBTA spec to work when using UD. The
        response should be going back to the sender though. The LID
        could also be picked. However, for ARP this won't work since
        it doesn't work with proxy ARP. The sender is fronting for
        someone else. 
	
	
	Vivek


> 
> As has been pointed out several times in the past, a unicast ARP
> response packet already includes the LID and an SL, both of which are
> guaranteed (by IBA Spec) to be valid for sending (small) packets back
to
> the source of the ARP response packet.
> 
> Since this is already present in the IB Datagram header (LRH:SLID,
> LRH:SL) and can be easily retrieved by HCAs built to the IB
> Specification, I cannot see any point in adding a LID to the unicast
> response.
> 
> Paul R. Culley
> Compaq Sr. Fellow
> 281-514-5543
> 
> 
> -----Original Message-----
> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> Sent: Tuesday, January 15, 2002 4:30 PM
> To: Roger Ronald
> Cc: Vivek Kashyap; ipoverib@ietf.org
> Subject: Re: [Ipoverib] Summary of the IPoIB link layer address debate
> 
> 
> > 
> > It seems like the general debate that's raging is trying to answer 
> > a single question:
> > 
> > "Is it wise to include the LID in order to enable a
> > shortcut that allows bypassing a trip to the SM/SA?"
> > 
> > My read is that second party dissemination of LIDs is not 
> > something that is prohibited (per the IBTA rules), but it is a
> > shortcut that the IBTA architects weren't expecting
> > and are uncomfortable seeing. Various objections
> > about PathRecords, rate control, SL, and MTU have
> > been mentioned, but don't seem to make for a conclusive
> > decision point.
> > 
> > In the end, the tradeoff seems to be simplicity versus
> > flexibility. Leaving out the LID makes the header
> > smaller and the processing is always done in one
> > way. Putting the LID into the header allows
> > skipping a step under (at least) some circumstances.
> 
> Leaving out the GID makes the header a lot less smaller. I am not sure
> what process is done in one way ? If you are talking about spanning IB
> subnets with a single IPoIB subnet the process is definitely not done
> in a single way. It is more easily a single way if the LID is used.
> This is noted in my note below.
> 
> 
> > > applies all over IB components and is not limited to LIDs/GIDs.
> > > 
> > > Let us first answer the question of why IPoIB subnets must span IB
> > > subnets ? We don't know why we need such a setup ? We don't know
> when
> > > such a setup will be possible ? We do know that IPoIB subnets
within
> > > an IB subnets can be setup easily with the current specification.
> > > 
> > > 
> > > Secondly, if you step back you'll realise that LID is hardly an
> issue
> > > when resolving across subnets. Unlike what has been postulated the
> > > process is not the same when resolving within the IB subnet and
when
> > > resolving across the IB subnet. 
> > > 
> > > Assumption: at IP level- a multicast request; receive a unicast
> reply
> > > 
> > >         Now, when sending the broadcast the sender doesn't know
> where
> > >         the responder is. So he includes his LID/GID. The receiver
> > >         gets it across multiple subnets. It gets sees a different
> > >         GID-prefix and hence replaces the LID with its default
> router
> > >         when recording the address. The same happens on the
> response.
> > > 
> > >         It works fine for both situations - IPoIB subnets
> > >         within/across IB subnets. You can easily have it that if
the
> > >         GID is local the routing lookup will return the local LID.
> > >         Single process.
> > > 
> > >         In the case of not including the LID you do the same
thing.
> If
> > >         the GID prefix is local you go to the SM/SA to get the
LID.
> If
> > >         the GID prefix is different you record your default
router's
> > >         LID. So the process for across the subnet is the same but
> you
> > >         shortchange the implementation that is more likely - the
one
> > >         whithin the subnet.
> > > 
> > >         For within the subnet you use a different method - going
to
> > >         The SA always. Not the same as going to the routing
> > >         infrastructure. 
> > > 
> > > However, I would still like to know what is the use of IPoIB
> spanning
> > > multiple subnets. Why do you think that is a requirement ?
> > > 
> > > 
> > > Vivek
> > > -- 
> > > Vivek Kashyap
> > > Linux Technology Center, IBM
> > > kashyapv@us.ibm.com 
> > > vivk@us.ibm.com 
> > > 503 578 3422 (o)
> > > -- 
> > > Vivek Kashyap
> > > Linux Technology Center, IBM
> > > kashyapv@us.ibm.com 
> > > vivk@us.ibm.com 
> > > 503 578 3422 (o)
> > > 
> > > _______________________________________________
> > > IPoverIB mailing list
> > > IPoverIB@ietf.org
> > > http://www.ietf.org/mailman/listinfo/ipoverib
> > > 
> > > 
> > 
> > 
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> > 
> 
> 
> -- 
> Vivek Kashyap
> Linux Technology Center, IBM
> kashyapv@us.ibm.com 
> vivk@us.ibm.com 
> 503 578 3422 (o)
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 16 09:42:59 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA09361
	for <ipoverib-archive@odin.ietf.org>; Wed, 16 Jan 2002 09:42:54 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id JAA03268
	for ipoverib-archive@odin.ietf.org; Wed, 16 Jan 2002 09:42:58 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id JAA03230;
	Wed, 16 Jan 2002 09:42:15 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id JAA03202
	for <ipoverib@optimus.ietf.org>; Wed, 16 Jan 2002 09:42:12 -0500 (EST)
Received: from vieo.com (vieo.com [216.30.79.131])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA09314
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 09:42:07 -0500 (EST)
Received: (from root@localhost)
	by vieo.com (8.11.2/8.11.2) id g0GEfdh53958
	for ipoverib@ietf.org; Wed, 16 Jan 2002 08:41:39 -0600 (CST)
	(envelope-from rronald@VIEO.com)
Received: from Fishhead (dhcp-10-2-15-151.vieo.com [10.2.15.151])
	by vieo.com (8.11.2/8.11.2) with SMTP id g0GEfcc53874;
	Wed, 16 Jan 2002 08:41:38 -0600 (CST)
	(envelope-from rronald@VIEO.com)
Message-ID: <00ce01c19e9b$df1b1ca0$970f020a@Fishhead>
From: "Roger Ronald" <rronald@VIEO.com>
To: "Culley, Paul" <Paul.Culley@compaq.com>
Cc: <ipoverib@ietf.org>
References: <4D027986353D1341ADA4F2A4F8A170760E3171@cceexc18.americas.cpqcorp.net>
Subject: Re: [Ipoverib] Summary of the IPoIB link layer address debate (proxy Arp)
Date: Wed, 16 Jan 2002 08:41:23 -0600
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 5.50.4133.2400
X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400
X-scanner: scanned by Inflex 0.1.5c+ on vieo.com
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Your question/point #3 is actually answered by your introduction :-)

The local LID of a bridging device may be a valid local address 
to use for a distant IB subnet (or bridged Ethernet). So
the shortcut of using the LID from an ARP reply 
is not necessarily invalid even if the destination is not local.

Perhaps the ARP reply should contain a field for LID that is zero'ed
if the source knows that a LID will not work, but is filled in if the source
knows that using the LID will work. With this behavior, either the source
or the destination can disallow the shortcut and the shortcut will
only be possible if BOTH sides agree it will work.

RR
----- Original Message ----- 
From: "Culley, Paul" <Paul.Culley@compaq.com>
Sent: Wednesday, January 16, 2002 8:12 AM
Subject: RE: [Ipoverib] Summary of the IPoIB link layer address debate (proxy Arp)


<snip>
> if some device was "fronting" for the actual device, then it would provide
> the ARP service, including ARP replies, and ALSO, pass the data comm
> packets through that interface to another network.
> 
> If such a device were to be used for IB, it seems to me that it would
> behave properly.  When replying to ARP, the unicast ARP response's
> SLID/SL would point back to the proxy device, which is also responsible
> for passing the data packet to some other interface.
> 
<snip> 

>3) If you decide to include the LID in the ARP response body, please
> point out to implementers when it may not be valid; for instance, it
> should be considered invalid whenever the GID prefix in the ARP response
> does not match the GID prefix of the ARP requestor.
> 
> Paul R. Culley
> Compaq Sr. Fellow
> 281-514-5543
<snip>


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 16 12:12:11 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA14330
	for <ipoverib-archive@odin.ietf.org>; Wed, 16 Jan 2002 12:12:11 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id MAA10521
	for ipoverib-archive@odin.ietf.org; Wed, 16 Jan 2002 12:12:14 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA10491;
	Wed, 16 Jan 2002 12:10:55 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id JAA03730
	for <ipoverib@optimus.ietf.org>; Wed, 16 Jan 2002 09:57:48 -0500 (EST)
Received: from omegaband.com (faust.seagullsemi.com [206.196.68.34])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA09901
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 09:57:43 -0500 (EST)
Date: Wed, 16 Jan 2002 08:50:54 -0600
Message-Id: <200201160850.AA268959840@omegaband.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
From: "Ashwin Kamath" <kamath@seagullsemi.com>
Reply-To: <kamath@seagullsemi.com>
X-Sender: <kamath@seagullsemi.com>
To: <ipoverib@ietf.org>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
X-Mailer: <IMail v7.04>
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Vivek, 
 This is a pure response to the statements below, not to be regarded as an opinion on inclusion/exclusion of the LID in ARPs.

 a. You asked for a counter example for simplicity of the including the LID. There is no example provided to counter on a simple way to use SLs and pathrates. The one example given is that there can be static tables, but the simplicity on administering these tables depends on the complexity of the IB Fabric topology. Why is an SA lookup more difficult? It would be if there were no SA and in many cases that could be a valid reason today.
 b. Another reason is to save the round trip for SA lookup. A round trip needs to be saved if the latency is significant to override the benefits. Current implentations have the following latencies for any given ARP and its response
  * 4 PCI burst transactions ( + some normal transactions )
  * 2 trips on Ethernet links + switches
 ARP with SA lookup will have (assuming HCA proximity with Memory and CPU, DMAing data will be atleast an order of magnitude smaller than PCI transactions and thus negligible)
  * 4 trips on Infiniband links + switches
 One can factor in the bus rates + switch latencies, and see that latency of an extra Infiniband round trip is not an issue. If there is a problem with the assumption of proximity, one will have to add 8 PCI burst transactions to the above math. I might be wrong about this but ARP latencies were never an issue even on 100Mbps Ethernet links and you will still come up with a number that is better than atleast 100Mbps Ethernet links with the above math.
 c. Data load on SAs. This has already been discussed. Data load on ethernet controllers today can add significant latencies in the ARP round trip, because of backing up of hardware queues at the end nodes. This fact was never used to promote or demote usage of static tables. 

 It is quite okay to have an architecture which can potentially use static tables, but in this case the justification for this choice may not be Simplicity, latency or data load because these parameters are relative to an implementation. Flexibility is the only justification albiet a very important one.

-Ashwin.


 


---------- Original Message ----------------------------------
From: Vivek Kashyap <kashyapv@us.ibm.com>
Date:  Tue, 15 Jan 2002 23:10:15 -0800 (PST)

>
>> 
>> Congestion management would be optional functionality given the V1.0 
>> specification has been public for over a year now.  As such, it cannot be 
>> relied upon to solve any problems only to improve a solution over 
>> time.  Further discussion into the details is not permitted due to the IBTA 
>> confidentiality rules but it is fair to state that injection rate control 
>> is one lever used by congestion management but congestion management does 
>> not lead to injection rate control which may be driven more by QoS 
>> requirements which go beyond the scope of congestion management.
>> 
>> The IP over IB spec should define the architectural minimums needed to 
>> implement a credible IP over IB solution and leave optimizations such as 
>> what have been debated these last few days to vendors to solve.  If the 
>> optimizations prove worthwhile after experience has been gained in real 
>> customer deployment situations such that a subsequent spec should be 
>> generated, then the workgroup can provide a new spec that provides an 
>> interoperable solution using these new techniques.
>
>        The only issue I am pointing out is that there are multiple
>        ways to solve the problem - as you yourself point out. The
>        architectural minimum should not be defined such that it ties
>        down every body to the minimum leaving the optimisations out
>        of scope in the future.
>
>        Therefore include LID/GID in the address resolution packets
>        and all sorts of implementations can coexist. Remove GID and
>        only one set is possible, remove LID and only another set is
>        possible. Include both, both are possible and there are no
>        negatives for either.
>
>> 
>> > > And what's the real benefit of saving just a lookup out of an already
>> > > infrequent operation like ARP due to cacheing?
>> >
>> >         I would like to know what is the drawback that you see that
>> >         you don't want one to go ahead with it? ARP is repeated every
>> >         so often and happens quite often. It also happens on failover
>> >         - it is preferable that these operations be simple, fast and
>> >         error-free.
>> 
>> No one is arguing with any of these assertions.   The issue is strictly 
>> what is the architectural minimum the spec should define and what should be 
>> left for future spec or vendor implementation to solve.
>> 
>> 
>> >         Secondly, things are more complicated the way you propose.
>> >         Note that the first level address would be first determined.
>> >         This completes ARP/ND. However, address resolution is not
>> >         complete. The packet is sent as far as IP is concerned and now
>> >         the SA/SM is consulted to get the LID etc. This has very
>> >         strong shades of ATM VCs. IP thought it had sent the packet
>> >         since the ARP/ND was complete - it took a while before the
>> >         lower layer completed the determination. The driver of course
>> >         keeps a GID to LID table too since the LID is a separate
>> >         entity. More complication.
>> 
>> This is one implementation choice and while it may be used by some, it does 
>> not require the spec to make this a requirement.
>
>        The assertion against LIDs was that they cause complication. I
>        am showing that it is the other way down. This description is
>        not part of the spec. but shows that the argument that
>        inclusion of LIDs causes more complexity is incorrect. It is
>        the GID to LID conversion that is more complicated. Including
>        LIDs in address resolution allows for a simple, 2 message,
>        fast failover solution.
>
>        I have also noted that the process, in another mail, of
>        resolving the IP address is simpler and fits in more with a
>        'single method' when LIDs are included than does the process
>        when the LID is not. I have not seen a counter example.
>
>
>> 
>> 
>> >         Include LID in the resolution - fast/simple/error-free.
>> 
>> It isn't that simple either
>
>        As shown above it is simpler than getting the GID and the
>        answer. I request a corresponding scenario that shows that the
>        GID to LID conversion is a better solution. I am not against
>        anyone implementing it differently. Allow those that want ot
>        implenet it in this way to do so.
>
>
>> 
>> >         I was pointing out another issue that many have raised.
>> >         In general I am ok with dropping GID since it is for the
>> >         future. IT buys us nothing as of now. So if we dropped GID we
>> >         can work with these 48 bits. When IPoIB in reality starts
>> >         spanning multiple IPoIB subnets (not for a long time) we can
>> >         include GID. Then the hardware length will become 22 bytes as
>> >         against 6 bytes. One can still interoperate quite easily.
>> 
>> GID is required because it is the primary way and application determines 
>> where a service resides on a given port as such not including it makes life 
>> quite difficult to implement.  GIDs are also used by applications within 
>> the address vector to identify a given vector to use for a given service 
>> such as IP.  GIDs cannot be dropped and have value.
>> 
>> LIDs are strictly a local issue that will be learned for all CA / router 
>> ports participating within a given partition that support IP services.  As 
>> such, their assignment, management, transitory nature, etc. make them 
>> unsuited as an architectural minimum to include within the spec.  I see no 
>> problem with people ascertaining the value and using it for optimizing 
>> their implementation but this is outside the scope of the specification.
>
>        I don't understand. If those that don't want this information
>        just ignore it what is the loss? We can format the link
>        description such that the LID is the last element. Those that
>        don't want it can drop this piece of information and ignore
>        it. Of course those that don't want to use GID can ignore it
>        too. 
>
>> 
>> >         Wouldn't that work for all ?
>> >
>> >         I included the GID because many want to put in the hooks for
>> >         the future case of IPoIB spanning IB subnets. I would though
>> >         like a valid reason specified for why do we need IP subnets to
>> >         span IB subnets. Isn't IP subnetting enough?
>> 
>> IB is viewed as a layer 2 fabric w.r.t. IP so the fact that IB has multiple 
>> subnets is independent of IP operation.  I see no point in dealing with the 
>> IB subnet issues within this specification other than to state that a LID 
>> is not an end-to-end address between two endnode ports and thus should not 
>> be mandated within the IP over IB spec for address resolution.
>> 
>> Mike
>> 
>> 
>> _______________________________________________
>> IPoverIB mailing list
>> IPoverIB@ietf.org
>> http://www.ietf.org/mailman/listinfo/ipoverib
>> 
>
>
>-- 
>Vivek Kashyap
>Linux Technology Center, IBM
>kashyapv@us.ibm.com 
>vivk@us.ibm.com 
>503 578 3422 (o)
>-- 
>Vivek Kashyap
>Linux Technology Center, IBM
>kashyapv@us.ibm.com 
>vivk@us.ibm.com 
>503 578 3422 (o)
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>http://www.ietf.org/mailman/listinfo/ipoverib
>


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 16 13:33:21 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA17059
	for <ipoverib-archive@odin.ietf.org>; Wed, 16 Jan 2002 13:33:16 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id NAA14311
	for ipoverib-archive@odin.ietf.org; Wed, 16 Jan 2002 13:33:18 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA13753;
	Wed, 16 Jan 2002 13:28:50 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA13721
	for <ipoverib@optimus.ietf.org>; Wed, 16 Jan 2002 13:28:48 -0500 (EST)
Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA16779
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 13:28:44 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id MAA34352;
	Wed, 16 Jan 2002 12:24:33 -0600
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0GISfs226382;
	Wed, 16 Jan 2002 13:28:45 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0GISeV26671;
	Wed, 16 Jan 2002 10:28:40 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0GISdE20152;
	Wed, 16 Jan 2002 10:28:39 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201161828.g0GISdE20152@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
To: kamath@seagullsemi.com
Date: Wed, 16 Jan 2002 10:28:38 -0800 (PST)
Cc: ipoverib@ietf.org
In-Reply-To: <200201160850.AA268959840@omegaband.com> from "Ashwin Kamath" at Jan 16, 2002 07:50:54 AM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Ashwin,
Ashwin,

        I agree that you get more flexibility if you include the LID.
        If we include the GID along with the LID we allow all
        implementations the choice to create different solutions. We
        don't tie them down to one thing. Choosing one or the other
        (LID or GID) blocks out various advantages that different
        implementations seek. Hence I suggested that let us include
        both and move on.

Vivek

> 
> Vivek, 
>  This is a pure response to the statements below, not to be regarded as an opinion on inclusion/exclusion of the LID in ARPs.
> 
>  a. You asked for a counter example for simplicity of the including the LID. There is no example provided to counter on a simple way to use SLs and pathrates. The one example given is that there can be static tables, but the simplicity on administering these tables depends on the complexity of the IB Fabric topology. Why is an SA lookup more difficult? It would be if there were no SA and in many cases that could be a valid reason today.
>  b. Another reason is to save the round trip for SA lookup. A round trip needs to be saved if the latency is significant to override the benefits. Current implentations have the following latencies for any given ARP and its response
>   * 4 PCI burst transactions ( + some normal transactions )
>   * 2 trips on Ethernet links + switches
>  ARP with SA lookup will have (assuming HCA proximity with Memory and CPU, DMAing data will be atleast an order of magnitude smaller than PCI transactions and thus negligible)
>   * 4 trips on Infiniband links + switches
>  One can factor in the bus rates + switch latencies, and see that latency of an extra Infiniband round trip is not an issue. If there is a problem with the assumption of proximity, one will have to add 8 PCI burst transactions to the above math. I might be wrong about this but ARP latencies were never an issue even on 100Mbps Ethernet links and you will still come up with a number that is better than atleast 100Mbps Ethernet links with the above math.
>  c. Data load on SAs. This has already been discussed. Data load on ethernet controllers today can add significant latencies in the ARP round trip, because of backing up of hardware queues at the end nodes. This fact was never used to promote or demote usage of static tables. 
> 
>  It is quite okay to have an architecture which can potentially use static tables, but in this case the justification for this choice may not be Simplicity, latency or data load because these parameters are relative to an implementation. Flexibility is the only justification albiet a very important one.
> 
> -Ashwin.
> 
> 
>  
> 
> 
> ---------- Original Message ----------------------------------
> From: Vivek Kashyap <kashyapv@us.ibm.com>
> Date:  Tue, 15 Jan 2002 23:10:15 -0800 (PST)
> 
> >
> >> 
> >> Congestion management would be optional functionality given the V1.0 
> >> specification has been public for over a year now.  As such, it cannot be 
> >> relied upon to solve any problems only to improve a solution over 
> >> time.  Further discussion into the details is not permitted due to the IBTA 
> >> confidentiality rules but it is fair to state that injection rate control 
> >> is one lever used by congestion management but congestion management does 
> >> not lead to injection rate control which may be driven more by QoS 
> >> requirements which go beyond the scope of congestion management.
> >> 
> >> The IP over IB spec should define the architectural minimums needed to 
> >> implement a credible IP over IB solution and leave optimizations such as 
> >> what have been debated these last few days to vendors to solve.  If the 
> >> optimizations prove worthwhile after experience has been gained in real 
> >> customer deployment situations such that a subsequent spec should be 
> >> generated, then the workgroup can provide a new spec that provides an 
> >> interoperable solution using these new techniques.
> >
> >        The only issue I am pointing out is that there are multiple
> >        ways to solve the problem - as you yourself point out. The
> >        architectural minimum should not be defined such that it ties
> >        down every body to the minimum leaving the optimisations out
> >        of scope in the future.
> >
> >        Therefore include LID/GID in the address resolution packets
> >        and all sorts of implementations can coexist. Remove GID and
> >        only one set is possible, remove LID and only another set is
> >        possible. Include both, both are possible and there are no
> >        negatives for either.
> >
> >> 
> >> > > And what's the real benefit of saving just a lookup out of an already
> >> > > infrequent operation like ARP due to cacheing?
> >> >
> >> >         I would like to know what is the drawback that you see that
> >> >         you don't want one to go ahead with it? ARP is repeated every
> >> >         so often and happens quite often. It also happens on failover
> >> >         - it is preferable that these operations be simple, fast and
> >> >         error-free.
> >> 
> >> No one is arguing with any of these assertions.   The issue is strictly 
> >> what is the architectural minimum the spec should define and what should be 
> >> left for future spec or vendor implementation to solve.
> >> 
> >> 
> >> >         Secondly, things are more complicated the way you propose.
> >> >         Note that the first level address would be first determined.
> >> >         This completes ARP/ND. However, address resolution is not
> >> >         complete. The packet is sent as far as IP is concerned and now
> >> >         the SA/SM is consulted to get the LID etc. This has very
> >> >         strong shades of ATM VCs. IP thought it had sent the packet
> >> >         since the ARP/ND was complete - it took a while before the
> >> >         lower layer completed the determination. The driver of course
> >> >         keeps a GID to LID table too since the LID is a separate
> >> >         entity. More complication.
> >> 
> >> This is one implementation choice and while it may be used by some, it does 
> >> not require the spec to make this a requirement.
> >
> >        The assertion against LIDs was that they cause complication. I
> >        am showing that it is the other way down. This description is
> >        not part of the spec. but shows that the argument that
> >        inclusion of LIDs causes more complexity is incorrect. It is
> >        the GID to LID conversion that is more complicated. Including
> >        LIDs in address resolution allows for a simple, 2 message,
> >        fast failover solution.
> >
> >        I have also noted that the process, in another mail, of
> >        resolving the IP address is simpler and fits in more with a
> >        'single method' when LIDs are included than does the process
> >        when the LID is not. I have not seen a counter example.
> >
> >
> >> 
> >> 
> >> >         Include LID in the resolution - fast/simple/error-free.
> >> 
> >> It isn't that simple either
> >
> >        As shown above it is simpler than getting the GID and the
> >        answer. I request a corresponding scenario that shows that the
> >        GID to LID conversion is a better solution. I am not against
> >        anyone implementing it differently. Allow those that want ot
> >        implenet it in this way to do so.
> >
> >
> >> 
> >> >         I was pointing out another issue that many have raised.
> >> >         In general I am ok with dropping GID since it is for the
> >> >         future. IT buys us nothing as of now. So if we dropped GID we
> >> >         can work with these 48 bits. When IPoIB in reality starts
> >> >         spanning multiple IPoIB subnets (not for a long time) we can
> >> >         include GID. Then the hardware length will become 22 bytes as
> >> >         against 6 bytes. One can still interoperate quite easily.
> >> 
> >> GID is required because it is the primary way and application determines 
> >> where a service resides on a given port as such not including it makes life 
> >> quite difficult to implement.  GIDs are also used by applications within 
> >> the address vector to identify a given vector to use for a given service 
> >> such as IP.  GIDs cannot be dropped and have value.
> >> 
> >> LIDs are strictly a local issue that will be learned for all CA / router 
> >> ports participating within a given partition that support IP services.  As 
> >> such, their assignment, management, transitory nature, etc. make them 
> >> unsuited as an architectural minimum to include within the spec.  I see no 
> >> problem with people ascertaining the value and using it for optimizing 
> >> their implementation but this is outside the scope of the specification.
> >
> >        I don't understand. If those that don't want this information
> >        just ignore it what is the loss? We can format the link
> >        description such that the LID is the last element. Those that
> >        don't want it can drop this piece of information and ignore
> >        it. Of course those that don't want to use GID can ignore it
> >        too. 
> >
> >> 
> >> >         Wouldn't that work for all ?
> >> >
> >> >         I included the GID because many want to put in the hooks for
> >> >         the future case of IPoIB spanning IB subnets. I would though
> >> >         like a valid reason specified for why do we need IP subnets to
> >> >         span IB subnets. Isn't IP subnetting enough?
> >> 
> >> IB is viewed as a layer 2 fabric w.r.t. IP so the fact that IB has multiple 
> >> subnets is independent of IP operation.  I see no point in dealing with the 
> >> IB subnet issues within this specification other than to state that a LID 
> >> is not an end-to-end address between two endnode ports and thus should not 
> >> be mandated within the IP over IB spec for address resolution.
> >> 
> >> Mike
> >> 
> >> 
> >> _______________________________________________
> >> IPoverIB mailing list
> >> IPoverIB@ietf.org
> >> http://www.ietf.org/mailman/listinfo/ipoverib
> >> 
> >
> >
> >-- 
> >Vivek Kashyap
> >Linux Technology Center, IBM
> >kashyapv@us.ibm.com 
> >vivk@us.ibm.com 
> >503 578 3422 (o)
> >-- 
> >Vivek Kashyap
> >Linux Technology Center, IBM
> >kashyapv@us.ibm.com 
> >vivk@us.ibm.com 
> >503 578 3422 (o)
> >
> >_______________________________________________
> >IPoverIB mailing list
> >IPoverIB@ietf.org
> >http://www.ietf.org/mailman/listinfo/ipoverib
> >
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 16 14:04:20 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA17788
	for <ipoverib-archive@odin.ietf.org>; Wed, 16 Jan 2002 14:04:20 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id OAA15634
	for ipoverib-archive@odin.ietf.org; Wed, 16 Jan 2002 14:04:23 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA15600;
	Wed, 16 Jan 2002 14:03:12 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA15008
	for <ipoverib@optimus.ietf.org>; Wed, 16 Jan 2002 13:59:46 -0500 (EST)
Received: from mail5.microsoft.com (mail5.microsoft.com [131.107.3.121])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA17668
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 13:59:40 -0500 (EST)
Received: from inet-vrs-05.redmond.corp.microsoft.com ([157.54.6.148]) by mail5.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Wed, 16 Jan 2002 10:59:11 -0800
Received: from 157.54.6.150 by inet-vrs-05.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Wed, 16 Jan 2002 10:59:11 -0800
Received: from red-imc-02.redmond.corp.microsoft.com ([157.54.9.107]) by inet-hub-05.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Wed, 16 Jan 2002 10:59:11 -0800
Received: from win-imc-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.39]) by red-imc-02.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Wed, 16 Jan 2002 10:59:10 -0800
Received: from win-msg-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.52]) by win-imc-01.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3588.0);
	 Wed, 16 Jan 2002 10:57:05 -0800
X-MimeOLE: Produced By Microsoft Exchange V6.0.6132.0
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
Date: Wed, 16 Jan 2002 10:57:05 -0800
Message-ID: <2E33960095B58E40A4D3345AB9F65EC102261862@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
Thread-Topic: [Ipoverib] What's inside an IPoIB link layer address?
Thread-Index: AcGeu17GK2Du5tBtQ6mWoV3vEOqMXwAA7hJQ
From: "Arvind Murching" <arvindm@windows.microsoft.com>
To: "Vivek Kashyap" <kashyapv@us.ibm.com>
Cc: <ipoverib@ietf.org>
X-OriginalArrivalTime: 16 Jan 2002 18:57:05.0657 (UTC) FILETIME=[973FD690:01C19EBF]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id NAA15009
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

Why do we need to include the LID inside the link layer address when it
is already available in the SLID field? Previous discussions indicate
that the SLID value works for proxy ARP/bridge implementations.

- Arvind

-----Original Message-----
From: Vivek Kashyap [mailto:kashyapv@us.ibm.com] 
Sent: Wednesday, January 16, 2002 10:29 AM
To: kamath@seagullsemi.com
Cc: ipoverib@ietf.org
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?

Ashwin,
Ashwin,

        I agree that you get more flexibility if you include the LID.
        If we include the GID along with the LID we allow all
        implementations the choice to create different solutions. We
        don't tie them down to one thing. Choosing one or the other
        (LID or GID) blocks out various advantages that different
        implementations seek. Hence I suggested that let us include
        both and move on.

Vivek

<snip>

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 16 14:31:46 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA18549
	for <ipoverib-archive@odin.ietf.org>; Wed, 16 Jan 2002 14:31:46 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id OAA16865
	for ipoverib-archive@odin.ietf.org; Wed, 16 Jan 2002 14:31:49 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA16825;
	Wed, 16 Jan 2002 14:31:02 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA16782
	for <ipoverib@optimus.ietf.org>; Wed, 16 Jan 2002 14:31:00 -0500 (EST)
Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA18529
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 14:30:55 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id NAA58892;
	Wed, 16 Jan 2002 13:26:44 -0600
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0GJUts54780;
	Wed, 16 Jan 2002 14:30:55 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id g0GJUrg23938;
	Wed, 16 Jan 2002 11:30:53 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0GJUrp24087;
	Wed, 16 Jan 2002 11:30:53 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201161930.g0GJUrp24087@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
To: arvindm@windows.microsoft.com (Arvind Murching)
Date: Wed, 16 Jan 2002 11:30:52 -0800 (PST)
Cc: kashyapv@us.ibm.com (Vivek Kashyap), ipoverib@ietf.org
In-Reply-To: <2E33960095B58E40A4D3345AB9F65EC102261862@win-msg-01.wingroup.windeploy.ntdev.microsoft.com> from "Arvind Murching" at Jan 16, 2002 09:57:05 AM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

        A device can always respond for another. The LID of the sender
        is not the same as the LID for which it has responded. It also
        allows any network monitoring/debugging device to work
        independently. This is the reason why none of the other media
        does anything like this. You don't pick ethernet source
        address from the frame but take it from the ARP/ND packet.

        [ If it was ok we wouldn't need anything except QPN/cap_flags
        - get the LID from response, consult SA using LID for GID if
        you want that, and you are done. ]

Vivek

> 
> Why do we need to include the LID inside the link layer address when it
> is already available in the SLID field? Previous discussions indicate
> that the SLID value works for proxy ARP/bridge implementations.
> 
> - Arvind
> 
> -----Original Message-----
> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com] 
> Sent: Wednesday, January 16, 2002 10:29 AM
> To: kamath@seagullsemi.com
> Cc: ipoverib@ietf.org
> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> 
> Ashwin,
> Ashwin,
> 
>         I agree that you get more flexibility if you include the LID.
>         If we include the GID along with the LID we allow all
>         implementations the choice to create different solutions. We
>         don't tie them down to one thing. Choosing one or the other
>         (LID or GID) blocks out various advantages that different
>         implementations seek. Hence I suggested that let us include
>         both and move on.
> 
> Vivek
> 
> <snip>
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 16 14:40:44 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA18809
	for <ipoverib-archive@odin.ietf.org>; Wed, 16 Jan 2002 14:40:44 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id OAA17069
	for ipoverib-archive@odin.ietf.org; Wed, 16 Jan 2002 14:40:47 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA17047;
	Wed, 16 Jan 2002 14:39:02 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA17009
	for <ipoverib@optimus.ietf.org>; Wed, 16 Jan 2002 14:38:59 -0500 (EST)
Received: from mail6.microsoft.com (mail6.microsoft.com [131.107.3.126])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA18758
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 14:38:54 -0500 (EST)
Received: from inet-vrs-06.redmond.corp.microsoft.com ([157.54.6.201]) by mail6.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Wed, 16 Jan 2002 11:38:25 -0800
Received: from 157.54.5.25 by inet-vrs-06.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Wed, 16 Jan 2002 11:38:25 -0800
Received: from red-imc-04.redmond.corp.microsoft.com ([157.54.2.168]) by inet-hub-03.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Wed, 16 Jan 2002 11:38:24 -0800
Received: from win-imc-02.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.84]) by red-imc-04.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Wed, 16 Jan 2002 11:38:24 -0800
Received: from win-msg-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.52]) by win-imc-02.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3588.0);
	 Wed, 16 Jan 2002 11:36:19 -0800
X-MimeOLE: Produced By Microsoft Exchange V6.0.6132.0
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
Date: Wed, 16 Jan 2002 11:36:19 -0800
Message-ID: <2E33960095B58E40A4D3345AB9F65EC102261863@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
Thread-Topic: [Ipoverib] What's inside an IPoIB link layer address?
Thread-Index: AcGexAqYyNk76AQQRs+O6/IbuvPdcwAAIB6w
From: "Arvind Murching" <arvindm@windows.microsoft.com>
To: "Vivek Kashyap" <kashyapv@us.ibm.com>
Cc: <ipoverib@ietf.org>
X-OriginalArrivalTime: 16 Jan 2002 19:36:19.0668 (UTC) FILETIME=[12596940:01C19EC5]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id OAA17010
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

You do need the GID in case you are crossing IB subnets.

Monitoring devices do not -need- to see a LID in the ARP response, it is
only implementations that are attempting to avoid an SM/SA lookup that
are interested in the LID. We shouldn't add redundant/relatively
transient values in the link layer address.

- Arvind

-----Original Message-----
From: Vivek Kashyap [mailto:kashyapv@us.ibm.com] 
Sent: Wednesday, January 16, 2002 11:31 AM
To: Arvind Murching
Cc: Vivek Kashyap; ipoverib@ietf.org
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?

        A device can always respond for another. The LID of the sender
        is not the same as the LID for which it has responded. It also
        allows any network monitoring/debugging device to work
        independently. This is the reason why none of the other media
        does anything like this. You don't pick ethernet source
        address from the frame but take it from the ARP/ND packet.

        [ If it was ok we wouldn't need anything except QPN/cap_flags
        - get the LID from response, consult SA using LID for GID if
        you want that, and you are done. ]

Vivek

> 
> Why do we need to include the LID inside the link layer address when
it
> is already available in the SLID field? Previous discussions indicate
> that the SLID value works for proxy ARP/bridge implementations.
> 
> - Arvind
> 
> -----Original Message-----
> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com] 
> Sent: Wednesday, January 16, 2002 10:29 AM
> To: kamath@seagullsemi.com
> Cc: ipoverib@ietf.org
> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> 
> Ashwin,
> Ashwin,
> 
>         I agree that you get more flexibility if you include the LID.
>         If we include the GID along with the LID we allow all
>         implementations the choice to create different solutions. We
>         don't tie them down to one thing. Choosing one or the other
>         (LID or GID) blocks out various advantages that different
>         implementations seek. Hence I suggested that let us include
>         both and move on.
> 
> Vivek
> 
> <snip>
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 16 15:17:51 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA19958
	for <ipoverib-archive@odin.ietf.org>; Wed, 16 Jan 2002 15:17:50 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id PAA18030
	for ipoverib-archive@odin.ietf.org; Wed, 16 Jan 2002 15:17:54 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA17934;
	Wed, 16 Jan 2002 15:09:29 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA17887
	for <ipoverib@optimus.ietf.org>; Wed, 16 Jan 2002 15:09:25 -0500 (EST)
Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA19665
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 15:09:20 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e31.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id PAA149744;
	Wed, 16 Jan 2002 15:06:19 -0500
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0GK9H055162;
	Wed, 16 Jan 2002 13:09:17 -0700
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id g0GK9Gg24801;
	Wed, 16 Jan 2002 12:09:16 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0GK9Gk26559;
	Wed, 16 Jan 2002 12:09:16 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201162009.g0GK9Gk26559@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
To: arvindm@windows.microsoft.com (Arvind Murching)
Date: Wed, 16 Jan 2002 12:09:15 -0800 (PST)
Cc: kashyapv@us.ibm.com (Vivek Kashyap), ipoverib@ietf.org
In-Reply-To: <2E33960095B58E40A4D3345AB9F65EC102261863@win-msg-01.wingroup.windeploy.ntdev.microsoft.com> from "Arvind Murching" at Jan 16, 2002 10:36:19 AM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Arvind, 

        Multiple pros/cons for both have been suggested over the last
        few days. As noted earlier let us allow for all solutions that
        people want. 

	So include LID and GID both.

Vivek

> 
> You do need the GID in case you are crossing IB subnets.

        Why are you doing that ? Why are you supporting IP subnets
        spanning IB subnets ? Advantages ?


> 
> Monitoring devices do not -need- to see a LID in the ARP response, it is
> only implementations that are attempting to avoid an SM/SA lookup that
> are interested in the LID. We shouldn't add redundant/relatively
> transient values in the link layer address.
	
        Depends on what you have in mind. Other reasons have been put
        forth in the last note and in previous discussions.

> 
> - Arvind
> 
> -----Original Message-----
> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com] 
> Sent: Wednesday, January 16, 2002 11:31 AM
> To: Arvind Murching
> Cc: Vivek Kashyap; ipoverib@ietf.org
> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> 
>         A device can always respond for another. The LID of the sender
>         is not the same as the LID for which it has responded. It also
>         allows any network monitoring/debugging device to work
>         independently. This is the reason why none of the other media
>         does anything like this. You don't pick ethernet source
>         address from the frame but take it from the ARP/ND packet.
> 
>         [ If it was ok we wouldn't need anything except QPN/cap_flags
>         - get the LID from response, consult SA using LID for GID if
>         you want that, and you are done. ]
> 
> Vivek
> 
> > 
> > Why do we need to include the LID inside the link layer address when
> it
> > is already available in the SLID field? Previous discussions indicate
> > that the SLID value works for proxy ARP/bridge implementations.
> > 
> > - Arvind
> > 
> > -----Original Message-----
> > From: Vivek Kashyap [mailto:kashyapv@us.ibm.com] 
> > Sent: Wednesday, January 16, 2002 10:29 AM
> > To: kamath@seagullsemi.com
> > Cc: ipoverib@ietf.org
> > Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> > 
> > Ashwin,
> > Ashwin,
> > 
> >         I agree that you get more flexibility if you include the LID.
> >         If we include the GID along with the LID we allow all
> >         implementations the choice to create different solutions. We
> >         don't tie them down to one thing. Choosing one or the other
> >         (LID or GID) blocks out various advantages that different
> >         implementations seek. Hence I suggested that let us include
> >         both and move on.
> > 
> > Vivek
> > 
> > <snip>
> > 
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> > 
> 
> 
> -- 
> Vivek Kashyap
> Linux Technology Center, IBM
> kashyapv@us.ibm.com 
> vivk@us.ibm.com 
> 503 578 3422 (o)
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 16 18:59:58 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA23972
	for <ipoverib-archive@odin.ietf.org>; Wed, 16 Jan 2002 18:59:53 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id SAA25184
	for ipoverib-archive@odin.ietf.org; Wed, 16 Jan 2002 18:59:57 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA25150;
	Wed, 16 Jan 2002 18:56:55 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA25120
	for <ipoverib@optimus.ietf.org>; Wed, 16 Jan 2002 18:56:54 -0500 (EST)
Received: from e24.nc.us.ibm.com (e24.nc.us.ibm.com [32.97.136.230])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA23928
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 18:56:49 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e24.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id RAA38930
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 17:53:51 -0600
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0GNuqs163738
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 18:56:52 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id g0GNupg29955
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 15:56:51 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0GNuoi10058
	for ipoverib@ietf.org; Wed, 16 Jan 2002 15:56:50 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201162356.g0GNuoi10058@eng4.beaverton.ibm.com>
To: ipoverib@ietf.org
Date: Wed, 16 Jan 2002 15:56:50 -0800 (PST)
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Subject: [Ipoverib] encapsulation issues
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit


1.      We have been arguing over the inclusion of LID/GID or both for
        a while now. Examples and advantages of including either have
        been espoused often enough. However, there is an agreement, I
        think, on the architecture not forcing a limited solution on
        any set of aspirations.

        As noted earlier the inclusion of both the identifiers does
        not handicap or restrict any solution. 
	
	Let us include both.


2. 	I believe based on the recent set of mails that the consensus no
        longer favours inclusion of the 'ethertype'. Is there a
        disagreement ?


3. 	As many have noted in the last couple of days that using IB
        over non-UD modes or utilising QPNs mapped to specific
        endpoints is desirable. A complete, interoperable solution has
        been presented in
        draft-kashyap-ipoib-advanced-capabilities-00.txt. 

	Please review and comment. 

Vivek


	

-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 16 20:24:04 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA25251
	for <ipoverib-archive@odin.ietf.org>; Wed, 16 Jan 2002 20:24:04 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id UAA27523
	for ipoverib-archive@odin.ietf.org; Wed, 16 Jan 2002 20:24:07 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA27451;
	Wed, 16 Jan 2002 20:18:33 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA27417
	for <ipoverib@optimus.ietf.org>; Wed, 16 Jan 2002 20:18:31 -0500 (EST)
Received: from calliope1.fm.intel.com (fmfdns01.fm.intel.com [132.233.247.10])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA25184
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 20:18:27 -0500 (EST)
Received: from fmsmsxvs041.fm.intel.com (fmsmsxv041-1.fm.intel.com [132.233.48.109])
	by calliope1.fm.intel.com (8.9.1a+p1/8.9.1/d: relay.m4,v 1.48 2001/12/13 16:27:50 root Exp $) with SMTP id BAA16030
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 01:18:29 GMT
Received: from fmsmsx28.fm.intel.com ([132.233.42.28])
 by fmsmsxvs041.fm.intel.com (NAVGW 2.5.1.16) with SMTP id M2002011617192709184
 ; Wed, 16 Jan 2002 17:19:27 -0800
Received: by fmsmsx28.fm.intel.com with Internet Mail Service (5.5.2653.19)
	id <C021Z2FQ>; Wed, 16 Jan 2002 17:18:29 -0800
Message-ID: <D9223EB959A5D511A98F00508B68C20C21A849@ORSMSX108>
From: "Larsen, Roy K" <roy.k.larsen@intel.com>
To: "'Vivek Kashyap'" <kashyapv@us.ibm.com>, ipoverib@ietf.org
Subject: RE: [Ipoverib] encapsulation issues
Date: Wed, 16 Jan 2002 17:18:22 -0800
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="ISO-8859-1"
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

> 
> 1.      We have been arguing over the inclusion of LID/GID or both for
>         a while now. Examples and advantages of including either have
>         been espoused often enough. However, there is an agreement, I
>         think, on the architecture not forcing a limited solution on
>         any set of aspirations.
> 
>         As noted earlier the inclusion of both the identifiers does
>         not handicap or restrict any solution. 
> 	
> 	Let us include both.
> 
> 
> 2. 	I believe based on the recent set of mails that the consensus no
>         longer favours inclusion of the 'ethertype'. Is there a
>         disagreement ?

There were many that said that an Ethertype was desirable and useful in
their implementations.  Your counter argument was that it was not needed
because it could be inferred through other means so why add an additional
mechanism.  If you would allow us to use the same logic used in #1 above
("the architecture not forcing a limited solution on any set of
aspirations"), then I would like an Ethertype defined as part of the
payload.  As noted earlier, the inclusion of an Ethertype does not handicap
or restrict any solution.

Roy

> 
> 
> 3. 	As many have noted in the last couple of days that using IB
>         over non-UD modes or utilising QPNs mapped to specific
>         endpoints is desirable. A complete, interoperable solution has
>         been presented in
>         draft-kashyap-ipoib-advanced-capabilities-00.txt. 
> 
> 	Please review and comment. 
> 
> Vivek
> 
> 
> 	
> 
> -- 
> Vivek Kashyap
> Linux Technology Center, IBM
> kashyapv@us.ibm.com 
> vivk@us.ibm.com 
> 503 578 3422 (o)
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 16 20:30:50 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA25366
	for <ipoverib-archive@odin.ietf.org>; Wed, 16 Jan 2002 20:30:50 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id UAA28051
	for ipoverib-archive@odin.ietf.org; Wed, 16 Jan 2002 20:30:53 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA27710;
	Wed, 16 Jan 2002 20:30:05 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA27670
	for <ipoverib@optimus.ietf.org>; Wed, 16 Jan 2002 20:30:03 -0500 (EST)
Received: from mxout1.netvision.net.il (mxout1.netvision.net.il [194.90.9.20])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA25334
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 20:29:58 -0500 (EST)
Received: from vaio505he ([139.92.208.14]) by mxout1.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GQ200M2F6T3VO@mxout1.netvision.net.il> for ipoverib@ietf.org;
 Thu, 17 Jan 2002 03:29:29 +0200 (IST)
Date: Thu, 17 Jan 2002 03:30:09 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: RE: [Ipoverib] encapsulation issues
In-reply-to: <200201162356.g0GNuoi10058@eng4.beaverton.ibm.com>
To: "'Vivek Kashyap'" <kashyapv@us.ibm.com>, ipoverib@ietf.org
Reply-to: yaronh@voltaire.com
Message-id: <01a501c19ef6$8127b0d0$0100007f@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7BIT

Vivek,

Why not wait with the capability field after we finish with the other more
pressing stuff
For now we could specify reserved capability bits that
must-be-zero on transmit and not checked on receive, and then define
when we get the first draft ready we could move to the next

Yaron

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Vivek Kashyap
Sent: Thursday, January 17, 2002 1:57 AM
To: ipoverib@ietf.org
Subject: [Ipoverib] encapsulation issues


1.      We have been arguing over the inclusion of LID/GID or both for
        a while now. Examples and advantages of including either have
        been espoused often enough. However, there is an agreement, I
        think, on the architecture not forcing a limited solution on
        any set of aspirations.

        As noted earlier the inclusion of both the identifiers does
        not handicap or restrict any solution.

        Let us include both.


2.      I believe based on the recent set of mails that the consensus no
        longer favours inclusion of the 'ethertype'. Is there a
        disagreement ?


3.      As many have noted in the last couple of days that using IB
        over non-UD modes or utilising QPNs mapped to specific
        endpoints is desirable. A complete, interoperable solution has
        been presented in
        draft-kashyap-ipoib-advanced-capabilities-00.txt.

        Please review and comment.

Vivek




--
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com
vivk@us.ibm.com
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 16 20:42:33 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA25457
	for <ipoverib-archive@odin.ietf.org>; Wed, 16 Jan 2002 20:42:33 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id UAA28268
	for ipoverib-archive@odin.ietf.org; Wed, 16 Jan 2002 20:42:35 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA28248;
	Wed, 16 Jan 2002 20:41:55 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA28219
	for <ipoverib@optimus.ietf.org>; Wed, 16 Jan 2002 20:41:52 -0500 (EST)
Received: from mxout2.netvision.net.il (mxout2.netvision.net.il [194.90.9.21])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA25451
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 20:41:48 -0500 (EST)
Received: from vaio505he ([139.92.208.14]) by mxout2.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GQ2000157CS2F@mxout2.netvision.net.il> for ipoverib@ietf.org;
 Thu, 17 Jan 2002 03:41:19 +0200 (IST)
Date: Thu, 17 Jan 2002 03:41:59 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
In-reply-to: <200201162009.g0GK9Gk26559@eng4.beaverton.ibm.com>
To: "'Vivek Kashyap'" <kashyapv@us.ibm.com>,
        "'Arvind Murching'" <arvindm@windows.microsoft.com>
Cc: ipoverib@ietf.org
Reply-to: yaronh@voltaire.com
Message-id: <01a601c19ef8$2837ad20$0100007f@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7BIT

Arvind, Vivek

I'm sure we must have the GID as the primary address
There is a need for IP spanning IB this is what I hear from many of the data
centers I visit, As noted in few of the emails there are also few other
reasons for GID
lets first agree on that !,
Vivek the key for including the LID is that people will have a primary (GID)
address that determine the route, to avoid interoperability issues
(I'm sure this was also the consensus in LWG befor we went to IETF)

We need to maintain the tables by the GID, however I support taking the LID
to return the answer to the caller (avoiding the SM as the main reason)
I'm not sure that we could take the LID from the packet since someone might
Forward it somehow (test tools, ARP server or something like that)
And we don't want a broken spec

No one mentioned in the arguments the additional latency caused by going to
The SM (for the LID)
As a central resource that probably has queuing it might take time
Not to mention the latency in IP span IB case (needing to resolve all the
Path first)

lets go for the LID in the header, and specify that the GID is the primary
address

Yaron

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Vivek Kashyap
Sent: Wednesday, January 16, 2002 10:09 PM
To: Arvind Murching
Cc: Vivek Kashyap; ipoverib@ietf.org
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?

Arvind,

        Multiple pros/cons for both have been suggested over the last
        few days. As noted earlier let us allow for all solutions that
        people want.

        So include LID and GID both.

Vivek

>
> You do need the GID in case you are crossing IB subnets.

        Why are you doing that ? Why are you supporting IP subnets
        spanning IB subnets ? Advantages ?


>
> Monitoring devices do not -need- to see a LID in the ARP response, it is
> only implementations that are attempting to avoid an SM/SA lookup that
> are interested in the LID. We shouldn't add redundant/relatively
> transient values in the link layer address.

        Depends on what you have in mind. Other reasons have been put
        forth in the last note and in previous discussions.

>
> - Arvind
>
> -----Original Message-----
> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> Sent: Wednesday, January 16, 2002 11:31 AM
> To: Arvind Murching
> Cc: Vivek Kashyap; ipoverib@ietf.org
> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>
>         A device can always respond for another. The LID of the sender
>         is not the same as the LID for which it has responded. It also
>         allows any network monitoring/debugging device to work
>         independently. This is the reason why none of the other media
>         does anything like this. You don't pick ethernet source
>         address from the frame but take it from the ARP/ND packet.
>
>         [ If it was ok we wouldn't need anything except QPN/cap_flags
>         - get the LID from response, consult SA using LID for GID if
>         you want that, and you are done. ]
>
> Vivek
>
> >
> > Why do we need to include the LID inside the link layer address when
> it
> > is already available in the SLID field? Previous discussions indicate
> > that the SLID value works for proxy ARP/bridge implementations.
> >
> > - Arvind
> >
> > -----Original Message-----
> > From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> > Sent: Wednesday, January 16, 2002 10:29 AM
> > To: kamath@seagullsemi.com
> > Cc: ipoverib@ietf.org
> > Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> >
> > Ashwin,
> > Ashwin,
> >
> >         I agree that you get more flexibility if you include the LID.
> >         If we include the GID along with the LID we allow all
> >         implementations the choice to create different solutions. We
> >         don't tie them down to one thing. Choosing one or the other
> >         (LID or GID) blocks out various advantages that different
> >         implementations seek. Hence I suggested that let us include
> >         both and move on.
> >
> > Vivek
> >
> > <snip>
> >
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> >
>
>
> --
> Vivek Kashyap
> Linux Technology Center, IBM
> kashyapv@us.ibm.com
> vivk@us.ibm.com
> 503 578 3422 (o)
>
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
>


--
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com
vivk@us.ibm.com
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 16 21:16:19 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA25676
	for <ipoverib-archive@odin.ietf.org>; Wed, 16 Jan 2002 21:16:19 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id VAA29150
	for ipoverib-archive@odin.ietf.org; Wed, 16 Jan 2002 21:16:22 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA29122;
	Wed, 16 Jan 2002 21:15:39 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA29090
	for <ipoverib@optimus.ietf.org>; Wed, 16 Jan 2002 21:15:37 -0500 (EST)
Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA25669
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 21:15:33 -0500 (EST)
Received: from sunmail1.Sun.COM ([129.145.1.2])
	by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id SAA00187;
	Wed, 16 Jan 2002 18:15:33 -0800 (PST)
Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.86.38])
	by sunmail1.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1-Sun.COM.mod.2) with ESMTP id SAA11685;
	Wed, 16 Jan 2002 18:16:47 -0800 (PST)
Received: from taipei (taipei.Eng.Sun.COM [129.146.85.178])
	by jurassic.eng.sun.com (8.12.2+Sun/8.12.2) with SMTP id g0H2FXqT592522;
	Wed, 16 Jan 2002 18:15:33 -0800 (PST)
Message-Id: <200201170215.g0H2FXqT592522@jurassic.eng.sun.com>
Date: Wed, 16 Jan 2002 18:13:01 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
To: arvindm@windows.microsoft.com, kashyapv@us.ibm.com
Cc: ipoverib@ietf.org
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: WWNshB158c6Nn5S9n34yUQ==
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.6_06 SunOS 5.8 sun4u sparc 
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

All the proxy ARP usages that I know of use the proxy's mac addr in the
ARP response. I believe the flavor of the proxy ARP you alluded below is
rare, and won't work very well anyway if the link address includes any
non-persistent component like LID and QPN because it will require another
protocol between the clients and their proxy to constantly update the
proxy database containing clients mac addrs, which may change quite
often.

That is also why I wanted to provide an option to not include QPN.
(But I haven't succeeded w/o requiring SIDR or something equivalent.)

If LID is included as part of link addr and if it ever changes on
a live node due to subnet merge...etc, the driver must inform its IP
stack about such change. I believe many stacks don't have such a
facility. (FOr most IP stacks the assumption is that its own MAC
addresses just don't change on the fly.) In that case you're totally
hosed. Can someone comment on how likely a LID assignment gets
rescinded?

Jerry

>        A device can always respond for another. The LID of the sender
>        is not the same as the LID for which it has responded. It also
>        allows any network monitoring/debugging device to work
>        independently. This is the reason why none of the other media
>        does anything like this. You don't pick ethernet source
>        address from the frame but take it from the ARP/ND packet.
>
>        [ If it was ok we wouldn't need anything except QPN/cap_flags
>        - get the LID from response, consult SA using LID for GID if
>        you want that, and you are done. ]
>
>Vivek
>
>> 
>> Why do we need to include the LID inside the link layer address when it
>> is already available in the SLID field? Previous discussions indicate
>> that the SLID value works for proxy ARP/bridge implementations.
>> 
>> - Arvind
>> 
>> -----Original Message-----
>> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com] 
>> Sent: Wednesday, January 16, 2002 10:29 AM
>> To: kamath@seagullsemi.com
>> Cc: ipoverib@ietf.org
>> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>> 
>> Ashwin,
>> Ashwin,
>> 
>>         I agree that you get more flexibility if you include the LID.
>>         If we include the GID along with the LID we allow all
>>         implementations the choice to create different solutions. We
>>         don't tie them down to one thing. Choosing one or the other
>>         (LID or GID) blocks out various advantages that different
>>         implementations seek. Hence I suggested that let us include
>>         both and move on.
>> 
>> Vivek
>> 
>> <snip>
>> 
>> _______________________________________________
>> IPoverIB mailing list
>> IPoverIB@ietf.org
>> http://www.ietf.org/mailman/listinfo/ipoverib
>> 
>
>
>-- 
>Vivek Kashyap
>Linux Technology Center, IBM
>kashyapv@us.ibm.com 
>vivk@us.ibm.com 
>503 578 3422 (o)
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 16 21:45:57 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA26925
	for <ipoverib-archive@odin.ietf.org>; Wed, 16 Jan 2002 21:45:57 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id VAA00030
	for ipoverib-archive@odin.ietf.org; Wed, 16 Jan 2002 21:46:00 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA29888;
	Wed, 16 Jan 2002 21:42:45 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA29857
	for <ipoverib@optimus.ietf.org>; Wed, 16 Jan 2002 21:42:43 -0500 (EST)
Received: from calliope1.fm.intel.com (fmfdns01.fm.intel.com [132.233.247.10])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA26878
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 21:42:38 -0500 (EST)
Received: from fmsmsxvs041.fm.intel.com (fmsmsxv041-1.fm.intel.com [132.233.48.109])
	by calliope1.fm.intel.com (8.9.1a+p1/8.9.1/d: relay.m4,v 1.48 2001/12/13 16:27:50 root Exp $) with SMTP id CAA04499
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 02:42:41 GMT
Received: from FMSMSX016.fm.intel.com ([132.233.42.195])
 by fmsmsxvs041.fm.intel.com (NAVGW 2.5.1.16) with SMTP id M2002011618433826502
 ; Wed, 16 Jan 2002 18:43:38 -0800
Received: by fmsmsx016.fm.intel.com with Internet Mail Service (5.5.2653.19)
	id <DAG4B552>; Wed, 16 Jan 2002 18:42:40 -0800
Message-ID: <D9223EB959A5D511A98F00508B68C20C21A84E@ORSMSX108>
From: "Larsen, Roy K" <roy.k.larsen@intel.com>
To: "'H.K. Jerry Chu'" <Jerry.Chu@eng.sun.com>, arvindm@windows.microsoft.com,
        kashyapv@us.ibm.com
Cc: ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
Date: Wed, 16 Jan 2002 18:42:31 -0800
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="ISO-8859-1"
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

> -----Original Message-----
> From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
> Sent: Wednesday, January 16, 2002 6:13 PM
> To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com
> Cc: ipoverib@ietf.org
> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> 
> 
> All the proxy ARP usages that I know of use the proxy's mac 
> addr in the
> ARP response. I believe the flavor of the proxy ARP you 
> alluded below is
> rare, and won't work very well anyway if the link address includes any
> non-persistent component like LID and QPN because it will 
> require another
> protocol between the clients and their proxy to constantly update the
> proxy database containing clients mac addrs, which may change quite
> often.
> 
> That is also why I wanted to provide an option to not include QPN.
> (But I haven't succeeded w/o requiring SIDR or something equivalent.)
> 
> If LID is included as part of link addr and if it ever changes on
> a live node due to subnet merge...etc, the driver must inform its IP
> stack about such change. I believe many stacks don't have such a
> facility. (FOr most IP stacks the assumption is that its own MAC
> addresses just don't change on the fly.) In that case you're totally
> hosed. Can someone comment on how likely a LID assignment gets
> rescinded?

It is possible anytime the link goes down and comes back up again and when
the fabic is reconfigured.  This can happen on reboot or simply
unplugging/plugging in a link.  It is ultimately a function of the SM and
how much trouble the implementation wants to go through to keep LIDs
constant.

But of course, that has no bearing on whether the LID is contained in the
ARP.  You always need the LID.  Unless you want to do a pathrecord lookup
each time your implementation is given a packet to send, you're going to use
a cached LID no matter if it came in the ARP or a pathrecord lookup on the
GID.  If it becomes stale, it may take an ARP cache flush timeout to
recover.  

Roy
 
> 
> Jerry
> 
> >        A device can always respond for another. The LID of 
> the sender
> >        is not the same as the LID for which it has 
> responded. It also
> >        allows any network monitoring/debugging device to work
> >        independently. This is the reason why none of the other media
> >        does anything like this. You don't pick ethernet source
> >        address from the frame but take it from the ARP/ND packet.
> >
> >        [ If it was ok we wouldn't need anything except QPN/cap_flags
> >        - get the LID from response, consult SA using LID for GID if
> >        you want that, and you are done. ]
> >
> >Vivek
> >
> >> 
> >> Why do we need to include the LID inside the link layer 
> address when it
> >> is already available in the SLID field? Previous 
> discussions indicate
> >> that the SLID value works for proxy ARP/bridge implementations.
> >> 
> >> - Arvind
> >> 
> >> -----Original Message-----
> >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com] 
> >> Sent: Wednesday, January 16, 2002 10:29 AM
> >> To: kamath@seagullsemi.com
> >> Cc: ipoverib@ietf.org
> >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> >> 
> >> Ashwin,
> >> Ashwin,
> >> 
> >>         I agree that you get more flexibility if you 
> include the LID.
> >>         If we include the GID along with the LID we allow all
> >>         implementations the choice to create different 
> solutions. We
> >>         don't tie them down to one thing. Choosing one or the other
> >>         (LID or GID) blocks out various advantages that different
> >>         implementations seek. Hence I suggested that let us include
> >>         both and move on.
> >> 
> >> Vivek
> >> 
> >> <snip>
> >> 
> >> _______________________________________________
> >> IPoverIB mailing list
> >> IPoverIB@ietf.org
> >> http://www.ietf.org/mailman/listinfo/ipoverib
> >> 
> >
> >
> >-- 
> >Vivek Kashyap
> >Linux Technology Center, IBM
> >kashyapv@us.ibm.com 
> >vivk@us.ibm.com 
> >503 578 3422 (o)
> >
> >_______________________________________________
> >IPoverIB mailing list
> >IPoverIB@ietf.org
> >http://www.ietf.org/mailman/listinfo/ipoverib
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 16 22:08:05 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA27274
	for <ipoverib-archive@odin.ietf.org>; Wed, 16 Jan 2002 22:08:05 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id WAA00844
	for ipoverib-archive@odin.ietf.org; Wed, 16 Jan 2002 22:08:09 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA00835;
	Wed, 16 Jan 2002 22:07:30 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA00797
	for <ipoverib@optimus.ietf.org>; Wed, 16 Jan 2002 22:07:28 -0500 (EST)
Received: from mail3.microsoft.com (mail3.microsoft.com [131.107.3.123])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA27269
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 22:07:23 -0500 (EST)
Received: from INET-VRS-03.redmond.corp.microsoft.com ([157.54.5.27]) by mail3.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Wed, 16 Jan 2002 19:06:55 -0800
Received: from 157.54.6.197 by INET-VRS-03.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Wed, 16 Jan 2002 19:06:55 -0800
Received: from red-imc-02.redmond.corp.microsoft.com ([157.54.9.107]) by inet-hub-06.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Wed, 16 Jan 2002 19:06:55 -0800
Received: from win-imc-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.39]) by red-imc-02.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Wed, 16 Jan 2002 19:06:54 -0800
Received: from win-msg-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.52]) by win-imc-01.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3588.0);
	 Wed, 16 Jan 2002 19:04:48 -0800
X-MimeOLE: Produced By Microsoft Exchange V6.0.6132.0
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
Date: Wed, 16 Jan 2002 19:04:48 -0800
Message-ID: <2E33960095B58E40A4D3345AB9F65EC10226186B@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
Thread-Topic: [Ipoverib] What's inside an IPoIB link layer address?
Thread-Index: AcGe98tUOuEipiqiQQmn7Et1Tn+pGgACabBA
From: "Arvind Murching" <arvindm@windows.microsoft.com>
To: <yaronh@voltaire.com>, <ipoverib@ietf.org>
X-OriginalArrivalTime: 17 Jan 2002 03:04:48.0548 (UTC) FILETIME=[B94AA640:01C19F03]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id WAA00798
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

When you have an IP subnet spanning IB subnets, the LID of a responding
node is of no use.

Consider filling in static ARP entries (IP -> Link layer address
mappings kept in persistent store) - LID fields in such entries are of
no use.

Implementations may find creative ways to work/hack around these
deficiencies, but why should we design a protocol that requires these
workarounds?

IMO Jerry's suggestion of only using the GID is the most "idealistic"
solution if not for a dependency on SIDR.

- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com] 
Sent: Wednesday, January 16, 2002 5:42 PM
To: 'Vivek Kashyap'; Arvind Murching
Cc: ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?

Arvind, Vivek

I'm sure we must have the GID as the primary address
There is a need for IP spanning IB this is what I hear from many of the
data
centers I visit, As noted in few of the emails there are also few other
reasons for GID
lets first agree on that !,
Vivek the key for including the LID is that people will have a primary
(GID)
address that determine the route, to avoid interoperability issues
(I'm sure this was also the consensus in LWG befor we went to IETF)

We need to maintain the tables by the GID, however I support taking the
LID
to return the answer to the caller (avoiding the SM as the main reason)
I'm not sure that we could take the LID from the packet since someone
might
Forward it somehow (test tools, ARP server or something like that)
And we don't want a broken spec

No one mentioned in the arguments the additional latency caused by going
to
The SM (for the LID)
As a central resource that probably has queuing it might take time
Not to mention the latency in IP span IB case (needing to resolve all
the
Path first)

lets go for the LID in the header, and specify that the GID is the
primary
address

Yaron

[Arvind] <snip>


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 16 22:31:49 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA28314
	for <ipoverib-archive@odin.ietf.org>; Wed, 16 Jan 2002 22:31:48 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id WAA01786
	for ipoverib-archive@odin.ietf.org; Wed, 16 Jan 2002 22:31:52 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA01462;
	Wed, 16 Jan 2002 22:30:14 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA01420
	for <ipoverib@optimus.ietf.org>; Wed, 16 Jan 2002 22:30:12 -0500 (EST)
Received: from patan.sun.com (patan.Sun.COM [192.18.98.43])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA27509
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 22:30:07 -0500 (EST)
Received: from sunmail1.Sun.COM ([129.145.1.2])
	by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id UAA05333;
	Wed, 16 Jan 2002 20:30:08 -0700 (MST)
Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.85.105])
	by sunmail1.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1-Sun.COM.mod.2) with ESMTP id TAA26927;
	Wed, 16 Jan 2002 19:31:21 -0800 (PST)
Received: from taipei (taipei.Eng.Sun.COM [129.146.85.178])
	by jurassic.eng.sun.com (8.12.2+Sun/8.12.2) with SMTP id g0H3U7qT600299;
	Wed, 16 Jan 2002 19:30:07 -0800 (PST)
Message-Id: <200201170330.g0H3U7qT600299@jurassic.eng.sun.com>
Date: Wed, 16 Jan 2002 19:27:36 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
To: arvindm@windows.microsoft.com, kashyapv@us.ibm.com, roy.k.larsen@intel.com
Cc: ipoverib@ietf.org
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: kb6p+2LjEisyOcHMBLjsDQ==
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.6_06 SunOS 5.8 sun4u sparc 
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Roy,

Thanks for the info. A comment on your statement of volatility of LID having
no bearing on whether LID should be included in ARP reply. Yes it does. One
of the goal of IPoIB is to minimize the changes to the existing IP stack
implementations. Note that I stated it as a goal, not a constraint. If we
must do it then we should.

Yes no matter what, LID is cached somewhere and must be flushed. The difference
is whether one must modify the existing stack code to add a facility to support
mac addr change, or do the necessary chore in a complete new IPoIB driver.
The former requires an upgrade of OS release that can take a long time. The
latter doesn't have such a problem since it's brand new.

One can also hope that LID flushing through SM can be made more reliable than
gratuitous ARP. But I don't know the SM archtecture enough to make such a claim.

To be fair I don't really know how many OSes lack support for mac add
change. The two that I know, one supports it, one doesn't.

I'm assuming there are solid facility in IBA to support the notification of
attribute changes to interested parties. Otherwise not only we should include
LID, but also all the rest in the link addr.

Jerry

>From: "Larsen, Roy K" <roy.k.larsen@intel.com>
>To: "'H.K. Jerry Chu'" <Jerry.Chu@eng.sun.com>, arvindm@windows.microsoft.com, kashyapv@us.ibm.com
>Cc: ipoverib@ietf.org
>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>Date: Wed, 16 Jan 2002 18:42:31 -0800
>MIME-Version: 1.0
>
>> -----Original Message-----
>> From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
>> Sent: Wednesday, January 16, 2002 6:13 PM
>> To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com
>> Cc: ipoverib@ietf.org
>> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>> 
>> 
>> All the proxy ARP usages that I know of use the proxy's mac 
>> addr in the
>> ARP response. I believe the flavor of the proxy ARP you 
>> alluded below is
>> rare, and won't work very well anyway if the link address includes any
>> non-persistent component like LID and QPN because it will 
>> require another
>> protocol between the clients and their proxy to constantly update the
>> proxy database containing clients mac addrs, which may change quite
>> often.
>> 
>> That is also why I wanted to provide an option to not include QPN.
>> (But I haven't succeeded w/o requiring SIDR or something equivalent.)
>> 
>> If LID is included as part of link addr and if it ever changes on
>> a live node due to subnet merge...etc, the driver must inform its IP
>> stack about such change. I believe many stacks don't have such a
>> facility. (FOr most IP stacks the assumption is that its own MAC
>> addresses just don't change on the fly.) In that case you're totally
>> hosed. Can someone comment on how likely a LID assignment gets
>> rescinded?
>
>It is possible anytime the link goes down and comes back up again and when
>the fabic is reconfigured.  This can happen on reboot or simply
>unplugging/plugging in a link.  It is ultimately a function of the SM and
>how much trouble the implementation wants to go through to keep LIDs
>constant.
>
>But of course, that has no bearing on whether the LID is contained in the
>ARP.  You always need the LID.  Unless you want to do a pathrecord lookup
>each time your implementation is given a packet to send, you're going to use
>a cached LID no matter if it came in the ARP or a pathrecord lookup on the
>GID.  If it becomes stale, it may take an ARP cache flush timeout to
>recover.  
>
>Roy
> 
>> 
>> Jerry
>> 
>> >        A device can always respond for another. The LID of 
>> the sender
>> >        is not the same as the LID for which it has 
>> responded. It also
>> >        allows any network monitoring/debugging device to work
>> >        independently. This is the reason why none of the other media
>> >        does anything like this. You don't pick ethernet source
>> >        address from the frame but take it from the ARP/ND packet.
>> >
>> >        [ If it was ok we wouldn't need anything except QPN/cap_flags
>> >        - get the LID from response, consult SA using LID for GID if
>> >        you want that, and you are done. ]
>> >
>> >Vivek
>> >
>> >> 
>> >> Why do we need to include the LID inside the link layer 
>> address when it
>> >> is already available in the SLID field? Previous 
>> discussions indicate
>> >> that the SLID value works for proxy ARP/bridge implementations.
>> >> 
>> >> - Arvind
>> >> 
>> >> -----Original Message-----
>> >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com] 
>> >> Sent: Wednesday, January 16, 2002 10:29 AM
>> >> To: kamath@seagullsemi.com
>> >> Cc: ipoverib@ietf.org
>> >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>> >> 
>> >> Ashwin,
>> >> Ashwin,
>> >> 
>> >>         I agree that you get more flexibility if you 
>> include the LID.
>> >>         If we include the GID along with the LID we allow all
>> >>         implementations the choice to create different 
>> solutions. We
>> >>         don't tie them down to one thing. Choosing one or the other
>> >>         (LID or GID) blocks out various advantages that different
>> >>         implementations seek. Hence I suggested that let us include
>> >>         both and move on.
>> >> 
>> >> Vivek
>> >> 
>> >> <snip>
>> >> 
>> >> _______________________________________________
>> >> IPoverIB mailing list
>> >> IPoverIB@ietf.org
>> >> http://www.ietf.org/mailman/listinfo/ipoverib
>> >> 
>> >
>> >
>> >-- 
>> >Vivek Kashyap
>> >Linux Technology Center, IBM
>> >kashyapv@us.ibm.com 
>> >vivk@us.ibm.com 
>> >503 578 3422 (o)
>> >
>> >_______________________________________________
>> >IPoverIB mailing list
>> >IPoverIB@ietf.org
>> >http://www.ietf.org/mailman/listinfo/ipoverib
>> 
>> 
>> _______________________________________________
>> IPoverIB mailing list
>> IPoverIB@ietf.org
>> http://www.ietf.org/mailman/listinfo/ipoverib
>> 


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 16 22:48:08 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA28804
	for <ipoverib-archive@odin.ietf.org>; Wed, 16 Jan 2002 22:48:04 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id WAA02213
	for ipoverib-archive@odin.ietf.org; Wed, 16 Jan 2002 22:48:08 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA02112;
	Wed, 16 Jan 2002 22:43:28 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA02080
	for <ipoverib@optimus.ietf.org>; Wed, 16 Jan 2002 22:43:26 -0500 (EST)
Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA28736
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 22:43:21 -0500 (EST)
Received: from sunmail1.Sun.COM ([129.145.1.2])
	by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id TAA18076;
	Wed, 16 Jan 2002 19:43:24 -0800 (PST)
Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.88.31])
	by sunmail1.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1-Sun.COM.mod.2) with ESMTP id TAA29712;
	Wed, 16 Jan 2002 19:44:37 -0800 (PST)
Received: from taipei (taipei.Eng.Sun.COM [129.146.85.178])
	by jurassic.eng.sun.com (8.12.2+Sun/8.12.2) with SMTP id g0H3hNqT601563;
	Wed, 16 Jan 2002 19:43:23 -0800 (PST)
Message-Id: <200201170343.g0H3hNqT601563@jurassic.eng.sun.com>
Date: Wed, 16 Jan 2002 19:40:52 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
To: adube@windows.microsoft.com
Cc: ipoverib@ietf.org
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: Ql95oLuLOYjbjfbzOvZWLg==
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.6_06 SunOS 5.8 sun4u sparc 
Subject: [Ipoverib] Re: MCGroup Attach Failure --- IPoIB link and multicast draft -
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Hello Aditya,

Do you have any reason to believe this case should be handled
differently by IPoIB stack than other IB multicast applications?

If the join operation is unreliable or intrinsically weak by IBA
design, we can always recommend a backup algorithm. (Any idea why
a join may fail?)

Jerry

>Hello Jerry
>
>I have a question from the Multicast draft that you posted a few days
>ago.
>
>What should a node that wishes to receive IP traffic on a MCast GID do
>if the 'Attach' (SubnAdmSet MCMemberRecord) operation on that IB MCast
>GID fails?
>
>
>Aditya Dube
>
>-----Original Message-----
>From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com] 
>Sent: Monday, January 14, 2002 12:20 PM
>To: ipoverib@ietf.org
>Subject: [Ipoverib] IPoIB link and multicast draft
>
>
>...<deleted>


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 16 22:51:10 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA28843
	for <ipoverib-archive@odin.ietf.org>; Wed, 16 Jan 2002 22:51:09 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id WAA02369
	for ipoverib-archive@odin.ietf.org; Wed, 16 Jan 2002 22:51:14 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA02343;
	Wed, 16 Jan 2002 22:50:12 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA01042
	for <ipoverib@optimus.ietf.org>; Wed, 16 Jan 2002 22:19:58 -0500 (EST)
Received: from mail2.microsoft.com (mail2.microsoft.com [131.107.3.124])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA27334
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 22:19:54 -0500 (EST)
Received: from INET-VRS-02.redmond.corp.microsoft.com ([157.54.8.110]) by mail2.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Wed, 16 Jan 2002 19:19:26 -0800
Received: from 157.54.8.23 by INET-VRS-02.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Wed, 16 Jan 2002 19:19:25 -0800
Received: from red-imc-04.redmond.corp.microsoft.com ([157.54.2.168]) by inet-hub-01.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Wed, 16 Jan 2002 19:19:25 -0800
Received: from win-imc-02.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.84]) by red-imc-04.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Wed, 16 Jan 2002 19:19:25 -0800
Received: from win-msg-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.52]) by win-imc-02.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3588.0);
	 Wed, 16 Jan 2002 19:17:19 -0800
X-MimeOLE: Produced By Microsoft Exchange V6.0.6132.0
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Subject: RE: [Ipoverib] MCGroup Attach Failure --- IPoIB link and multicast draft -
Date: Wed, 16 Jan 2002 19:17:19 -0800
Message-ID: <2E33960095B58E40A4D3345AB9F65EC102450C8E@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
Thread-Topic: [Ipoverib] MCGroup Attach Failure --- IPoIB link and multicast draft -
Thread-Index: AcGeVQ6/C9t+YiKvRVavW22aS12QGwAr29lA
From: "Aditya Dube" <adube@windows.microsoft.com>
To: "Vivek Kashyap" <kashyapv@us.ibm.com>
Cc: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>, <ipoverib@ietf.org>
X-OriginalArrivalTime: 17 Jan 2002 03:17:19.0949 (UTC) FILETIME=[792957D0:01C19F05]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id WAA01043
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

Hello Vivek

Which document is the Architecture draft referring to in the relevant
section (4.2.2.1, IB join of MGIDs by a listener):
           
		"The exact parameters and timers for such retries are
           specified in another document."

Aditya Dube

-----Original Message-----
From: Vivek Kashyap [mailto:kashyapv@us.ibm.com] 
Sent: Tuesday, January 15, 2002 10:16 PM
To: Aditya Dube
Cc: H.K. Jerry Chu; ipoverib@ietf.org
Subject: Re: [Ipoverib] MCGroup Attach Failure --- IPoIB link and
multicast draft -

Aditya,
        The failure cases at the IB level are covered in the
        architecture draft.

Vivek

> 
> Hello Jerry
> 
> I have a question from the Multicast draft that you posted a few days
> ago.
> 
> What should a node that wishes to receive IP traffic on a MCast GID do
> if the 'Attach' (SubnAdmSet MCMemberRecord) operation on that IB MCast
> GID fails?
> 
> 
> Aditya Dube
> 
> -----Original Message-----
> From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com] 
> Sent: Monday, January 14, 2002 12:20 PM
> To: ipoverib@ietf.org
> Subject: [Ipoverib] IPoIB link and multicast draft
> 
> 
> ...<deleted>
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 16 22:58:37 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA28899
	for <ipoverib-archive@odin.ietf.org>; Wed, 16 Jan 2002 22:58:37 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id WAA02556
	for ipoverib-archive@odin.ietf.org; Wed, 16 Jan 2002 22:58:41 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA02529;
	Wed, 16 Jan 2002 22:57:50 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA02498
	for <ipoverib@optimus.ietf.org>; Wed, 16 Jan 2002 22:57:48 -0500 (EST)
Received: from pdxpo.dsl-only.net (pdxpo.dsl-only.net [63.105.16.3])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA28875
	for <ipoverib@ietf.org>; Wed, 16 Jan 2002 22:57:43 -0500 (EST)
Received: from bonfire (unverified [63.105.18.201]) by pdxpo.dsl-only.net
 (Rockliffe SMTPRA 4.5.4) with SMTP id <B0009208438@pdxpo.dsl-only.net> for <ipoverib@ietf.org>;
 Wed, 16 Jan 2002 19:56:49 -0800
From: "Bill Strahm" <bill@strahm.net>
To: <ipoverib@ietf.org>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
Date: Wed, 16 Jan 2002 20:06:06 -0800
Message-ID: <HHEDIJGOEFLPCPNILMPAOEPOCEAA.bill@strahm.net>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0)
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
In-Reply-To: <D9223EB959A5D511A98F00508B68C20C21A84E@ORSMSX108>
Importance: Normal
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Just free form thinking here...

Could we say that GIDs are the equivalent of a Fully Qualified Domain Name
(www.strahm.net for example) and LIDs are the equivalent of IP addresses
(63.105.18.42).  Now in many cases IP addresses are quite stable, and do not
change.  However would you ship an application with an IP address encoded in
it, or with a FQDN.  It would be of course more efficient to use the IP
address and avoid the DNS lookup (that has to go VERY far away, and can take
quite a while for the first SYN message to get out), however if the address
EVER changes, things become hard.

Can anyone tell me why this is a bad analogy ???

Bill

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Larsen, Roy K
Sent: Wednesday, January 16, 2002 6:43 PM
To: 'H.K. Jerry Chu'; arvindm@windows.microsoft.com; kashyapv@us.ibm.com
Cc: ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?


> -----Original Message-----
> From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
> Sent: Wednesday, January 16, 2002 6:13 PM
> To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com
> Cc: ipoverib@ietf.org
> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>
>
> All the proxy ARP usages that I know of use the proxy's mac
> addr in the
> ARP response. I believe the flavor of the proxy ARP you
> alluded below is
> rare, and won't work very well anyway if the link address includes any
> non-persistent component like LID and QPN because it will
> require another
> protocol between the clients and their proxy to constantly update the
> proxy database containing clients mac addrs, which may change quite
> often.
>
> That is also why I wanted to provide an option to not include QPN.
> (But I haven't succeeded w/o requiring SIDR or something equivalent.)
>
> If LID is included as part of link addr and if it ever changes on
> a live node due to subnet merge...etc, the driver must inform its IP
> stack about such change. I believe many stacks don't have such a
> facility. (FOr most IP stacks the assumption is that its own MAC
> addresses just don't change on the fly.) In that case you're totally
> hosed. Can someone comment on how likely a LID assignment gets
> rescinded?

It is possible anytime the link goes down and comes back up again and when
the fabic is reconfigured.  This can happen on reboot or simply
unplugging/plugging in a link.  It is ultimately a function of the SM and
how much trouble the implementation wants to go through to keep LIDs
constant.

But of course, that has no bearing on whether the LID is contained in the
ARP.  You always need the LID.  Unless you want to do a pathrecord lookup
each time your implementation is given a packet to send, you're going to use
a cached LID no matter if it came in the ARP or a pathrecord lookup on the
GID.  If it becomes stale, it may take an ARP cache flush timeout to
recover.

Roy

>
> Jerry
>
> >        A device can always respond for another. The LID of
> the sender
> >        is not the same as the LID for which it has
> responded. It also
> >        allows any network monitoring/debugging device to work
> >        independently. This is the reason why none of the other media
> >        does anything like this. You don't pick ethernet source
> >        address from the frame but take it from the ARP/ND packet.
> >
> >        [ If it was ok we wouldn't need anything except QPN/cap_flags
> >        - get the LID from response, consult SA using LID for GID if
> >        you want that, and you are done. ]
> >
> >Vivek
> >
> >>
> >> Why do we need to include the LID inside the link layer
> address when it
> >> is already available in the SLID field? Previous
> discussions indicate
> >> that the SLID value works for proxy ARP/bridge implementations.
> >>
> >> - Arvind
> >>
> >> -----Original Message-----
> >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> >> Sent: Wednesday, January 16, 2002 10:29 AM
> >> To: kamath@seagullsemi.com
> >> Cc: ipoverib@ietf.org
> >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> >>
> >> Ashwin,
> >> Ashwin,
> >>
> >>         I agree that you get more flexibility if you
> include the LID.
> >>         If we include the GID along with the LID we allow all
> >>         implementations the choice to create different
> solutions. We
> >>         don't tie them down to one thing. Choosing one or the other
> >>         (LID or GID) blocks out various advantages that different
> >>         implementations seek. Hence I suggested that let us include
> >>         both and move on.
> >>
> >> Vivek
> >>
> >> <snip>
> >>
> >> _______________________________________________
> >> IPoverIB mailing list
> >> IPoverIB@ietf.org
> >> http://www.ietf.org/mailman/listinfo/ipoverib
> >>
> >
> >
> >--
> >Vivek Kashyap
> >Linux Technology Center, IBM
> >kashyapv@us.ibm.com
> >vivk@us.ibm.com
> >503 578 3422 (o)
> >
> >_______________________________________________
> >IPoverIB mailing list
> >IPoverIB@ietf.org
> >http://www.ietf.org/mailman/listinfo/ipoverib
>
>
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
>

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 04:30:09 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id EAA10540
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 04:30:04 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id EAA22731
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 04:30:06 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id EAA22639;
	Thu, 17 Jan 2002 04:28:23 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id EAA22609
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 04:28:22 -0500 (EST)
Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id EAA10524
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 04:28:19 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id DAA57550;
	Thu, 17 Jan 2002 03:23:26 -0600
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0H9ROs273258;
	Thu, 17 Jan 2002 04:27:29 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0H9RCV22742;
	Thu, 17 Jan 2002 01:27:12 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0H9RBg12021;
	Thu, 17 Jan 2002 01:27:11 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201170927.g0H9RBg12021@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
To: bill@strahm.net (Bill Strahm)
Date: Thu, 17 Jan 2002 01:27:10 -0800 (PST)
Cc: ipoverib@ietf.org
In-Reply-To: <HHEDIJGOEFLPCPNILMPAOEPOCEAA.bill@strahm.net> from "Bill Strahm" at Jan 16, 2002 07:06:06 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit


> 
> Just free form thinking here...
> 
> Could we say that GIDs are the equivalent of a Fully Qualified Domain Name
> (www.strahm.net for example) and LIDs are the equivalent of IP addresses
> (63.105.18.42).  Now in many cases IP addresses are quite stable, and do not
> change.  However would you ship an application with an IP address encoded in
> it, or with a FQDN.  It would be of course more efficient to use the IP
> address and avoid the DNS lookup (that has to go VERY far away, and can take
> quite a while for the first SYN message to get out), however if the address
> EVER changes, things become hard.
> 
> Can anyone tell me why this is a bad analogy ???

        If it was pure IB then GID to LID would be a similar analogy.

        I'll try taking the your analogy further..a couple of points
        though - the SM corresonds to the zone primary but the nodes
        themselves corresond to both the secondary and the IP nodes
        asking for the resolution. So the analogy is not exactly apt
        but I'll try..

        If the IP changes the DNS must be updated and looked up. In
        this case if LID changes the SM is updated (yes) but it then
        requires that the SM be looked up. There is no backlink here
        to trigger this lookup. This implies that the full lookup all
        the way to the DNS root must be done - if like DNS it is done
        till the local secondary server ie. referring to the local
        cached copy and/or merely re-receiving the peer's GID, the
        communication will fail. You must have seen the hell that
        breaks loose if any of the zone updates doesn't happen due to
        some error and things are left pointing the wrong way.

        Now, this would correspond to the internal LID cache flush -
        since the GID is the same but the LID has changed. There is no
        hook to say, hey!, go do a zone transfer i.e. lookup the SM
        right now. You need a timer.

        Now, you could make this time small - you are going to the SM
        too often because things are not changing that much- make it
        long and LID change is surely going to happen at a critical
        time. Additionally this is another timer per entry for the
        programmers/maintainers/administrators to worry about.

        I have all along been arguing against keeping LID out due to
        this reason too (code is more complex if no LID is another
        reason). It has been my assumption that if GID is used without
        LID then one is forced to go to the SM at every address
        resolution. Every arp broadcast will have the whole subnet
        scurrying off to the SM to ask the requester's LID. Yes, some
        would mark the entry to be completed but not necessarily - a
        thundering herd is likely.

        This implies that if we don't use the LID we MUST ALWAYS go to
        the SM for the LID - at every ARP since we don't know if LID
        has changed. Of course if the GID changed too then the SM has
        to be consulted anyway.

        Also note that if we did have a trap from the SM on every LID
        change it still wouldn't be a good solution - too much
        latency; points of synchrnonisation; asynchronous updates; and
        most of all the SM would end up doing nothing else but issueing
        traps to every node that registered with it. IB cannot mandate
        broadcasts/multicast (though making everyone look is bad too)
        because multicasting is optional.

        In the case of inclusion of the LID - some of the issues are
        the same but a gratuitous ARP solves the problem if the LID
        changed. It is an instant zone transfer. One could claim that
        with GID only, on receiving any gratuitous ARP entry, the node
        will go to the SM but that will require specialised coding to
        recognise and handle gratuitous ARP differently than regular
        updates. Not a good idea.

        Even in the absence of gratuitous ARP the changed LID will be
        known when the ARP is done after a cache expiry if LID is
        included. If LID is not included one will get a GID back. That
        will require the expiry of the GID to LID cache before one
        goes to the SM.

        Inclusion of LID makes things very simple. This is because LID
        is always necessary - can't escape it.

        I also like inclusion of LID because it cuts off 2x latency of
        GID to LID lookup in the SM and 4x the RTT and the realted
        extra interrupts. Fast failover in data centers is necessary.

Vivek



> 
> Bill
> 
> -----Original Message-----
> From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
> Of Larsen, Roy K
> Sent: Wednesday, January 16, 2002 6:43 PM
> To: 'H.K. Jerry Chu'; arvindm@windows.microsoft.com; kashyapv@us.ibm.com
> Cc: ipoverib@ietf.org
> Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
> 
> 
> > -----Original Message-----
> > From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
> > Sent: Wednesday, January 16, 2002 6:13 PM
> > To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com
> > Cc: ipoverib@ietf.org
> > Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> >
> >
> > All the proxy ARP usages that I know of use the proxy's mac
> > addr in the
> > ARP response. I believe the flavor of the proxy ARP you
> > alluded below is
> > rare, and won't work very well anyway if the link address includes any
> > non-persistent component like LID and QPN because it will
> > require another
> > protocol between the clients and their proxy to constantly update the
> > proxy database containing clients mac addrs, which may change quite
> > often.
> >
> > That is also why I wanted to provide an option to not include QPN.
> > (But I haven't succeeded w/o requiring SIDR or something equivalent.)
> >
> > If LID is included as part of link addr and if it ever changes on
> > a live node due to subnet merge...etc, the driver must inform its IP
> > stack about such change. I believe many stacks don't have such a
> > facility. (FOr most IP stacks the assumption is that its own MAC
> > addresses just don't change on the fly.) In that case you're totally
> > hosed. Can someone comment on how likely a LID assignment gets
> > rescinded?
> 
> It is possible anytime the link goes down and comes back up again and when
> the fabic is reconfigured.  This can happen on reboot or simply
> unplugging/plugging in a link.  It is ultimately a function of the SM and
> how much trouble the implementation wants to go through to keep LIDs
> constant.
> 
> But of course, that has no bearing on whether the LID is contained in the
> ARP.  You always need the LID.  Unless you want to do a pathrecord lookup
> each time your implementation is given a packet to send, you're going to use
> a cached LID no matter if it came in the ARP or a pathrecord lookup on the
> GID.  If it becomes stale, it may take an ARP cache flush timeout to
> recover.
> 
> Roy
> 
> >
> > Jerry
> >
> > >        A device can always respond for another. The LID of
> > the sender
> > >        is not the same as the LID for which it has
> > responded. It also
> > >        allows any network monitoring/debugging device to work
> > >        independently. This is the reason why none of the other media
> > >        does anything like this. You don't pick ethernet source
> > >        address from the frame but take it from the ARP/ND packet.
> > >
> > >        [ If it was ok we wouldn't need anything except QPN/cap_flags
> > >        - get the LID from response, consult SA using LID for GID if
> > >        you want that, and you are done. ]
> > >
> > >Vivek
> > >
> > >>
> > >> Why do we need to include the LID inside the link layer
> > address when it
> > >> is already available in the SLID field? Previous
> > discussions indicate
> > >> that the SLID value works for proxy ARP/bridge implementations.
> > >>
> > >> - Arvind
> > >>
> > >> -----Original Message-----
> > >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> > >> Sent: Wednesday, January 16, 2002 10:29 AM
> > >> To: kamath@seagullsemi.com
> > >> Cc: ipoverib@ietf.org
> > >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> > >>
> > >> Ashwin,
> > >> Ashwin,
> > >>
> > >>         I agree that you get more flexibility if you
> > include the LID.
> > >>         If we include the GID along with the LID we allow all
> > >>         implementations the choice to create different
> > solutions. We
> > >>         don't tie them down to one thing. Choosing one or the other
> > >>         (LID or GID) blocks out various advantages that different
> > >>         implementations seek. Hence I suggested that let us include
> > >>         both and move on.
> > >>
> > >> Vivek
> > >>
> > >> <snip>
> > >>
> > >> _______________________________________________
> > >> IPoverIB mailing list
> > >> IPoverIB@ietf.org
> > >> http://www.ietf.org/mailman/listinfo/ipoverib
> > >>
> > >
> > >
> > >--
> > >Vivek Kashyap
> > >Linux Technology Center, IBM
> > >kashyapv@us.ibm.com
> > >vivk@us.ibm.com
> > >503 578 3422 (o)
> > >
> > >_______________________________________________
> > >IPoverIB mailing list
> > >IPoverIB@ietf.org
> > >http://www.ietf.org/mailman/listinfo/ipoverib
> >
> >
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> >
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 09:07:06 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA15762
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 09:07:05 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id JAA03111
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 09:07:05 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id JAA03053;
	Thu, 17 Jan 2002 09:06:07 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id JAA03021
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 09:06:05 -0500 (EST)
Received: from ztxmail04.ztx.compaq.com (ztxmail04.ztx.compaq.com [161.114.1.208])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA15707
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 09:06:02 -0500 (EST)
Received: by ztxmail04.ztx.compaq.com (Postfix, from userid 12345)
	id 7F0C027B9; Thu, 17 Jan 2002 08:05:33 -0600 (CST)
Received: from cceexg11.americas.cpqcorp.net (cceexg11.americas.cpqcorp.net [16.110.250.125])
	by ztxmail04.ztx.compaq.com (Postfix) with ESMTP
	id 71F473A8; Thu, 17 Jan 2002 08:05:33 -0600 (CST)
Received: from cceexc18.americas.cpqcorp.net ([16.110.250.64]) by cceexg11.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966);
	 Thu, 17 Jan 2002 08:05:33 -0600
X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="US-ASCII"
Subject: RE: [Ipoverib] encapsulation issues
Date: Thu, 17 Jan 2002 08:04:59 -0600
Message-ID: <4D027986353D1341ADA4F2A4F8A170760E3172@cceexc18.americas.cpqcorp.net>
Thread-Topic: [Ipoverib] encapsulation issues
Thread-Index: AcGe6YmdBt6yHyc+TqGydktO7A/5pAAcs3Xw
From: "Culley, Paul" <Paul.Culley@compaq.com>
To: "Vivek Kashyap" <kashyapv@us.ibm.com>, <ipoverib@ietf.org>
X-OriginalArrivalTime: 17 Jan 2002 14:05:33.0271 (UTC) FILETIME=[07634E70:01C19F60]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id JAA03022
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

If you insist on including the LID (presumably in the body of the ARP
Response) then I have the following thoughts:

1) Include a means where the ARP Requestor can decide when not to use
that LID.  For example, if IP across multiple IB subnets someday becomes
feasible, any time the subnet prefixes between the ARP Requestor and ARP
Responder are different, the LID (in the body of the response) will be
invalid.  There may also be circumstances where the ARP Responder knows
that the LID is not valid; in that case there should be a defined method
of indicating such.

2) Since the LID alone in the body of the ARP Response is not sufficient
to route packets, also provide the SL.  IB provides NO WAY to select an
SL, based only on the LID, unless the ARP requestor already has that LID
and an SL cached or is willing to look it up with the SA.  Since one of
the reasons for including the LID at all is to avoid this lookup, the
solution is not complete without SL.  [Please don't respond that some
implementation specific mechanism can be chosen to select SL; there is
no way for an endnode implementation to determine the SA policies for SL
usage at this time, so it is not possible for the endnode to reliably
chose an SL without asking the SA (or caching a LID/SL derived from the
SA).

3) A potential solution to the above problems that seems like it would
satisfy everyone would be to use both the SLID and SL from the IB
Header.  Since there may be some circumstances where the ARP Responder
KNOWs that this won't work (odd 3rd party ARP thingies have been
suggested), include a flag that indicates that the SLID/SL is invalid.
The GID is available in this case, or for those devices that want more
than a "best effort" path; it can be used for PathRecord Lookup.  This
will work properly for normal devices in the subnet, IB routers (when
they appear), and Proxy ARP types of routing.

Paul R. Culley
Compaq Sr. Fellow
281-514-5543


-----Original Message-----
From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
Sent: Wednesday, January 16, 2002 5:57 PM
To: ipoverib@ietf.org
Subject: [Ipoverib] encapsulation issues



1.      We have been arguing over the inclusion of LID/GID or both for
        a while now. Examples and advantages of including either have
        been espoused often enough. However, there is an agreement, I
        think, on the architecture not forcing a limited solution on
        any set of aspirations.

        As noted earlier the inclusion of both the identifiers does
        not handicap or restrict any solution. 
	
	Let us include both.


2. 	I believe based on the recent set of mails that the consensus no
        longer favours inclusion of the 'ethertype'. Is there a
        disagreement ?


3. 	As many have noted in the last couple of days that using IB
        over non-UD modes or utilising QPNs mapped to specific
        endpoints is desirable. A complete, interoperable solution has
        been presented in
        draft-kashyap-ipoib-advanced-capabilities-00.txt. 

	Please review and comment. 

Vivek


	

-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Thu Jan 17 11:55:13 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA24142
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 11:55:13 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id LAA11222
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 11:55:16 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id LAA11181;
	Thu, 17 Jan 2002 11:54:06 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id LAA11116
	for <ipoverib@ns.ietf.org>; Thu, 17 Jan 2002 11:54:03 -0500 (EST)
Received: from palrel10.hp.com (palrel10.hp.com [156.153.255.245])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA24089
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 11:53:59 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel10.hp.com (Postfix) with ESMTP id 1F2CC4008E5
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 08:53:32 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id IAA18854
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 08:53:31 -0800 (PST)
Message-Id: <5.1.0.14.2.20020117084042.09df3950@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Thu, 17 Jan 2002 08:52:00 -0800
To: ipoverib@ietf.org
From: Michael Krause <krause@cup.hp.com>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
In-Reply-To: <01a601c19ef8$2837ad20$0100007f@vaio505he>
References: <200201162009.g0GK9Gk26559@eng4.beaverton.ibm.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

At 03:41 AM 1/17/2002 +0200, Yaron Haviv wrote:
>Vivek the key for including the LID is that people will have a primary (GID)
>address that determine the route

A GID does not determine the route used between any two endnodes - it only 
identifies the port.  Within an IB subnet, the LID is used to identify the 
path taken this is why we have the base LID and LMC concept in order to 
support multi-path.  Applications must supply an address vector which is 
composed of several elements which are then translated into a specific 
tuple used by the fabric for packet switching, i.e. <DLID, SL>.

The GID prefix is used by the IB routing protocol as one of the inputs 
required to identify a path to take as it crosses one or more IB subnets 
between the two endnodes.  The router uses some or all of these same inputs 
to determine how to map a given DGID to a local <DLID, SL> for final 
delivery to the destination endnode port.

>, to avoid interoperability issues
>(I'm sure this was also the consensus in LWG befor we went to IETF)
>
>We need to maintain the tables by the GID, however I support taking the LID
>to return the answer to the caller (avoiding the SM as the main reason)
>I'm not sure that we could take the LID from the packet since someone might
>Forward it somehow (test tools, ARP server or something like that)
>And we don't want a broken spec

If you cannot trust what is available from the IB packet headers for the 
above reasons, you will not be able to use something included within the 
link address either unless you take the step to consult the SM/SA for the 
appropriate value which was the step that was supposed to be eliminated by 
the proposals put forth to date.

>No one mentioned in the arguments the additional latency caused by going 
>to The SM (for the LID) As a central resource that probably has queuing it 
>might take time Not to mention the latency in IP span IB case (needing to 
>resolve all the Path first)

The latency can be eliminated depending upon how one distributes the GID / 
<LID, SL> mapping data and whether one makes use of the IB Notify 
mechanisms to note fabric event changes.   With some minor improvements, 
many implementations will only traverse to the SA once upon initial contact 
of another endnode and then only refresh upon a notification of a fabric 
event.  I fail to see the validity of the arguments presented to date 
concerning SA performance and their negative impact of fabric convergence 
upon a failure / merge / etc. event.  I also will contend that fabric 
events do not happen that often and with again some intelligence in terms 
of use of existing IB management definitions, the scalability / performance 
/ convergence issues can be dealt with quite well.

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Thu Jan 17 11:55:23 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA24162
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 11:55:22 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id LAA11237
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 11:55:25 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id LAA11178;
	Thu, 17 Jan 2002 11:54:06 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id LAA11111
	for <ipoverib@ns.ietf.org>; Thu, 17 Jan 2002 11:54:03 -0500 (EST)
Received: from palrel10.hp.com (palrel10.hp.com [156.153.255.245])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA24088
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 11:53:59 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel10.hp.com (Postfix) with ESMTP id D3E9940084A
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 08:53:31 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id IAA18851
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 08:53:31 -0800 (PST)
Message-Id: <5.1.0.14.2.20020117082959.09dec908@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Thu, 17 Jan 2002 08:32:42 -0800
To: <ipoverib@ietf.org>
From: Michael Krause <krause@cup.hp.com>
Subject: RE: [Ipoverib] encapsulation issues
In-Reply-To: <4D027986353D1341ADA4F2A4F8A170760E3172@cceexc18.americas.c
 pqcorp.net>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

At 08:04 AM 1/17/2002 -0600, Culley, Paul wrote:
>If you insist on including the LID (presumably in the body of the ARP
>Response) then I have the following thoughts:
>
>1) Include a means where the ARP Requestor can decide when not to use
>that LID.  For example, if IP across multiple IB subnets someday becomes
>feasible, any time the subnet prefixes between the ARP Requestor and ARP
>Responder are different, the LID (in the body of the response) will be
>invalid.  There may also be circumstances where the ARP Responder knows
>that the LID is not valid; in that case there should be a defined method
>of indicating such.
>
>2) Since the LID alone in the body of the ARP Response is not sufficient
>to route packets, also provide the SL.  IB provides NO WAY to select an
>SL, based only on the LID, unless the ARP requestor already has that LID
>and an SL cached or is willing to look it up with the SA.  Since one of
>the reasons for including the LID at all is to avoid this lookup, the
>solution is not complete without SL.  [Please don't respond that some
>implementation specific mechanism can be chosen to select SL; there is
>no way for an endnode implementation to determine the SA policies for SL
>usage at this time, so it is not possible for the endnode to reliably
>chose an SL without asking the SA (or caching a LID/SL derived from the
>SA).
>
>3) A potential solution to the above problems that seems like it would
>satisfy everyone would be to use both the SLID and SL from the IB
>Header.  Since there may be some circumstances where the ARP Responder
>KNOWs that this won't work (odd 3rd party ARP thingies have been
>suggested), include a flag that indicates that the SLID/SL is invalid.
>The GID is available in this case, or for those devices that want more
>than a "best effort" path; it can be used for PathRecord Lookup.  This
>will work properly for normal devices in the subnet, IB routers (when
>they appear), and Proxy ARP types of routing.

If the group can only reach consensus on this issue through the addition of 
these fields , then the above list seems like a reasonable starting point 
for such compromise.

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Thu Jan 17 12:00:38 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA24335
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 12:00:31 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id MAA12037
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 12:00:34 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id LAA11327;
	Thu, 17 Jan 2002 11:59:30 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id LAA11293
	for <ipoverib@ns.ietf.org>; Thu, 17 Jan 2002 11:59:28 -0500 (EST)
Received: from mail5.microsoft.com (mail5.microsoft.com [131.107.3.121])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA24263
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 11:59:24 -0500 (EST)
Received: from inet-vrs-05.redmond.corp.microsoft.com ([157.54.6.148]) by mail5.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Thu, 17 Jan 2002 08:58:54 -0800
Received: from 157.54.5.25 by inet-vrs-05.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Thu, 17 Jan 2002 08:58:53 -0800
Received: from red-imc-02.redmond.corp.microsoft.com ([157.54.9.107]) by inet-hub-03.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Thu, 17 Jan 2002 08:58:53 -0800
Received: from win-imc-02.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.84]) by red-imc-02.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Thu, 17 Jan 2002 08:58:52 -0800
Received: from win-msg-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.52]) by win-imc-02.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3588.0);
	 Thu, 17 Jan 2002 08:56:46 -0800
X-MimeOLE: Produced By Microsoft Exchange V6.0.6132.0
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Subject: RE: [Ipoverib] encapsulation issues
Date: Thu, 17 Jan 2002 08:56:46 -0800
Message-ID: <2E33960095B58E40A4D3345AB9F65EC10353E256@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
Thread-Topic: [Ipoverib] encapsulation issues
Thread-Index: AcGfd0pmKnYun7znRmWfRxlAaWg7mQAAJASQ
From: "Arvind Murching" <arvindm@windows.microsoft.com>
To: "Michael Krause" <krause@cup.hp.com>, <ipoverib@ietf.org>
X-OriginalArrivalTime: 17 Jan 2002 16:56:46.0752 (UTC) FILETIME=[F2DC1A00:01C19F77]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id LAA11294
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

Proposal #3 below looks attractive.

Thanks
Arvind

-----Original Message-----
From: Michael Krause [mailto:krause@cup.hp.com] 
Sent: Thursday, January 17, 2002 8:33 AM
To: ipoverib@ietf.org
Subject: RE: [Ipoverib] encapsulation issues

At 08:04 AM 1/17/2002 -0600, Culley, Paul wrote:
>If you insist on including the LID (presumably in the body of the ARP
>Response) then I have the following thoughts:
>
>1) Include a means where the ARP Requestor can decide when not to use
>that LID.  For example, if IP across multiple IB subnets someday
becomes
>feasible, any time the subnet prefixes between the ARP Requestor and
ARP
>Responder are different, the LID (in the body of the response) will be
>invalid.  There may also be circumstances where the ARP Responder knows
>that the LID is not valid; in that case there should be a defined
method
>of indicating such.
>
>2) Since the LID alone in the body of the ARP Response is not
sufficient
>to route packets, also provide the SL.  IB provides NO WAY to select an
>SL, based only on the LID, unless the ARP requestor already has that
LID
>and an SL cached or is willing to look it up with the SA.  Since one of
>the reasons for including the LID at all is to avoid this lookup, the
>solution is not complete without SL.  [Please don't respond that some
>implementation specific mechanism can be chosen to select SL; there is
>no way for an endnode implementation to determine the SA policies for
SL
>usage at this time, so it is not possible for the endnode to reliably
>chose an SL without asking the SA (or caching a LID/SL derived from the
>SA).
>
>3) A potential solution to the above problems that seems like it would
>satisfy everyone would be to use both the SLID and SL from the IB
>Header.  Since there may be some circumstances where the ARP Responder
>KNOWs that this won't work (odd 3rd party ARP thingies have been
>suggested), include a flag that indicates that the SLID/SL is invalid.
>The GID is available in this case, or for those devices that want more
>than a "best effort" path; it can be used for PathRecord Lookup.  This
>will work properly for normal devices in the subnet, IB routers (when
>they appear), and Proxy ARP types of routing.

If the group can only reach consensus on this issue through the addition
of 
these fields , then the above list seems like a reasonable starting
point 
for such compromise.

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Thu Jan 17 12:12:28 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA26443
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 12:12:28 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id MAA13005
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 12:12:31 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA12833;
	Thu, 17 Jan 2002 12:09:52 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA12798
	for <ipoverib@ns.ietf.org>; Thu, 17 Jan 2002 12:09:49 -0500 (EST)
Received: from pdxpo.dsl-only.net (pdxpo.dsl-only.net [63.105.16.3])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA24724
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 12:09:46 -0500 (EST)
Received: from bonfire (unverified [63.105.18.201]) by pdxpo.dsl-only.net
 (Rockliffe SMTPRA 4.5.4) with SMTP id <B0009223454@pdxpo.dsl-only.net> for <ipoverib@ietf.org>;
 Thu, 17 Jan 2002 09:08:50 -0800
From: "Bill Strahm" <bill@strahm.net>
To: <ipoverib@ietf.org>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
Date: Thu, 17 Jan 2002 09:18:05 -0800
Message-ID: <HHEDIJGOEFLPCPNILMPAEEAFCFAA.bill@strahm.net>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0)
Importance: Normal
In-Reply-To: <200201170927.g0H9RBg12021@eng4.beaverton.ibm.com>
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Not sure I agree with a few things here...

The SM IS the ROOT DNS server, because it is the ultimate authority on what
is where in the fabric.  Is there any doubt to that.  Also the end nodes
have nothing to do with resolution, I do not believe there is any IB
mechanism to go to the end node and ask it for a Path Record is there ?

You are correct that if the LID changes there is no back path to tell
everyone that the LID has changed, this is just like DNS so the analogy
holds still.  Since the local SM is the DNS Root going "All the way to the
root" is not a problem.

You need a timer just like in DNS, so the analogy still holds.  The timer
works at the same level as the ARP timer.  When you ARP, you go get the
answer from the SM... Until then you can deal with LID changes because
either a) your communication fails - you have this problem in either
scenario b) you notice a LID change on an incoming packet so update your ARP
table

No, only the responder needs to reply to an ARP broadcast, so therefor only
one entry.  Now other nodes in the network may decide to go to the SM to
cache that particular entry as well, but then you remove latencies when you
want to talk to that node anyway... I would assume that any decent SM could
handle the traffic from 65,000 nodes asking it to do a similar lookup at the
same time...  This of course is a worst case scenario.

The problem I have is that unless you have manually configured the LID-SL
mapping for your subnet, you could deadlock the fabric this way.  Now would
you want to manually configure LID-SL lookup for those same 65000 nodes ???

Bill

-----Original Message-----
From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
Sent: Thursday, January 17, 2002 1:27 AM
To: Bill Strahm
Cc: ipoverib@ietf.org
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?



>
> Just free form thinking here...
>
> Could we say that GIDs are the equivalent of a Fully Qualified Domain Name
> (www.strahm.net for example) and LIDs are the equivalent of IP addresses
> (63.105.18.42).  Now in many cases IP addresses are quite stable, and do
not
> change.  However would you ship an application with an IP address encoded
in
> it, or with a FQDN.  It would be of course more efficient to use the IP
> address and avoid the DNS lookup (that has to go VERY far away, and can
take
> quite a while for the first SYN message to get out), however if the
address
> EVER changes, things become hard.
>
> Can anyone tell me why this is a bad analogy ???

        If it was pure IB then GID to LID would be a similar analogy.

        I'll try taking the your analogy further..a couple of points
        though - the SM corresonds to the zone primary but the nodes
        themselves corresond to both the secondary and the IP nodes
        asking for the resolution. So the analogy is not exactly apt
        but I'll try..

        If the IP changes the DNS must be updated and looked up. In
        this case if LID changes the SM is updated (yes) but it then
        requires that the SM be looked up. There is no backlink here
        to trigger this lookup. This implies that the full lookup all
        the way to the DNS root must be done - if like DNS it is done
        till the local secondary server ie. referring to the local
        cached copy and/or merely re-receiving the peer's GID, the
        communication will fail. You must have seen the hell that
        breaks loose if any of the zone updates doesn't happen due to
        some error and things are left pointing the wrong way.

        Now, this would correspond to the internal LID cache flush -
        since the GID is the same but the LID has changed. There is no
        hook to say, hey!, go do a zone transfer i.e. lookup the SM
        right now. You need a timer.

        Now, you could make this time small - you are going to the SM
        too often because things are not changing that much- make it
        long and LID change is surely going to happen at a critical
        time. Additionally this is another timer per entry for the
        programmers/maintainers/administrators to worry about.

        I have all along been arguing against keeping LID out due to
        this reason too (code is more complex if no LID is another
        reason). It has been my assumption that if GID is used without
        LID then one is forced to go to the SM at every address
        resolution. Every arp broadcast will have the whole subnet
        scurrying off to the SM to ask the requester's LID. Yes, some
        would mark the entry to be completed but not necessarily - a
        thundering herd is likely.

        This implies that if we don't use the LID we MUST ALWAYS go to
        the SM for the LID - at every ARP since we don't know if LID
        has changed. Of course if the GID changed too then the SM has
        to be consulted anyway.

        Also note that if we did have a trap from the SM on every LID
        change it still wouldn't be a good solution - too much
        latency; points of synchrnonisation; asynchronous updates; and
        most of all the SM would end up doing nothing else but issueing
        traps to every node that registered with it. IB cannot mandate
        broadcasts/multicast (though making everyone look is bad too)
        because multicasting is optional.

        In the case of inclusion of the LID - some of the issues are
        the same but a gratuitous ARP solves the problem if the LID
        changed. It is an instant zone transfer. One could claim that
        with GID only, on receiving any gratuitous ARP entry, the node
        will go to the SM but that will require specialised coding to
        recognise and handle gratuitous ARP differently than regular
        updates. Not a good idea.

        Even in the absence of gratuitous ARP the changed LID will be
        known when the ARP is done after a cache expiry if LID is
        included. If LID is not included one will get a GID back. That
        will require the expiry of the GID to LID cache before one
        goes to the SM.

        Inclusion of LID makes things very simple. This is because LID
        is always necessary - can't escape it.

        I also like inclusion of LID because it cuts off 2x latency of
        GID to LID lookup in the SM and 4x the RTT and the realted
        extra interrupts. Fast failover in data centers is necessary.

Vivek



>
> Bill
>
> -----Original Message-----
> From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
> Of Larsen, Roy K
> Sent: Wednesday, January 16, 2002 6:43 PM
> To: 'H.K. Jerry Chu'; arvindm@windows.microsoft.com; kashyapv@us.ibm.com
> Cc: ipoverib@ietf.org
> Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>
>
> > -----Original Message-----
> > From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
> > Sent: Wednesday, January 16, 2002 6:13 PM
> > To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com
> > Cc: ipoverib@ietf.org
> > Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> >
> >
> > All the proxy ARP usages that I know of use the proxy's mac
> > addr in the
> > ARP response. I believe the flavor of the proxy ARP you
> > alluded below is
> > rare, and won't work very well anyway if the link address includes any
> > non-persistent component like LID and QPN because it will
> > require another
> > protocol between the clients and their proxy to constantly update the
> > proxy database containing clients mac addrs, which may change quite
> > often.
> >
> > That is also why I wanted to provide an option to not include QPN.
> > (But I haven't succeeded w/o requiring SIDR or something equivalent.)
> >
> > If LID is included as part of link addr and if it ever changes on
> > a live node due to subnet merge...etc, the driver must inform its IP
> > stack about such change. I believe many stacks don't have such a
> > facility. (FOr most IP stacks the assumption is that its own MAC
> > addresses just don't change on the fly.) In that case you're totally
> > hosed. Can someone comment on how likely a LID assignment gets
> > rescinded?
>
> It is possible anytime the link goes down and comes back up again and when
> the fabic is reconfigured.  This can happen on reboot or simply
> unplugging/plugging in a link.  It is ultimately a function of the SM and
> how much trouble the implementation wants to go through to keep LIDs
> constant.
>
> But of course, that has no bearing on whether the LID is contained in the
> ARP.  You always need the LID.  Unless you want to do a pathrecord lookup
> each time your implementation is given a packet to send, you're going to
use
> a cached LID no matter if it came in the ARP or a pathrecord lookup on the
> GID.  If it becomes stale, it may take an ARP cache flush timeout to
> recover.
>
> Roy
>
> >
> > Jerry
> >
> > >        A device can always respond for another. The LID of
> > the sender
> > >        is not the same as the LID for which it has
> > responded. It also
> > >        allows any network monitoring/debugging device to work
> > >        independently. This is the reason why none of the other media
> > >        does anything like this. You don't pick ethernet source
> > >        address from the frame but take it from the ARP/ND packet.
> > >
> > >        [ If it was ok we wouldn't need anything except QPN/cap_flags
> > >        - get the LID from response, consult SA using LID for GID if
> > >        you want that, and you are done. ]
> > >
> > >Vivek
> > >
> > >>
> > >> Why do we need to include the LID inside the link layer
> > address when it
> > >> is already available in the SLID field? Previous
> > discussions indicate
> > >> that the SLID value works for proxy ARP/bridge implementations.
> > >>
> > >> - Arvind
> > >>
> > >> -----Original Message-----
> > >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> > >> Sent: Wednesday, January 16, 2002 10:29 AM
> > >> To: kamath@seagullsemi.com
> > >> Cc: ipoverib@ietf.org
> > >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> > >>
> > >> Ashwin,
> > >> Ashwin,
> > >>
> > >>         I agree that you get more flexibility if you
> > include the LID.
> > >>         If we include the GID along with the LID we allow all
> > >>         implementations the choice to create different
> > solutions. We
> > >>         don't tie them down to one thing. Choosing one or the other
> > >>         (LID or GID) blocks out various advantages that different
> > >>         implementations seek. Hence I suggested that let us include
> > >>         both and move on.
> > >>
> > >> Vivek
> > >>
> > >> <snip>
> > >>
> > >> _______________________________________________
> > >> IPoverIB mailing list
> > >> IPoverIB@ietf.org
> > >> http://www.ietf.org/mailman/listinfo/ipoverib
> > >>
> > >
> > >
> > >--
> > >Vivek Kashyap
> > >Linux Technology Center, IBM
> > >kashyapv@us.ibm.com
> > >vivk@us.ibm.com
> > >503 578 3422 (o)
> > >
> > >_______________________________________________
> > >IPoverIB mailing list
> > >IPoverIB@ietf.org
> > >http://www.ietf.org/mailman/listinfo/ipoverib
> >
> >
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> >
>
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
>
>
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
>


--
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com
vivk@us.ibm.com
503 578 3422 (o)


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 12:59:09 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA28376
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 12:59:09 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id MAA15337
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 12:59:10 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA15255;
	Thu, 17 Jan 2002 12:56:52 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA15223
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 12:56:50 -0500 (EST)
Received: from thalia.fm.intel.com (fmfdns02.fm.intel.com [132.233.247.11])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA28331
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 12:56:43 -0500 (EST)
Received: from fmsmsxvs041.fm.intel.com (fmsmsxv041-1.fm.intel.com [132.233.48.109])
	by thalia.fm.intel.com (8.9.1a+p1/8.9.1/d: relay.m4,v 1.48 2001/12/13 16:27:50 root Exp $) with SMTP id RAA09377
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 17:56:42 GMT
Received: from fmsmsx28.fm.intel.com ([132.233.42.28])
 by fmsmsxvs041.fm.intel.com (NAVGW 2.5.1.16) with SMTP id M2002011709574013591
 ; Thu, 17 Jan 2002 09:57:40 -0800
Received: by fmsmsx28.fm.intel.com with Internet Mail Service (5.5.2653.19)
	id <C02151R4>; Thu, 17 Jan 2002 09:56:42 -0800
Message-ID: <D9223EB959A5D511A98F00508B68C20C21A84F@ORSMSX108>
From: "Larsen, Roy K" <roy.k.larsen@intel.com>
To: "'Bill Strahm'" <bill@strahm.net>, ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
Date: Thu, 17 Jan 2002 09:56:37 -0800
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="ISO-8859-1"
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Bill's last paragraph gets to the crux of the issue on pathrecord looks for
ARP.  There seems to be a camp that will deploy their solution in a
controlled environment where they can dictate the fabric topology and it's
configuration.  A cluster product would fit this model.  There is another
camp that needs to supply a generalized IPoverIB solution that will work in
any fabric topology and configuration such as a very large data center with
a mixture of vendor products.

I believe both of these models are valid.  For the fixed cluster type of
environment, I believe it is reasonable to add things like the LID to the
ARP if that will allow foregoing the pathrecord lookup.  I say this with the
proviso that there are no other constraints on ARP contents and semantics
that would prevent those who simply want to use the GID to consult the SM
for correct address vector information.

If we can agree that these models are equally valid, perhaps we can spend
less bandwidth trying to sway others to a model that will not meet their
market requirements and more on finalizing our work.

Roy

-----Original Message-----
From: Bill Strahm [mailto:bill@strahm.net]
Sent: Thursday, January 17, 2002 9:18 AM
To: ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?


Not sure I agree with a few things here...

The SM IS the ROOT DNS server, because it is the ultimate authority on what
is where in the fabric.  Is there any doubt to that.  Also the end nodes
have nothing to do with resolution, I do not believe there is any IB
mechanism to go to the end node and ask it for a Path Record is there ?

You are correct that if the LID changes there is no back path to tell
everyone that the LID has changed, this is just like DNS so the analogy
holds still.  Since the local SM is the DNS Root going "All the way to the
root" is not a problem.

You need a timer just like in DNS, so the analogy still holds.  The timer
works at the same level as the ARP timer.  When you ARP, you go get the
answer from the SM... Until then you can deal with LID changes because
either a) your communication fails - you have this problem in either
scenario b) you notice a LID change on an incoming packet so update your ARP
table

No, only the responder needs to reply to an ARP broadcast, so therefor only
one entry.  Now other nodes in the network may decide to go to the SM to
cache that particular entry as well, but then you remove latencies when you
want to talk to that node anyway... I would assume that any decent SM could
handle the traffic from 65,000 nodes asking it to do a similar lookup at the
same time...  This of course is a worst case scenario.

The problem I have is that unless you have manually configured the LID-SL
mapping for your subnet, you could deadlock the fabric this way.  Now would
you want to manually configure LID-SL lookup for those same 65000 nodes ???

Bill

-----Original Message-----
From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
Sent: Thursday, January 17, 2002 1:27 AM
To: Bill Strahm
Cc: ipoverib@ietf.org
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?



>
> Just free form thinking here...
>
> Could we say that GIDs are the equivalent of a Fully Qualified Domain Name
> (www.strahm.net for example) and LIDs are the equivalent of IP addresses
> (63.105.18.42).  Now in many cases IP addresses are quite stable, and do
not
> change.  However would you ship an application with an IP address encoded
in
> it, or with a FQDN.  It would be of course more efficient to use the IP
> address and avoid the DNS lookup (that has to go VERY far away, and can
take
> quite a while for the first SYN message to get out), however if the
address
> EVER changes, things become hard.
>
> Can anyone tell me why this is a bad analogy ???

        If it was pure IB then GID to LID would be a similar analogy.

        I'll try taking the your analogy further..a couple of points
        though - the SM corresonds to the zone primary but the nodes
        themselves corresond to both the secondary and the IP nodes
        asking for the resolution. So the analogy is not exactly apt
        but I'll try..

        If the IP changes the DNS must be updated and looked up. In
        this case if LID changes the SM is updated (yes) but it then
        requires that the SM be looked up. There is no backlink here
        to trigger this lookup. This implies that the full lookup all
        the way to the DNS root must be done - if like DNS it is done
        till the local secondary server ie. referring to the local
        cached copy and/or merely re-receiving the peer's GID, the
        communication will fail. You must have seen the hell that
        breaks loose if any of the zone updates doesn't happen due to
        some error and things are left pointing the wrong way.

        Now, this would correspond to the internal LID cache flush -
        since the GID is the same but the LID has changed. There is no
        hook to say, hey!, go do a zone transfer i.e. lookup the SM
        right now. You need a timer.

        Now, you could make this time small - you are going to the SM
        too often because things are not changing that much- make it
        long and LID change is surely going to happen at a critical
        time. Additionally this is another timer per entry for the
        programmers/maintainers/administrators to worry about.

        I have all along been arguing against keeping LID out due to
        this reason too (code is more complex if no LID is another
        reason). It has been my assumption that if GID is used without
        LID then one is forced to go to the SM at every address
        resolution. Every arp broadcast will have the whole subnet
        scurrying off to the SM to ask the requester's LID. Yes, some
        would mark the entry to be completed but not necessarily - a
        thundering herd is likely.

        This implies that if we don't use the LID we MUST ALWAYS go to
        the SM for the LID - at every ARP since we don't know if LID
        has changed. Of course if the GID changed too then the SM has
        to be consulted anyway.

        Also note that if we did have a trap from the SM on every LID
        change it still wouldn't be a good solution - too much
        latency; points of synchrnonisation; asynchronous updates; and
        most of all the SM would end up doing nothing else but issueing
        traps to every node that registered with it. IB cannot mandate
        broadcasts/multicast (though making everyone look is bad too)
        because multicasting is optional.

        In the case of inclusion of the LID - some of the issues are
        the same but a gratuitous ARP solves the problem if the LID
        changed. It is an instant zone transfer. One could claim that
        with GID only, on receiving any gratuitous ARP entry, the node
        will go to the SM but that will require specialised coding to
        recognise and handle gratuitous ARP differently than regular
        updates. Not a good idea.

        Even in the absence of gratuitous ARP the changed LID will be
        known when the ARP is done after a cache expiry if LID is
        included. If LID is not included one will get a GID back. That
        will require the expiry of the GID to LID cache before one
        goes to the SM.

        Inclusion of LID makes things very simple. This is because LID
        is always necessary - can't escape it.

        I also like inclusion of LID because it cuts off 2x latency of
        GID to LID lookup in the SM and 4x the RTT and the realted
        extra interrupts. Fast failover in data centers is necessary.

Vivek



>
> Bill
>
> -----Original Message-----
> From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
> Of Larsen, Roy K
> Sent: Wednesday, January 16, 2002 6:43 PM
> To: 'H.K. Jerry Chu'; arvindm@windows.microsoft.com; kashyapv@us.ibm.com
> Cc: ipoverib@ietf.org
> Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>
>
> > -----Original Message-----
> > From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
> > Sent: Wednesday, January 16, 2002 6:13 PM
> > To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com
> > Cc: ipoverib@ietf.org
> > Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> >
> >
> > All the proxy ARP usages that I know of use the proxy's mac
> > addr in the
> > ARP response. I believe the flavor of the proxy ARP you
> > alluded below is
> > rare, and won't work very well anyway if the link address includes any
> > non-persistent component like LID and QPN because it will
> > require another
> > protocol between the clients and their proxy to constantly update the
> > proxy database containing clients mac addrs, which may change quite
> > often.
> >
> > That is also why I wanted to provide an option to not include QPN.
> > (But I haven't succeeded w/o requiring SIDR or something equivalent.)
> >
> > If LID is included as part of link addr and if it ever changes on
> > a live node due to subnet merge...etc, the driver must inform its IP
> > stack about such change. I believe many stacks don't have such a
> > facility. (FOr most IP stacks the assumption is that its own MAC
> > addresses just don't change on the fly.) In that case you're totally
> > hosed. Can someone comment on how likely a LID assignment gets
> > rescinded?
>
> It is possible anytime the link goes down and comes back up again and when
> the fabic is reconfigured.  This can happen on reboot or simply
> unplugging/plugging in a link.  It is ultimately a function of the SM and
> how much trouble the implementation wants to go through to keep LIDs
> constant.
>
> But of course, that has no bearing on whether the LID is contained in the
> ARP.  You always need the LID.  Unless you want to do a pathrecord lookup
> each time your implementation is given a packet to send, you're going to
use
> a cached LID no matter if it came in the ARP or a pathrecord lookup on the
> GID.  If it becomes stale, it may take an ARP cache flush timeout to
> recover.
>
> Roy
>
> >
> > Jerry
> >
> > >        A device can always respond for another. The LID of
> > the sender
> > >        is not the same as the LID for which it has
> > responded. It also
> > >        allows any network monitoring/debugging device to work
> > >        independently. This is the reason why none of the other media
> > >        does anything like this. You don't pick ethernet source
> > >        address from the frame but take it from the ARP/ND packet.
> > >
> > >        [ If it was ok we wouldn't need anything except QPN/cap_flags
> > >        - get the LID from response, consult SA using LID for GID if
> > >        you want that, and you are done. ]
> > >
> > >Vivek
> > >
> > >>
> > >> Why do we need to include the LID inside the link layer
> > address when it
> > >> is already available in the SLID field? Previous
> > discussions indicate
> > >> that the SLID value works for proxy ARP/bridge implementations.
> > >>
> > >> - Arvind
> > >>
> > >> -----Original Message-----
> > >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> > >> Sent: Wednesday, January 16, 2002 10:29 AM
> > >> To: kamath@seagullsemi.com
> > >> Cc: ipoverib@ietf.org
> > >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> > >>
> > >> Ashwin,
> > >> Ashwin,
> > >>
> > >>         I agree that you get more flexibility if you
> > include the LID.
> > >>         If we include the GID along with the LID we allow all
> > >>         implementations the choice to create different
> > solutions. We
> > >>         don't tie them down to one thing. Choosing one or the other
> > >>         (LID or GID) blocks out various advantages that different
> > >>         implementations seek. Hence I suggested that let us include
> > >>         both and move on.
> > >>
> > >> Vivek
> > >>
> > >> <snip>
> > >>
> > >> _______________________________________________
> > >> IPoverIB mailing list
> > >> IPoverIB@ietf.org
> > >> http://www.ietf.org/mailman/listinfo/ipoverib
> > >>
> > >
> > >
> > >--
> > >Vivek Kashyap
> > >Linux Technology Center, IBM
> > >kashyapv@us.ibm.com
> > >vivk@us.ibm.com
> > >503 578 3422 (o)
> > >
> > >_______________________________________________
> > >IPoverIB mailing list
> > >IPoverIB@ietf.org
> > >http://www.ietf.org/mailman/listinfo/ipoverib
> >
> >
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> >
>
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
>
>
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
>


--
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com
vivk@us.ibm.com
503 578 3422 (o)


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 13:11:26 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA28786
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 13:11:26 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id NAA16284
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 13:11:27 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA16201;
	Thu, 17 Jan 2002 13:09:54 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA16168
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 13:09:52 -0500 (EST)
Received: from mail1.microsoft.com (mail1.microsoft.com [131.107.3.125])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA28740
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 13:09:49 -0500 (EST)
Received: from inet-vrs-01.redmond.corp.microsoft.com ([157.54.8.27]) by mail1.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Thu, 17 Jan 2002 10:09:14 -0800
Received: from 157.54.6.197 by inet-vrs-01.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Thu, 17 Jan 2002 10:09:13 -0800
Received: from red-msg-02.redmond.corp.microsoft.com ([157.54.12.70]) by inet-hub-06.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Thu, 17 Jan 2002 10:09:14 -0800
x-mimeole: Produced By Microsoft Exchange V6.0.5762.3
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
Date: Thu, 17 Jan 2002 10:09:13 -0800
Message-ID: <24A715275661C8428C00432EFCA5CB7C05E3C1AC@red-msg-02.redmond.corp.microsoft.com>
Thread-Topic: [Ipoverib] What's inside an IPoIB link layer address?
Thread-Index: AcGexGlB1ODvHK6MQMCjZfne/vqGqQAlSR6w
From: "Jim Pinkerton" <jpink@microsoft.com>
To: "Vivek Kashyap" <kashyapv@us.ibm.com>,
        "Arvind Murching" <ARVINDM@windows.microsoft.com>
Cc: <ipoverib@ietf.org>
X-OriginalArrivalTime: 17 Jan 2002 18:09:14.0397 (UTC) FILETIME=[124214D0:01C19F82]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id NAA16170
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit


Vivek,

Using the source address for the Spanning Tree algorithm for bridging
(not routing) is required on Ethernet. Thus on Ethernet at least, the
source address is *always* the same as the true source - otherwise
spanning tree doesn't work. Technically you could make the argument that
the true restriction is that the side of the bridge that the spoofer
sits on just has to be the same as the spoofee - but no-one does it that
way that I'm aware of. KISS rules(keep it simple stupid). 

In fact, if you allow what you are suggesting it is technically a
security hole. That's why on some networks even gratuitous ARP replies
are intentionally discarded. If you only allow ARP replies to ARP
requests, then if you have a spoofing malicious user out there you'll
get two replies (and thus detect the spoofer and signal a problem to the
network admin).


Jim






> -----Original Message-----
> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> Sent: Wednesday, January 16, 2002 11:31 AM
> To: Arvind Murching
> Cc: Vivek Kashyap; ipoverib@ietf.org
> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> 
>         A device can always respond for another. The LID of the sender
>         is not the same as the LID for which it has responded. It also
>         allows any network monitoring/debugging device to work
>         independently. This is the reason why none of the other media
>         does anything like this. You don't pick ethernet source
>         address from the frame but take it from the ARP/ND packet.
> 
>         [ If it was ok we wouldn't need anything except QPN/cap_flags
>         - get the LID from response, consult SA using LID for GID if
>         you want that, and you are done. ]
> 
> Vivek
> 
> >
> > Why do we need to include the LID inside the link layer address when
it
> > is already available in the SLID field? Previous discussions
indicate
> > that the SLID value works for proxy ARP/bridge implementations.
> >
> > - Arvind
> >
> > -----Original Message-----
> > From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> > Sent: Wednesday, January 16, 2002 10:29 AM
> > To: kamath@seagullsemi.com
> > Cc: ipoverib@ietf.org
> > Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> >
> > Ashwin,
> > Ashwin,
> >
> >         I agree that you get more flexibility if you include the
LID.
> >         If we include the GID along with the LID we allow all
> >         implementations the choice to create different solutions. We
> >         don't tie them down to one thing. Choosing one or the other
> >         (LID or GID) blocks out various advantages that different
> >         implementations seek. Hence I suggested that let us include
> >         both and move on.
> >
> > Vivek
> >
> > <snip>
> >
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> >
> 
> 
> --
> Vivek Kashyap
> Linux Technology Center, IBM
> kashyapv@us.ibm.com
> vivk@us.ibm.com
> 503 578 3422 (o)
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 13:31:03 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA29406
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 13:30:59 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id NAA17298
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 13:31:00 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA16885;
	Thu, 17 Jan 2002 13:28:15 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA16838
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 13:28:13 -0500 (EST)
Received: from e34.co.us.ibm.com (e34.co.us.ibm.com [32.97.110.132])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA29305
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 13:28:10 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e34.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id NAA38906;
	Thu, 17 Jan 2002 13:25:09 -0500
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0HISAA98808;
	Thu, 17 Jan 2002 11:28:10 -0700
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0HIS7V07861;
	Thu, 17 Jan 2002 10:28:07 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0HIS6Y14909;
	Thu, 17 Jan 2002 10:28:06 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201171828.g0HIS6Y14909@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
To: jpink@microsoft.com (Jim Pinkerton)
Date: Thu, 17 Jan 2002 10:28:05 -0800 (PST)
Cc: kashyapv@us.ibm.com (Vivek Kashyap),
        ARVINDM@windows.microsoft.com (Arvind Murching), ipoverib@ietf.org
In-Reply-To: <24A715275661C8428C00432EFCA5CB7C05E3C1AC@red-msg-02.redmond.corp.microsoft.com> from "Jim Pinkerton" at Jan 17, 2002 09:09:13 AM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Jim,
        The spanning tree is a different than IP connectivity
        information exchanged via ARP. Notice that the ARP packets
        duplicate the ethernet source/destination information in the
        ARP packet. The address is not picked up from the frame.

        A spoofer can give the wrong GID too. Kiss any security
        good-bye if that is all it is based on. If you think that is
        hard - there are no safeguards against any of the SA updates.
        It can be a DoS attack if non-existent GIDs are returned.

        Additionally, consider the case at present where there is one
        particular point across the IB subnet that can be contacted.
        It is only one single point of that type. A local IPoIB device
        responds with the IB routers LID and the remote points GID in
        response to the ARP. Things work. It is a valid example
        whether you like it or not. I think Roger and Yaron have
        talked of such devices.

        Thirdly, there is the implementational simplicity and
        flexibility that comes with including the LID. Those that
        always want to go to the SA, ignore the LID and do so. Those
        that think the responders value is good enough can use it. Why
        is that a problem ? I would like to know the answer to that.

        I have explained in earlier mails how the LID can be ignored,
        I probably don't need to tell anyone that, but it can be
        ignored very easily by those that want to. 


Vivek

> 
> 
> Vivek,
> 
> Using the source address for the Spanning Tree algorithm for bridging
> (not routing) is required on Ethernet. Thus on Ethernet at least, the
> source address is *always* the same as the true source - otherwise
> spanning tree doesn't work. Technically you could make the argument that
> the true restriction is that the side of the bridge that the spoofer
> sits on just has to be the same as the spoofee - but no-one does it that
> way that I'm aware of. KISS rules(keep it simple stupid). 
> 
> In fact, if you allow what you are suggesting it is technically a
> security hole. That's why on some networks even gratuitous ARP replies
> are intentionally discarded. If you only allow ARP replies to ARP
> requests, then if you have a spoofing malicious user out there you'll
> get two replies (and thus detect the spoofer and signal a problem to the
> network admin).
> 
> 
> Jim
> 
> 
> 
> 
> 
> 
> > -----Original Message-----
> > From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> > Sent: Wednesday, January 16, 2002 11:31 AM
> > To: Arvind Murching
> > Cc: Vivek Kashyap; ipoverib@ietf.org
> > Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> > 
> >         A device can always respond for another. The LID of the sender
> >         is not the same as the LID for which it has responded. It also
> >         allows any network monitoring/debugging device to work
> >         independently. This is the reason why none of the other media
> >         does anything like this. You don't pick ethernet source
> >         address from the frame but take it from the ARP/ND packet.
> > 
> >         [ If it was ok we wouldn't need anything except QPN/cap_flags
> >         - get the LID from response, consult SA using LID for GID if
> >         you want that, and you are done. ]
> > 
> > Vivek
> > 
> > >
> > > Why do we need to include the LID inside the link layer address when
> it
> > > is already available in the SLID field? Previous discussions
> indicate
> > > that the SLID value works for proxy ARP/bridge implementations.
> > >
> > > - Arvind
> > >
> > > -----Original Message-----
> > > From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> > > Sent: Wednesday, January 16, 2002 10:29 AM
> > > To: kamath@seagullsemi.com
> > > Cc: ipoverib@ietf.org
> > > Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> > >
> > > Ashwin,
> > > Ashwin,
> > >
> > >         I agree that you get more flexibility if you include the
> LID.
> > >         If we include the GID along with the LID we allow all
> > >         implementations the choice to create different solutions. We
> > >         don't tie them down to one thing. Choosing one or the other
> > >         (LID or GID) blocks out various advantages that different
> > >         implementations seek. Hence I suggested that let us include
> > >         both and move on.
> > >
> > > Vivek
> > >
> > > <snip>
> > >
> > > _______________________________________________
> > > IPoverIB mailing list
> > > IPoverIB@ietf.org
> > > http://www.ietf.org/mailman/listinfo/ipoverib
> > >
> > 
> > 
> > --
> > Vivek Kashyap
> > Linux Technology Center, IBM
> > kashyapv@us.ibm.com
> > vivk@us.ibm.com
> > 503 578 3422 (o)
> > 
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 13:49:34 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA00068
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 13:49:34 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id NAA17854
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 13:49:36 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA17775;
	Thu, 17 Jan 2002 13:47:56 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA17746
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 13:47:55 -0500 (EST)
Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA00029
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 13:47:52 -0500 (EST)
Received: from sunmail1.Sun.COM ([129.145.1.2])
	by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id KAA03852;
	Thu, 17 Jan 2002 10:47:51 -0800 (PST)
Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.88.130])
	by sunmail1.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1-Sun.COM.mod.2) with ESMTP id KAA27825;
	Thu, 17 Jan 2002 10:49:00 -0800 (PST)
Received: from taipei (taipei.Eng.Sun.COM [129.146.85.178])
	by jurassic.eng.sun.com (8.12.2+Sun/8.12.2) with SMTP id g0HIlkqT704559;
	Thu, 17 Jan 2002 10:47:47 -0800 (PST)
Message-Id: <200201171847.g0HIlkqT704559@jurassic.eng.sun.com>
Date: Thu, 17 Jan 2002 10:45:14 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Subject: RE: [Ipoverib] encapsulation issues
To: Paul.Culley@compaq.com
Cc: ipoverib@ietf.org
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: dkaxhiZLcRCVavtpfHGS/w==
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.6_06 SunOS 5.8 sun4u sparc 
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Paul,

Using additional bits to flag something being optional seems like an
attractive compromise in order to suite everybody's need. But there is
a constraint in IPv4 due to the broadcast nature of the ARP requests.
A node must advertise the same MAC addr to all the other nodes on the
link. In other words the response can not be requester dependent.
(If I tell node A don't use this field, I'd better tell node B the
same.) IPv6 doesn't have this constraint so it provides more flexibility.
But v4 is still dominating so we must find a solution for it.

It looks like for Proxy ARP the above won't be a problem though.
For IB router it could be a problem. (But the driver has the knowledge
to do the right thing. Oh, I just discover it requires driver snooping
ARP pkts, not the best soluion one would hope for.)

Also I'm very concerned about the refresh of the pathrecord cache (for
attributes like LID, SL, static rate, LMC...), most likely maintained
by the driver. Does IBA provide solid mechanism for one to build an
reliable event notification mechanism that anyone can register for
attribute changes, either local or remote?

We can (and should) only rely on ARP/ND to flush attributes that are part
of IPoIB link layer addrs. For the rest of attributes IBA must provide
a solution.

Rgds,

Jerry

>If you insist on including the LID (presumably in the body of the ARP
>Response) then I have the following thoughts:
>
>1) Include a means where the ARP Requestor can decide when not to use
>that LID.  For example, if IP across multiple IB subnets someday becomes
>feasible, any time the subnet prefixes between the ARP Requestor and ARP
>Responder are different, the LID (in the body of the response) will be
>invalid.  There may also be circumstances where the ARP Responder knows
>that the LID is not valid; in that case there should be a defined method
>of indicating such.
>
>2) Since the LID alone in the body of the ARP Response is not sufficient
>to route packets, also provide the SL.  IB provides NO WAY to select an
>SL, based only on the LID, unless the ARP requestor already has that LID
>and an SL cached or is willing to look it up with the SA.  Since one of
>the reasons for including the LID at all is to avoid this lookup, the
>solution is not complete without SL.  [Please don't respond that some
>implementation specific mechanism can be chosen to select SL; there is
>no way for an endnode implementation to determine the SA policies for SL
>usage at this time, so it is not possible for the endnode to reliably
>chose an SL without asking the SA (or caching a LID/SL derived from the
>SA).
>
>3) A potential solution to the above problems that seems like it would
>satisfy everyone would be to use both the SLID and SL from the IB
>Header.  Since there may be some circumstances where the ARP Responder
>KNOWs that this won't work (odd 3rd party ARP thingies have been
>suggested), include a flag that indicates that the SLID/SL is invalid.
>The GID is available in this case, or for those devices that want more
>than a "best effort" path; it can be used for PathRecord Lookup.  This
>will work properly for normal devices in the subnet, IB routers (when
>they appear), and Proxy ARP types of routing.
>
>Paul R. Culley
>Compaq Sr. Fellow
>281-514-5543
>
>
>-----Original Message-----
>From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
>Sent: Wednesday, January 16, 2002 5:57 PM
>To: ipoverib@ietf.org
>Subject: [Ipoverib] encapsulation issues
>
>
>
>1.      We have been arguing over the inclusion of LID/GID or both for
>        a while now. Examples and advantages of including either have
>        been espoused often enough. However, there is an agreement, I
>        think, on the architecture not forcing a limited solution on
>        any set of aspirations.
>
>        As noted earlier the inclusion of both the identifiers does
>        not handicap or restrict any solution. 
>	
>	Let us include both.
>
>
>2. 	I believe based on the recent set of mails that the consensus no
>        longer favours inclusion of the 'ethertype'. Is there a
>        disagreement ?
>
>
>3. 	As many have noted in the last couple of days that using IB
>        over non-UD modes or utilising QPNs mapped to specific
>        endpoints is desirable. A complete, interoperable solution has
>        been presented in
>        draft-kashyap-ipoib-advanced-capabilities-00.txt. 
>
>	Please review and comment. 
>
>Vivek
>
>
>	
>
>-- 
>Vivek Kashyap
>Linux Technology Center, IBM
>kashyapv@us.ibm.com 
>vivk@us.ibm.com 
>503 578 3422 (o)
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>http://www.ietf.org/mailman/listinfo/ipoverib
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 14:00:40 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA00439
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 14:00:40 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id OAA18476
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 14:00:42 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA18140;
	Thu, 17 Jan 2002 13:59:35 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA18108
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 13:59:33 -0500 (EST)
Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA00402
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 13:59:31 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id MAA106020;
	Thu, 17 Jan 2002 12:55:17 -0600
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0HIxTE194566;
	Thu, 17 Jan 2002 13:59:30 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id g0HIxTg28807;
	Thu, 17 Jan 2002 10:59:29 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0HIxSV17126;
	Thu, 17 Jan 2002 10:59:28 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201171859.g0HIxSV17126@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
To: bill@strahm.net (Bill Strahm)
Date: Thu, 17 Jan 2002 10:59:27 -0800 (PST)
Cc: ipoverib@ietf.org
In-Reply-To: <HHEDIJGOEFLPCPNILMPAEEAFCFAA.bill@strahm.net> from "Bill Strahm" at Jan 17, 2002 08:18:05 AM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

> 
> Not sure I agree with a few things here...
> 
> The SM IS the ROOT DNS server, because it is the ultimate authority on what
> is where in the fabric.  Is there any doubt to that.  Also the end nodes
> have nothing to do with resolution, I do not believe there is any IB
> mechanism to go to the end node and ask it for a Path Record is there ?
> 
> You are correct that if the LID changes there is no back path to tell
> everyone that the LID has changed, this is just like DNS so the analogy
> holds still.  Since the local SM is the DNS Root going "All the way to the
> root" is not a problem.
> 
> You need a timer just like in DNS, so the analogy still holds.  The timer
> works at the same level as the ARP timer.  When you ARP, you go get the
> answer from the SM... Until then you can deal with LID changes because
> either a) your communication fails - you have this problem in either
> scenario b) you notice a LID change on an incoming packet so update your ARP
> table

In the case of only GID you don't know if the LID changed. You have to
have your secondary GID to LID timer go off too.

> 
> No, only the responder needs to reply to an ARP broadcast, so therefor only
> one entry.  Now other nodes in the network may decide to go to the SM to
> cache that particular entry as well, but then you remove latencies when you
> want to talk to that node anyway... I would assume that any decent SM could
> handle the traffic from 65,000 nodes asking it to do a similar lookup at the
> same time...  This of course is a worst case scenario.

You are right that the responder needs to respond to the broadcast but
all receivers as per the ARP algorithm update their entries. That
means all rush off to the SA.

> 
> The problem I have is that unless you have manually configured the LID-SL
> mapping for your subnet, you could deadlock the fabric this way.  Now would
> you want to manually configure LID-SL lookup for those same 65000 nodes ???

Yes. That is a very valid point.  you have made it better than I did. 
The SL lookup, management etc. and the modes of determining it are
upto the implementation. Also note that the high spike in traffic in
itself is undesirable even if just getting the LID from the SA for the
GID just advertised.

Vivek

> 
> Bill
> 
> -----Original Message-----
> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> Sent: Thursday, January 17, 2002 1:27 AM
> To: Bill Strahm
> Cc: ipoverib@ietf.org
> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> 
> 
> 
> >
> > Just free form thinking here...
> >
> > Could we say that GIDs are the equivalent of a Fully Qualified Domain Name
> > (www.strahm.net for example) and LIDs are the equivalent of IP addresses
> > (63.105.18.42).  Now in many cases IP addresses are quite stable, and do
> not
> > change.  However would you ship an application with an IP address encoded
> in
> > it, or with a FQDN.  It would be of course more efficient to use the IP
> > address and avoid the DNS lookup (that has to go VERY far away, and can
> take
> > quite a while for the first SYN message to get out), however if the
> address
> > EVER changes, things become hard.
> >
> > Can anyone tell me why this is a bad analogy ???
> 
>         If it was pure IB then GID to LID would be a similar analogy.
> 
>         I'll try taking the your analogy further..a couple of points
>         though - the SM corresonds to the zone primary but the nodes
>         themselves corresond to both the secondary and the IP nodes
>         asking for the resolution. So the analogy is not exactly apt
>         but I'll try..
> 
>         If the IP changes the DNS must be updated and looked up. In
>         this case if LID changes the SM is updated (yes) but it then
>         requires that the SM be looked up. There is no backlink here
>         to trigger this lookup. This implies that the full lookup all
>         the way to the DNS root must be done - if like DNS it is done
>         till the local secondary server ie. referring to the local
>         cached copy and/or merely re-receiving the peer's GID, the
>         communication will fail. You must have seen the hell that
>         breaks loose if any of the zone updates doesn't happen due to
>         some error and things are left pointing the wrong way.
> 
>         Now, this would correspond to the internal LID cache flush -
>         since the GID is the same but the LID has changed. There is no
>         hook to say, hey!, go do a zone transfer i.e. lookup the SM
>         right now. You need a timer.
> 
>         Now, you could make this time small - you are going to the SM
>         too often because things are not changing that much- make it
>         long and LID change is surely going to happen at a critical
>         time. Additionally this is another timer per entry for the
>         programmers/maintainers/administrators to worry about.
> 
>         I have all along been arguing against keeping LID out due to
>         this reason too (code is more complex if no LID is another
>         reason). It has been my assumption that if GID is used without
>         LID then one is forced to go to the SM at every address
>         resolution. Every arp broadcast will have the whole subnet
>         scurrying off to the SM to ask the requester's LID. Yes, some
>         would mark the entry to be completed but not necessarily - a
>         thundering herd is likely.
> 
>         This implies that if we don't use the LID we MUST ALWAYS go to
>         the SM for the LID - at every ARP since we don't know if LID
>         has changed. Of course if the GID changed too then the SM has
>         to be consulted anyway.
> 
>         Also note that if we did have a trap from the SM on every LID
>         change it still wouldn't be a good solution - too much
>         latency; points of synchrnonisation; asynchronous updates; and
>         most of all the SM would end up doing nothing else but issueing
>         traps to every node that registered with it. IB cannot mandate
>         broadcasts/multicast (though making everyone look is bad too)
>         because multicasting is optional.
> 
>         In the case of inclusion of the LID - some of the issues are
>         the same but a gratuitous ARP solves the problem if the LID
>         changed. It is an instant zone transfer. One could claim that
>         with GID only, on receiving any gratuitous ARP entry, the node
>         will go to the SM but that will require specialised coding to
>         recognise and handle gratuitous ARP differently than regular
>         updates. Not a good idea.
> 
>         Even in the absence of gratuitous ARP the changed LID will be
>         known when the ARP is done after a cache expiry if LID is
>         included. If LID is not included one will get a GID back. That
>         will require the expiry of the GID to LID cache before one
>         goes to the SM.
> 
>         Inclusion of LID makes things very simple. This is because LID
>         is always necessary - can't escape it.
> 
>         I also like inclusion of LID because it cuts off 2x latency of
>         GID to LID lookup in the SM and 4x the RTT and the realted
>         extra interrupts. Fast failover in data centers is necessary.
> 
> Vivek
> 
> 
> 
> >
> > Bill
> >
> > -----Original Message-----
> > From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
> > Of Larsen, Roy K
> > Sent: Wednesday, January 16, 2002 6:43 PM
> > To: 'H.K. Jerry Chu'; arvindm@windows.microsoft.com; kashyapv@us.ibm.com
> > Cc: ipoverib@ietf.org
> > Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
> >
> >
> > > -----Original Message-----
> > > From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
> > > Sent: Wednesday, January 16, 2002 6:13 PM
> > > To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com
> > > Cc: ipoverib@ietf.org
> > > Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> > >
> > >
> > > All the proxy ARP usages that I know of use the proxy's mac
> > > addr in the
> > > ARP response. I believe the flavor of the proxy ARP you
> > > alluded below is
> > > rare, and won't work very well anyway if the link address includes any
> > > non-persistent component like LID and QPN because it will
> > > require another
> > > protocol between the clients and their proxy to constantly update the
> > > proxy database containing clients mac addrs, which may change quite
> > > often.
> > >
> > > That is also why I wanted to provide an option to not include QPN.
> > > (But I haven't succeeded w/o requiring SIDR or something equivalent.)
> > >
> > > If LID is included as part of link addr and if it ever changes on
> > > a live node due to subnet merge...etc, the driver must inform its IP
> > > stack about such change. I believe many stacks don't have such a
> > > facility. (FOr most IP stacks the assumption is that its own MAC
> > > addresses just don't change on the fly.) In that case you're totally
> > > hosed. Can someone comment on how likely a LID assignment gets
> > > rescinded?
> >
> > It is possible anytime the link goes down and comes back up again and when
> > the fabic is reconfigured.  This can happen on reboot or simply
> > unplugging/plugging in a link.  It is ultimately a function of the SM and
> > how much trouble the implementation wants to go through to keep LIDs
> > constant.
> >
> > But of course, that has no bearing on whether the LID is contained in the
> > ARP.  You always need the LID.  Unless you want to do a pathrecord lookup
> > each time your implementation is given a packet to send, you're going to
> use
> > a cached LID no matter if it came in the ARP or a pathrecord lookup on the
> > GID.  If it becomes stale, it may take an ARP cache flush timeout to
> > recover.
> >
> > Roy
> >
> > >
> > > Jerry
> > >
> > > >        A device can always respond for another. The LID of
> > > the sender
> > > >        is not the same as the LID for which it has
> > > responded. It also
> > > >        allows any network monitoring/debugging device to work
> > > >        independently. This is the reason why none of the other media
> > > >        does anything like this. You don't pick ethernet source
> > > >        address from the frame but take it from the ARP/ND packet.
> > > >
> > > >        [ If it was ok we wouldn't need anything except QPN/cap_flags
> > > >        - get the LID from response, consult SA using LID for GID if
> > > >        you want that, and you are done. ]
> > > >
> > > >Vivek
> > > >
> > > >>
> > > >> Why do we need to include the LID inside the link layer
> > > address when it
> > > >> is already available in the SLID field? Previous
> > > discussions indicate
> > > >> that the SLID value works for proxy ARP/bridge implementations.
> > > >>
> > > >> - Arvind
> > > >>
> > > >> -----Original Message-----
> > > >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> > > >> Sent: Wednesday, January 16, 2002 10:29 AM
> > > >> To: kamath@seagullsemi.com
> > > >> Cc: ipoverib@ietf.org
> > > >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> > > >>
> > > >> Ashwin,
> > > >> Ashwin,
> > > >>
> > > >>         I agree that you get more flexibility if you
> > > include the LID.
> > > >>         If we include the GID along with the LID we allow all
> > > >>         implementations the choice to create different
> > > solutions. We
> > > >>         don't tie them down to one thing. Choosing one or the other
> > > >>         (LID or GID) blocks out various advantages that different
> > > >>         implementations seek. Hence I suggested that let us include
> > > >>         both and move on.
> > > >>
> > > >> Vivek
> > > >>
> > > >> <snip>
> > > >>
> > > >> _______________________________________________
> > > >> IPoverIB mailing list
> > > >> IPoverIB@ietf.org
> > > >> http://www.ietf.org/mailman/listinfo/ipoverib
> > > >>
> > > >
> > > >
> > > >--
> > > >Vivek Kashyap
> > > >Linux Technology Center, IBM
> > > >kashyapv@us.ibm.com
> > > >vivk@us.ibm.com
> > > >503 578 3422 (o)
> > > >
> > > >_______________________________________________
> > > >IPoverIB mailing list
> > > >IPoverIB@ietf.org
> > > >http://www.ietf.org/mailman/listinfo/ipoverib
> > >
> > >
> > > _______________________________________________
> > > IPoverIB mailing list
> > > IPoverIB@ietf.org
> > > http://www.ietf.org/mailman/listinfo/ipoverib
> > >
> >
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> >
> >
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> >
> 
> 
> --
> Vivek Kashyap
> Linux Technology Center, IBM
> kashyapv@us.ibm.com
> vivk@us.ibm.com
> 503 578 3422 (o)
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 14:44:21 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA04528
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 14:44:21 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id OAA20802
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 14:44:23 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA20738;
	Thu, 17 Jan 2002 14:42:36 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA20702
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 14:42:32 -0500 (EST)
Received: from thalia.fm.intel.com (fmfdns02.fm.intel.com [132.233.247.11])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA03978
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 14:42:28 -0500 (EST)
Received: from fmsmsxvs041.fm.intel.com (fmsmsxv041-1.fm.intel.com [132.233.48.109])
	by thalia.fm.intel.com (8.9.1a+p1/8.9.1/d: relay.m4,v 1.48 2001/12/13 16:27:50 root Exp $) with SMTP id TAA16754
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 19:42:29 GMT
Received: from FMSMSX016.fm.intel.com ([132.233.42.195])
 by fmsmsxvs041.fm.intel.com (NAVGW 2.5.1.16) with SMTP id M2002011711432623813
 ; Thu, 17 Jan 2002 11:43:26 -0800
Received: by fmsmsx016.fm.intel.com with Internet Mail Service (5.5.2653.19)
	id <DAG4FFTM>; Thu, 17 Jan 2002 11:42:28 -0800
Message-ID: <D9223EB959A5D511A98F00508B68C20C21A850@ORSMSX108>
From: "Larsen, Roy K" <roy.k.larsen@intel.com>
To: "'H.K. Jerry Chu'" <Jerry.Chu@eng.sun.com>, arvindm@windows.microsoft.com,
        kashyapv@us.ibm.com
Cc: ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
Date: Thu, 17 Jan 2002 11:42:20 -0800
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="ISO-8859-1"
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Jerry,

Ok, your concern is a mechanism for notifying a node that a LID change has
occurred regardless of how the LID was initially obtained.  I'm no SM expert
but seem to recall asking my resident expert about this a while back and he
said there wasn't.  However, if the LID was included in the ARP, wouldn't
sending a gratuitous ARP when the link comes up solve the problem? 

Roy

-----Original Message-----
From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
Sent: Wednesday, January 16, 2002 7:28 PM
To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com;
roy.k.larsen@intel.com
Cc: ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?


Roy,

Thanks for the info. A comment on your statement of volatility of LID having
no bearing on whether LID should be included in ARP reply. Yes it does. One
of the goal of IPoIB is to minimize the changes to the existing IP stack
implementations. Note that I stated it as a goal, not a constraint. If we
must do it then we should.

Yes no matter what, LID is cached somewhere and must be flushed. The
difference
is whether one must modify the existing stack code to add a facility to
support
mac addr change, or do the necessary chore in a complete new IPoIB driver.
The former requires an upgrade of OS release that can take a long time. The
latter doesn't have such a problem since it's brand new.

One can also hope that LID flushing through SM can be made more reliable
than
gratuitous ARP. But I don't know the SM archtecture enough to make such a
claim.

To be fair I don't really know how many OSes lack support for mac add
change. The two that I know, one supports it, one doesn't.

I'm assuming there are solid facility in IBA to support the notification of
attribute changes to interested parties. Otherwise not only we should
include
LID, but also all the rest in the link addr.

Jerry

>From: "Larsen, Roy K" <roy.k.larsen@intel.com>
>To: "'H.K. Jerry Chu'" <Jerry.Chu@eng.sun.com>,
arvindm@windows.microsoft.com, kashyapv@us.ibm.com
>Cc: ipoverib@ietf.org
>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>Date: Wed, 16 Jan 2002 18:42:31 -0800
>MIME-Version: 1.0
>
>> -----Original Message-----
>> From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
>> Sent: Wednesday, January 16, 2002 6:13 PM
>> To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com
>> Cc: ipoverib@ietf.org
>> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>> 
>> 
>> All the proxy ARP usages that I know of use the proxy's mac 
>> addr in the
>> ARP response. I believe the flavor of the proxy ARP you 
>> alluded below is
>> rare, and won't work very well anyway if the link address includes any
>> non-persistent component like LID and QPN because it will 
>> require another
>> protocol between the clients and their proxy to constantly update the
>> proxy database containing clients mac addrs, which may change quite
>> often.
>> 
>> That is also why I wanted to provide an option to not include QPN.
>> (But I haven't succeeded w/o requiring SIDR or something equivalent.)
>> 
>> If LID is included as part of link addr and if it ever changes on
>> a live node due to subnet merge...etc, the driver must inform its IP
>> stack about such change. I believe many stacks don't have such a
>> facility. (FOr most IP stacks the assumption is that its own MAC
>> addresses just don't change on the fly.) In that case you're totally
>> hosed. Can someone comment on how likely a LID assignment gets
>> rescinded?
>
>It is possible anytime the link goes down and comes back up again and when
>the fabic is reconfigured.  This can happen on reboot or simply
>unplugging/plugging in a link.  It is ultimately a function of the SM and
>how much trouble the implementation wants to go through to keep LIDs
>constant.
>
>But of course, that has no bearing on whether the LID is contained in the
>ARP.  You always need the LID.  Unless you want to do a pathrecord lookup
>each time your implementation is given a packet to send, you're going to
use
>a cached LID no matter if it came in the ARP or a pathrecord lookup on the
>GID.  If it becomes stale, it may take an ARP cache flush timeout to
>recover.  
>
>Roy
> 
>> 
>> Jerry
>> 
>> >        A device can always respond for another. The LID of 
>> the sender
>> >        is not the same as the LID for which it has 
>> responded. It also
>> >        allows any network monitoring/debugging device to work
>> >        independently. This is the reason why none of the other media
>> >        does anything like this. You don't pick ethernet source
>> >        address from the frame but take it from the ARP/ND packet.
>> >
>> >        [ If it was ok we wouldn't need anything except QPN/cap_flags
>> >        - get the LID from response, consult SA using LID for GID if
>> >        you want that, and you are done. ]
>> >
>> >Vivek
>> >
>> >> 
>> >> Why do we need to include the LID inside the link layer 
>> address when it
>> >> is already available in the SLID field? Previous 
>> discussions indicate
>> >> that the SLID value works for proxy ARP/bridge implementations.
>> >> 
>> >> - Arvind
>> >> 
>> >> -----Original Message-----
>> >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com] 
>> >> Sent: Wednesday, January 16, 2002 10:29 AM
>> >> To: kamath@seagullsemi.com
>> >> Cc: ipoverib@ietf.org
>> >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>> >> 
>> >> Ashwin,
>> >> Ashwin,
>> >> 
>> >>         I agree that you get more flexibility if you 
>> include the LID.
>> >>         If we include the GID along with the LID we allow all
>> >>         implementations the choice to create different 
>> solutions. We
>> >>         don't tie them down to one thing. Choosing one or the other
>> >>         (LID or GID) blocks out various advantages that different
>> >>         implementations seek. Hence I suggested that let us include
>> >>         both and move on.
>> >> 
>> >> Vivek
>> >> 
>> >> <snip>
>> >> 
>> >> _______________________________________________
>> >> IPoverIB mailing list
>> >> IPoverIB@ietf.org
>> >> http://www.ietf.org/mailman/listinfo/ipoverib
>> >> 
>> >
>> >
>> >-- 
>> >Vivek Kashyap
>> >Linux Technology Center, IBM
>> >kashyapv@us.ibm.com 
>> >vivk@us.ibm.com 
>> >503 578 3422 (o)
>> >
>> >_______________________________________________
>> >IPoverIB mailing list
>> >IPoverIB@ietf.org
>> >http://www.ietf.org/mailman/listinfo/ipoverib
>> 
>> 
>> _______________________________________________
>> IPoverIB mailing list
>> IPoverIB@ietf.org
>> http://www.ietf.org/mailman/listinfo/ipoverib
>> 

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 15:15:39 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA07404
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 15:15:38 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id PAA22099
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 15:15:41 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA21979;
	Thu, 17 Jan 2002 15:14:47 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA21929
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 15:14:45 -0500 (EST)
Received: from vieo.com (vieo.com [216.30.79.131])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA07323
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 15:14:41 -0500 (EST)
Received: (from root@localhost)
	by vieo.com (8.11.2/8.11.2) id g0HKECo22707
	for ipoverib@ietf.org; Thu, 17 Jan 2002 14:14:12 -0600 (CST)
	(envelope-from rronald@VIEO.com)
Received: from Fishhead (dhcp-10-2-15-151.vieo.com [10.2.15.151])
	by vieo.com (8.11.2/8.11.2) with SMTP id g0HKEBc22625;
	Thu, 17 Jan 2002 14:14:11 -0600 (CST)
	(envelope-from rronald@VIEO.com)
Message-ID: <00ed01c19f93$7e74b070$970f020a@Fishhead>
From: "Roger Ronald" <rronald@VIEO.com>
To: "Vivek Kashyap" <kashyapv@us.ibm.com>
Cc: <ipoverib@ietf.org>
References: <200201170927.g0H9RBg12021@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
Date: Thu, 17 Jan 2002 14:13:56 -0600
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 5.50.4133.2400
X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400
X-scanner: scanned by Inflex 0.1.5c+ on vieo.com
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

I think it's important to note that changes in LID should be infrequent.
Very infrequent.

VIEO is one of a small number of companies that has a SM at this time.
We plan to maintain LID assignments whenever possible. That means
across reboots, across power fails and across subnet merges (when possible).

There are corner cases where LID may need to change, just like IP
addresses occasionaly need to change. But these are VERY odd
events...like the first time merge of fabrics. They won't happen
hourly, daily, monthly..... They may happen once or twice in a 
fabric. Ever (unless the admin wants to make changes for the heck of it).

RR


----- Original Message ----- 
From: "Vivek Kashyap" <kashyapv@us.ibm.com>
To: "Bill Strahm" <bill@strahm.net>
Cc: <ipoverib@ietf.org>
Sent: Thursday, January 17, 2002 3:27 AM
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?


> 
> > 
> > Just free form thinking here...
> > 
> > Could we say that GIDs are the equivalent of a Fully Qualified Domain Name
> > (www.strahm.net for example) and LIDs are the equivalent of IP addresses
> > (63.105.18.42).  Now in many cases IP addresses are quite stable, and do not
> > change.  However would you ship an application with an IP address encoded in
> > it, or with a FQDN.  It would be of course more efficient to use the IP
> > address and avoid the DNS lookup (that has to go VERY far away, and can take
> > quite a while for the first SYN message to get out), however if the address
> > EVER changes, things become hard.
> > 
> > Can anyone tell me why this is a bad analogy ???
> 
>         If it was pure IB then GID to LID would be a similar analogy.
> 
>         I'll try taking the your analogy further..a couple of points
>         though - the SM corresonds to the zone primary but the nodes
>         themselves corresond to both the secondary and the IP nodes
>         asking for the resolution. So the analogy is not exactly apt
>         but I'll try..
> 
>         If the IP changes the DNS must be updated and looked up. In
>         this case if LID changes the SM is updated (yes) but it then
>         requires that the SM be looked up. There is no backlink here
>         to trigger this lookup. This implies that the full lookup all
>         the way to the DNS root must be done - if like DNS it is done
>         till the local secondary server ie. referring to the local
>         cached copy and/or merely re-receiving the peer's GID, the
>         communication will fail. You must have seen the hell that
>         breaks loose if any of the zone updates doesn't happen due to
>         some error and things are left pointing the wrong way.
> 
>         Now, this would correspond to the internal LID cache flush -
>         since the GID is the same but the LID has changed. There is no
>         hook to say, hey!, go do a zone transfer i.e. lookup the SM
>         right now. You need a timer.
> 
>         Now, you could make this time small - you are going to the SM
>         too often because things are not changing that much- make it
>         long and LID change is surely going to happen at a critical
>         time. Additionally this is another timer per entry for the
>         programmers/maintainers/administrators to worry about.
> 
>         I have all along been arguing against keeping LID out due to
>         this reason too (code is more complex if no LID is another
>         reason). It has been my assumption that if GID is used without
>         LID then one is forced to go to the SM at every address
>         resolution. Every arp broadcast will have the whole subnet
>         scurrying off to the SM to ask the requester's LID. Yes, some
>         would mark the entry to be completed but not necessarily - a
>         thundering herd is likely.
> 
>         This implies that if we don't use the LID we MUST ALWAYS go to
>         the SM for the LID - at every ARP since we don't know if LID
>         has changed. Of course if the GID changed too then the SM has
>         to be consulted anyway.
> 
>         Also note that if we did have a trap from the SM on every LID
>         change it still wouldn't be a good solution - too much
>         latency; points of synchrnonisation; asynchronous updates; and
>         most of all the SM would end up doing nothing else but issueing
>         traps to every node that registered with it. IB cannot mandate
>         broadcasts/multicast (though making everyone look is bad too)
>         because multicasting is optional.
> 
>         In the case of inclusion of the LID - some of the issues are
>         the same but a gratuitous ARP solves the problem if the LID
>         changed. It is an instant zone transfer. One could claim that
>         with GID only, on receiving any gratuitous ARP entry, the node
>         will go to the SM but that will require specialised coding to
>         recognise and handle gratuitous ARP differently than regular
>         updates. Not a good idea.
> 
>         Even in the absence of gratuitous ARP the changed LID will be
>         known when the ARP is done after a cache expiry if LID is
>         included. If LID is not included one will get a GID back. That
>         will require the expiry of the GID to LID cache before one
>         goes to the SM.
> 
>         Inclusion of LID makes things very simple. This is because LID
>         is always necessary - can't escape it.
> 
>         I also like inclusion of LID because it cuts off 2x latency of
>         GID to LID lookup in the SM and 4x the RTT and the realted
>         extra interrupts. Fast failover in data centers is necessary.
> 
> Vivek
> 
> 
> 
> > 
> > Bill
> > 
> > -----Original Message-----
> > From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
> > Of Larsen, Roy K
> > Sent: Wednesday, January 16, 2002 6:43 PM
> > To: 'H.K. Jerry Chu'; arvindm@windows.microsoft.com; kashyapv@us.ibm.com
> > Cc: ipoverib@ietf.org
> > Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
> > 
> > 
> > > -----Original Message-----
> > > From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
> > > Sent: Wednesday, January 16, 2002 6:13 PM
> > > To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com
> > > Cc: ipoverib@ietf.org
> > > Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> > >
> > >
> > > All the proxy ARP usages that I know of use the proxy's mac
> > > addr in the
> > > ARP response. I believe the flavor of the proxy ARP you
> > > alluded below is
> > > rare, and won't work very well anyway if the link address includes any
> > > non-persistent component like LID and QPN because it will
> > > require another
> > > protocol between the clients and their proxy to constantly update the
> > > proxy database containing clients mac addrs, which may change quite
> > > often.
> > >
> > > That is also why I wanted to provide an option to not include QPN.
> > > (But I haven't succeeded w/o requiring SIDR or something equivalent.)
> > >
> > > If LID is included as part of link addr and if it ever changes on
> > > a live node due to subnet merge...etc, the driver must inform its IP
> > > stack about such change. I believe many stacks don't have such a
> > > facility. (FOr most IP stacks the assumption is that its own MAC
> > > addresses just don't change on the fly.) In that case you're totally
> > > hosed. Can someone comment on how likely a LID assignment gets
> > > rescinded?
> > 
> > It is possible anytime the link goes down and comes back up again and when
> > the fabic is reconfigured.  This can happen on reboot or simply
> > unplugging/plugging in a link.  It is ultimately a function of the SM and
> > how much trouble the implementation wants to go through to keep LIDs
> > constant.
> > 
> > But of course, that has no bearing on whether the LID is contained in the
> > ARP.  You always need the LID.  Unless you want to do a pathrecord lookup
> > each time your implementation is given a packet to send, you're going to use
> > a cached LID no matter if it came in the ARP or a pathrecord lookup on the
> > GID.  If it becomes stale, it may take an ARP cache flush timeout to
> > recover.
> > 
> > Roy
> > 
> > >
> > > Jerry
> > >
> > > >        A device can always respond for another. The LID of
> > > the sender
> > > >        is not the same as the LID for which it has
> > > responded. It also
> > > >        allows any network monitoring/debugging device to work
> > > >        independently. This is the reason why none of the other media
> > > >        does anything like this. You don't pick ethernet source
> > > >        address from the frame but take it from the ARP/ND packet.
> > > >
> > > >        [ If it was ok we wouldn't need anything except QPN/cap_flags
> > > >        - get the LID from response, consult SA using LID for GID if
> > > >        you want that, and you are done. ]
> > > >
> > > >Vivek
> > > >
> > > >>
> > > >> Why do we need to include the LID inside the link layer
> > > address when it
> > > >> is already available in the SLID field? Previous
> > > discussions indicate
> > > >> that the SLID value works for proxy ARP/bridge implementations.
> > > >>
> > > >> - Arvind
> > > >>
> > > >> -----Original Message-----
> > > >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> > > >> Sent: Wednesday, January 16, 2002 10:29 AM
> > > >> To: kamath@seagullsemi.com
> > > >> Cc: ipoverib@ietf.org
> > > >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> > > >>
> > > >> Ashwin,
> > > >> Ashwin,
> > > >>
> > > >>         I agree that you get more flexibility if you
> > > include the LID.
> > > >>         If we include the GID along with the LID we allow all
> > > >>         implementations the choice to create different
> > > solutions. We
> > > >>         don't tie them down to one thing. Choosing one or the other
> > > >>         (LID or GID) blocks out various advantages that different
> > > >>         implementations seek. Hence I suggested that let us include
> > > >>         both and move on.
> > > >>
> > > >> Vivek
> > > >>
> > > >> <snip>
> > > >>
> > > >> _______________________________________________
> > > >> IPoverIB mailing list
> > > >> IPoverIB@ietf.org
> > > >> http://www.ietf.org/mailman/listinfo/ipoverib
> > > >>
> > > >
> > > >
> > > >--
> > > >Vivek Kashyap
> > > >Linux Technology Center, IBM
> > > >kashyapv@us.ibm.com
> > > >vivk@us.ibm.com
> > > >503 578 3422 (o)
> > > >
> > > >_______________________________________________
> > > >IPoverIB mailing list
> > > >IPoverIB@ietf.org
> > > >http://www.ietf.org/mailman/listinfo/ipoverib
> > >
> > >
> > > _______________________________________________
> > > IPoverIB mailing list
> > > IPoverIB@ietf.org
> > > http://www.ietf.org/mailman/listinfo/ipoverib
> > >
> > 
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> > 
> > 
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> > 
> 
> 
> -- 
> Vivek Kashyap
> Linux Technology Center, IBM
> kashyapv@us.ibm.com 
> vivk@us.ibm.com 
> 503 578 3422 (o)
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib
> 
> 


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 15:21:55 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA07762
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 15:21:24 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id PAA22342
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 15:21:26 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA22287;
	Thu, 17 Jan 2002 15:19:44 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA22253
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 15:19:42 -0500 (EST)
Received: from mail1.microsoft.com (mail1.microsoft.com [131.107.3.125])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA07602
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 15:19:39 -0500 (EST)
Received: from inet-vrs-01.redmond.corp.microsoft.com ([157.54.8.27]) by mail1.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Thu, 17 Jan 2002 12:19:07 -0800
Received: from 157.54.5.25 by inet-vrs-01.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Thu, 17 Jan 2002 12:19:05 -0800
Received: from red-imc-04.redmond.corp.microsoft.com ([157.54.2.168]) by inet-hub-03.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Thu, 17 Jan 2002 12:19:06 -0800
Received: from win-imc-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.39]) by red-imc-04.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Thu, 17 Jan 2002 12:19:06 -0800
Received: from win-msg-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.52]) by win-imc-01.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3588.0);
	 Thu, 17 Jan 2002 12:16:59 -0800
X-MimeOLE: Produced By Microsoft Exchange V6.0.6132.0
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
Date: Thu, 17 Jan 2002 12:16:58 -0800
Message-ID: <2E33960095B58E40A4D3345AB9F65EC102261870@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
Thread-Topic: [Ipoverib] What's inside an IPoIB link layer address?
Thread-Index: AcGfB0UvmJua7wr4Edao5ABQi2jWzAAecbrgAAR3N1A=
From: "Arvind Murching" <arvindm@windows.microsoft.com>
To: <ipoverib@ietf.org>
X-OriginalArrivalTime: 17 Jan 2002 20:16:59.0162 (UTC) FILETIME=[EAD067A0:01C19F93]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id PAA22254
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

I'm not an IB expert, but I would think that -if- we decided to use
gratuitious ARP to notify remote nodes of changes, then those remote
nodes would consult with the SA and re-create their address handles
using the indicated GID.

"Optimized/simplified" implementations would pick up new SLIDs from the
headers of these gratuitous ARP packets.

-----Original Message-----
From: Larsen, Roy K [mailto:roy.k.larsen@intel.com] 
Sent: Thursday, January 17, 2002 11:42 AM
To: 'H.K. Jerry Chu'; Arvind Murching; kashyapv@us.ibm.com
Cc: ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?

Jerry,

Ok, your concern is a mechanism for notifying a node that a LID change
has occurred regardless of how the LID was initially obtained.  I'm no
SM expert but seem to recall asking my resident expert about this a
while back and he said there wasn't.  However, if the LID was included
in the ARP, wouldn't sending a gratuitous ARP when the link comes up
solve the problem? 

Roy

-----Original Message-----
From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
Sent: Wednesday, January 16, 2002 7:28 PM
To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com;
roy.k.larsen@intel.com
Cc: ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?


Roy,

Thanks for the info. A comment on your statement of volatility of LID
having
no bearing on whether LID should be included in ARP reply. Yes it does.
One
of the goal of IPoIB is to minimize the changes to the existing IP stack
implementations. Note that I stated it as a goal, not a constraint. If
we
must do it then we should.

Yes no matter what, LID is cached somewhere and must be flushed. The
difference
is whether one must modify the existing stack code to add a facility to
support
mac addr change, or do the necessary chore in a complete new IPoIB
driver.
The former requires an upgrade of OS release that can take a long time.
The
latter doesn't have such a problem since it's brand new.

One can also hope that LID flushing through SM can be made more reliable
than
gratuitous ARP. But I don't know the SM archtecture enough to make such
a claim.

To be fair I don't really know how many OSes lack support for mac add
change. The two that I know, one supports it, one doesn't.

I'm assuming there are solid facility in IBA to support the notification
of
attribute changes to interested parties. Otherwise not only we should
include
LID, but also all the rest in the link addr.

Jerry

>From: "Larsen, Roy K" <roy.k.larsen@intel.com>
>To: "'H.K. Jerry Chu'" <Jerry.Chu@eng.sun.com>,
arvindm@windows.microsoft.com, kashyapv@us.ibm.com
>Cc: ipoverib@ietf.org
>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>Date: Wed, 16 Jan 2002 18:42:31 -0800
>MIME-Version: 1.0
>
>> -----Original Message-----
>> From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
>> Sent: Wednesday, January 16, 2002 6:13 PM
>> To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com
>> Cc: ipoverib@ietf.org
>> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>> 
>> 
>> All the proxy ARP usages that I know of use the proxy's mac 
>> addr in the
>> ARP response. I believe the flavor of the proxy ARP you 
>> alluded below is
>> rare, and won't work very well anyway if the link address includes
any
>> non-persistent component like LID and QPN because it will 
>> require another
>> protocol between the clients and their proxy to constantly update the
>> proxy database containing clients mac addrs, which may change quite
>> often.
>> 
>> That is also why I wanted to provide an option to not include QPN.
>> (But I haven't succeeded w/o requiring SIDR or something equivalent.)
>> 
>> If LID is included as part of link addr and if it ever changes on
>> a live node due to subnet merge...etc, the driver must inform its IP
>> stack about such change. I believe many stacks don't have such a
>> facility. (FOr most IP stacks the assumption is that its own MAC
>> addresses just don't change on the fly.) In that case you're totally
>> hosed. Can someone comment on how likely a LID assignment gets
>> rescinded?
>
>It is possible anytime the link goes down and comes back up again and
when
>the fabic is reconfigured.  This can happen on reboot or simply
>unplugging/plugging in a link.  It is ultimately a function of the SM
and
>how much trouble the implementation wants to go through to keep LIDs
>constant.
>
>But of course, that has no bearing on whether the LID is contained in
the
>ARP.  You always need the LID.  Unless you want to do a pathrecord
lookup
>each time your implementation is given a packet to send, you're going
to use
>a cached LID no matter if it came in the ARP or a pathrecord lookup on
the
>GID.  If it becomes stale, it may take an ARP cache flush timeout to
>recover.  
>
>Roy
> 
>> 
>> Jerry
>> 
>> >        A device can always respond for another. The LID of 
>> the sender
>> >        is not the same as the LID for which it has 
>> responded. It also
>> >        allows any network monitoring/debugging device to work
>> >        independently. This is the reason why none of the other
media
>> >        does anything like this. You don't pick ethernet source
>> >        address from the frame but take it from the ARP/ND packet.
>> >
>> >        [ If it was ok we wouldn't need anything except
QPN/cap_flags
>> >        - get the LID from response, consult SA using LID for GID if
>> >        you want that, and you are done. ]
>> >
>> >Vivek
>> >
>> >> 
>> >> Why do we need to include the LID inside the link layer 
>> address when it
>> >> is already available in the SLID field? Previous 
>> discussions indicate
>> >> that the SLID value works for proxy ARP/bridge implementations.
>> >> 
>> >> - Arvind
>> >> 
>> >> -----Original Message-----
>> >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com] 
>> >> Sent: Wednesday, January 16, 2002 10:29 AM
>> >> To: kamath@seagullsemi.com
>> >> Cc: ipoverib@ietf.org
>> >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>> >> 
>> >> Ashwin,
>> >> Ashwin,
>> >> 
>> >>         I agree that you get more flexibility if you 
>> include the LID.
>> >>         If we include the GID along with the LID we allow all
>> >>         implementations the choice to create different 
>> solutions. We
>> >>         don't tie them down to one thing. Choosing one or the
other
>> >>         (LID or GID) blocks out various advantages that different
>> >>         implementations seek. Hence I suggested that let us
include
>> >>         both and move on.
>> >> 
>> >> Vivek
>> >> 
>> >> <snip>
>> >> 
>> >> _______________________________________________
>> >> IPoverIB mailing list
>> >> IPoverIB@ietf.org
>> >> http://www.ietf.org/mailman/listinfo/ipoverib
>> >> 
>> >
>> >
>> >-- 
>> >Vivek Kashyap
>> >Linux Technology Center, IBM
>> >kashyapv@us.ibm.com 
>> >vivk@us.ibm.com 
>> >503 578 3422 (o)
>> >
>> >_______________________________________________
>> >IPoverIB mailing list
>> >IPoverIB@ietf.org
>> >http://www.ietf.org/mailman/listinfo/ipoverib
>> 
>> 
>> _______________________________________________
>> IPoverIB mailing list
>> IPoverIB@ietf.org
>> http://www.ietf.org/mailman/listinfo/ipoverib
>> 

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 15:26:54 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA08174
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 15:26:54 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id PAA22520
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 15:26:57 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA22452;
	Thu, 17 Jan 2002 15:25:06 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA22423
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 15:25:04 -0500 (EST)
Received: from mail6.microsoft.com (mail6.microsoft.com [131.107.3.126])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA08060
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 15:25:00 -0500 (EST)
Received: from inet-vrs-06.redmond.corp.microsoft.com ([157.54.6.201]) by mail6.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Thu, 17 Jan 2002 12:24:31 -0800
Received: from 157.54.8.23 by inet-vrs-06.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Thu, 17 Jan 2002 12:24:31 -0800
Received: from red-imc-02.redmond.corp.microsoft.com ([157.54.9.107]) by inet-hub-01.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Thu, 17 Jan 2002 12:24:31 -0800
Received: from win-imc-02.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.84]) by red-imc-02.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Thu, 17 Jan 2002 12:24:24 -0800
Received: from win-msg-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.52]) by win-imc-02.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3588.0);
	 Thu, 17 Jan 2002 12:22:16 -0800
X-MimeOLE: Produced By Microsoft Exchange V6.0.6132.0
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
Date: Thu, 17 Jan 2002 12:22:16 -0800
Message-ID: <2E33960095B58E40A4D3345AB9F65EC10353E267@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
Thread-Topic: [Ipoverib] What's inside an IPoIB link layer address?
Thread-Index: AcGfB0UvmJua7wr4Edao5ABQi2jWzAAecbrgAAR3N1AAAHYa4A==
From: "Arvind Murching" <arvindm@windows.microsoft.com>
To: <ipoverib@ietf.org>
X-OriginalArrivalTime: 17 Jan 2002 20:22:16.0584 (UTC) FILETIME=[A8032880:01C19F94]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id PAA22424
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

Also note that you can't send your LID to a node on a different IB
subnet anyway.

-----Original Message-----
From: Arvind Murching 
Sent: Thursday, January 17, 2002 12:17 PM
To: ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?

I'm not an IB expert, but I would think that -if- we decided to use
gratuitious ARP to notify remote nodes of changes, then those remote
nodes would consult with the SA and re-create their address handles
using the indicated GID.

"Optimized/simplified" implementations would pick up new SLIDs from the
headers of these gratuitous ARP packets.

-----Original Message-----
From: Larsen, Roy K [mailto:roy.k.larsen@intel.com] 
Sent: Thursday, January 17, 2002 11:42 AM
To: 'H.K. Jerry Chu'; Arvind Murching; kashyapv@us.ibm.com
Cc: ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?

Jerry,

Ok, your concern is a mechanism for notifying a node that a LID change
has occurred regardless of how the LID was initially obtained.  I'm no
SM expert but seem to recall asking my resident expert about this a
while back and he said there wasn't.  However, if the LID was included
in the ARP, wouldn't sending a gratuitous ARP when the link comes up
solve the problem? 

Roy

-----Original Message-----
From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
Sent: Wednesday, January 16, 2002 7:28 PM
To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com;
roy.k.larsen@intel.com
Cc: ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?


Roy,

Thanks for the info. A comment on your statement of volatility of LID
having
no bearing on whether LID should be included in ARP reply. Yes it does.
One
of the goal of IPoIB is to minimize the changes to the existing IP stack
implementations. Note that I stated it as a goal, not a constraint. If
we
must do it then we should.

Yes no matter what, LID is cached somewhere and must be flushed. The
difference
is whether one must modify the existing stack code to add a facility to
support
mac addr change, or do the necessary chore in a complete new IPoIB
driver.
The former requires an upgrade of OS release that can take a long time.
The
latter doesn't have such a problem since it's brand new.

One can also hope that LID flushing through SM can be made more reliable
than
gratuitous ARP. But I don't know the SM archtecture enough to make such
a claim.

To be fair I don't really know how many OSes lack support for mac add
change. The two that I know, one supports it, one doesn't.

I'm assuming there are solid facility in IBA to support the notification
of
attribute changes to interested parties. Otherwise not only we should
include
LID, but also all the rest in the link addr.

Jerry

>From: "Larsen, Roy K" <roy.k.larsen@intel.com>
>To: "'H.K. Jerry Chu'" <Jerry.Chu@eng.sun.com>,
arvindm@windows.microsoft.com, kashyapv@us.ibm.com
>Cc: ipoverib@ietf.org
>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>Date: Wed, 16 Jan 2002 18:42:31 -0800
>MIME-Version: 1.0
>
>> -----Original Message-----
>> From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
>> Sent: Wednesday, January 16, 2002 6:13 PM
>> To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com
>> Cc: ipoverib@ietf.org
>> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>> 
>> 
>> All the proxy ARP usages that I know of use the proxy's mac 
>> addr in the
>> ARP response. I believe the flavor of the proxy ARP you 
>> alluded below is
>> rare, and won't work very well anyway if the link address includes
any
>> non-persistent component like LID and QPN because it will 
>> require another
>> protocol between the clients and their proxy to constantly update the
>> proxy database containing clients mac addrs, which may change quite
>> often.
>> 
>> That is also why I wanted to provide an option to not include QPN.
>> (But I haven't succeeded w/o requiring SIDR or something equivalent.)
>> 
>> If LID is included as part of link addr and if it ever changes on
>> a live node due to subnet merge...etc, the driver must inform its IP
>> stack about such change. I believe many stacks don't have such a
>> facility. (FOr most IP stacks the assumption is that its own MAC
>> addresses just don't change on the fly.) In that case you're totally
>> hosed. Can someone comment on how likely a LID assignment gets
>> rescinded?
>
>It is possible anytime the link goes down and comes back up again and
when
>the fabic is reconfigured.  This can happen on reboot or simply
>unplugging/plugging in a link.  It is ultimately a function of the SM
and
>how much trouble the implementation wants to go through to keep LIDs
>constant.
>
>But of course, that has no bearing on whether the LID is contained in
the
>ARP.  You always need the LID.  Unless you want to do a pathrecord
lookup
>each time your implementation is given a packet to send, you're going
to use
>a cached LID no matter if it came in the ARP or a pathrecord lookup on
the
>GID.  If it becomes stale, it may take an ARP cache flush timeout to
>recover.  
>
>Roy
> 
>> 
>> Jerry
>> 
>> >        A device can always respond for another. The LID of 
>> the sender
>> >        is not the same as the LID for which it has 
>> responded. It also
>> >        allows any network monitoring/debugging device to work
>> >        independently. This is the reason why none of the other
media
>> >        does anything like this. You don't pick ethernet source
>> >        address from the frame but take it from the ARP/ND packet.
>> >
>> >        [ If it was ok we wouldn't need anything except
QPN/cap_flags
>> >        - get the LID from response, consult SA using LID for GID if
>> >        you want that, and you are done. ]
>> >
>> >Vivek
>> >
>> >> 
>> >> Why do we need to include the LID inside the link layer 
>> address when it
>> >> is already available in the SLID field? Previous 
>> discussions indicate
>> >> that the SLID value works for proxy ARP/bridge implementations.
>> >> 
>> >> - Arvind
>> >> 
>> >> -----Original Message-----
>> >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com] 
>> >> Sent: Wednesday, January 16, 2002 10:29 AM
>> >> To: kamath@seagullsemi.com
>> >> Cc: ipoverib@ietf.org
>> >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>> >> 
>> >> Ashwin,
>> >> Ashwin,
>> >> 
>> >>         I agree that you get more flexibility if you 
>> include the LID.
>> >>         If we include the GID along with the LID we allow all
>> >>         implementations the choice to create different 
>> solutions. We
>> >>         don't tie them down to one thing. Choosing one or the
other
>> >>         (LID or GID) blocks out various advantages that different
>> >>         implementations seek. Hence I suggested that let us
include
>> >>         both and move on.
>> >> 
>> >> Vivek
>> >> 
>> >> <snip>
>> >> 
>> >> _______________________________________________
>> >> IPoverIB mailing list
>> >> IPoverIB@ietf.org
>> >> http://www.ietf.org/mailman/listinfo/ipoverib
>> >> 
>> >
>> >
>> >-- 
>> >Vivek Kashyap
>> >Linux Technology Center, IBM
>> >kashyapv@us.ibm.com 
>> >vivk@us.ibm.com 
>> >503 578 3422 (o)
>> >
>> >_______________________________________________
>> >IPoverIB mailing list
>> >IPoverIB@ietf.org
>> >http://www.ietf.org/mailman/listinfo/ipoverib
>> 
>> 
>> _______________________________________________
>> IPoverIB mailing list
>> IPoverIB@ietf.org
>> http://www.ietf.org/mailman/listinfo/ipoverib
>> 

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 15:53:11 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA12389
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 15:53:11 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id PAA23719
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 15:53:14 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA23685;
	Thu, 17 Jan 2002 15:51:48 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA23657
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 15:51:47 -0500 (EST)
Received: from patan.sun.com (patan.Sun.COM [192.18.98.43])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA12325
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 15:51:43 -0500 (EST)
Received: from sunmail1.Sun.COM ([129.145.1.2])
	by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id NAA06097;
	Thu, 17 Jan 2002 13:51:43 -0700 (MST)
Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.86.38])
	by sunmail1.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1-Sun.COM.mod.2) with ESMTP id MAA27488;
	Thu, 17 Jan 2002 12:52:56 -0800 (PST)
Received: from taipei (taipei.Eng.Sun.COM [129.146.85.178])
	by jurassic.eng.sun.com (8.12.2+Sun/8.12.2) with SMTP id g0HKpgqT734328;
	Thu, 17 Jan 2002 12:51:43 -0800 (PST)
Message-Id: <200201172051.g0HKpgqT734328@jurassic.eng.sun.com>
Date: Thu, 17 Jan 2002 12:49:09 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
To: roy.k.larsen@intel.com
Cc: ipoverib@ietf.org
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: xcgqS68HlVzUkv5EoeQrng==
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.6_06 SunOS 5.8 sun4u sparc 
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Roy,

Sorry I didn't explain it well. The potential issue I was trying to
point out is inside a local node that has its LID change. The question
is how will the LID change be propergated up to the IP stack. Many IP 
implementations never expect a local NIC's  MAC address can change on
the fly, all by itself. This poses a potential implementation problem.

Jerry

>From: "Larsen, Roy K" <roy.k.larsen@intel.com>
>To: "'H.K. Jerry Chu'" <Jerry.Chu@eng.sun.com>, arvindm@windows.microsoft.com, 
kashyapv@us.ibm.com
>Cc: ipoverib@ietf.org
>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>Date: Thu, 17 Jan 2002 11:42:20 -0800
>MIME-Version: 1.0
>
>Jerry,
>
>Ok, your concern is a mechanism for notifying a node that a LID change has
>occurred regardless of how the LID was initially obtained.  I'm no SM expert
>but seem to recall asking my resident expert about this a while back and he
>said there wasn't.  However, if the LID was included in the ARP, wouldn't
>sending a gratuitous ARP when the link comes up solve the problem? 
>
>Roy
>
>-----Original Message-----
>From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
>Sent: Wednesday, January 16, 2002 7:28 PM
>To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com;
>roy.k.larsen@intel.com
>Cc: ipoverib@ietf.org
>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>
>
>Roy,
>
>Thanks for the info. A comment on your statement of volatility of LID having
>no bearing on whether LID should be included in ARP reply. Yes it does. One
>of the goal of IPoIB is to minimize the changes to the existing IP stack
>implementations. Note that I stated it as a goal, not a constraint. If we
>must do it then we should.
>
>Yes no matter what, LID is cached somewhere and must be flushed. The
>difference
>is whether one must modify the existing stack code to add a facility to
>support
>mac addr change, or do the necessary chore in a complete new IPoIB driver.
>The former requires an upgrade of OS release that can take a long time. The
>latter doesn't have such a problem since it's brand new.
>
>One can also hope that LID flushing through SM can be made more reliable
>than
>gratuitous ARP. But I don't know the SM archtecture enough to make such a
>claim.
>
>To be fair I don't really know how many OSes lack support for mac add
>change. The two that I know, one supports it, one doesn't.
>
>I'm assuming there are solid facility in IBA to support the notification of
>attribute changes to interested parties. Otherwise not only we should
>include
>LID, but also all the rest in the link addr.
>
>Jerry
>
>>From: "Larsen, Roy K" <roy.k.larsen@intel.com>
>>To: "'H.K. Jerry Chu'" <Jerry.Chu@eng.sun.com>,
>arvindm@windows.microsoft.com, kashyapv@us.ibm.com
>>Cc: ipoverib@ietf.org
>>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>>Date: Wed, 16 Jan 2002 18:42:31 -0800
>>MIME-Version: 1.0
>>
>>> -----Original Message-----
>>> From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
>>> Sent: Wednesday, January 16, 2002 6:13 PM
>>> To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com
>>> Cc: ipoverib@ietf.org
>>> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>>> 
>>> 
>>> All the proxy ARP usages that I know of use the proxy's mac 
>>> addr in the
>>> ARP response. I believe the flavor of the proxy ARP you 
>>> alluded below is
>>> rare, and won't work very well anyway if the link address includes any
>>> non-persistent component like LID and QPN because it will 
>>> require another
>>> protocol between the clients and their proxy to constantly update the
>>> proxy database containing clients mac addrs, which may change quite
>>> often.
>>> 
>>> That is also why I wanted to provide an option to not include QPN.
>>> (But I haven't succeeded w/o requiring SIDR or something equivalent.)
>>> 
>>> If LID is included as part of link addr and if it ever changes on
>>> a live node due to subnet merge...etc, the driver must inform its IP
>>> stack about such change. I believe many stacks don't have such a
>>> facility. (FOr most IP stacks the assumption is that its own MAC
>>> addresses just don't change on the fly.) In that case you're totally
>>> hosed. Can someone comment on how likely a LID assignment gets
>>> rescinded?
>>
>>It is possible anytime the link goes down and comes back up again and when
>>the fabic is reconfigured.  This can happen on reboot or simply
>>unplugging/plugging in a link.  It is ultimately a function of the SM and
>>how much trouble the implementation wants to go through to keep LIDs
>>constant.
>>
>>But of course, that has no bearing on whether the LID is contained in the
>>ARP.  You always need the LID.  Unless you want to do a pathrecord lookup
>>each time your implementation is given a packet to send, you're going to
>use
>>a cached LID no matter if it came in the ARP or a pathrecord lookup on the
>>GID.  If it becomes stale, it may take an ARP cache flush timeout to
>>recover.  
>>
>>Roy
>> 
>>> 
>>> Jerry
>>> 
>>> >        A device can always respond for another. The LID of 
>>> the sender
>>> >        is not the same as the LID for which it has 
>>> responded. It also
>>> >        allows any network monitoring/debugging device to work
>>> >        independently. This is the reason why none of the other media
>>> >        does anything like this. You don't pick ethernet source
>>> >        address from the frame but take it from the ARP/ND packet.
>>> >
>>> >        [ If it was ok we wouldn't need anything except QPN/cap_flags
>>> >        - get the LID from response, consult SA using LID for GID if
>>> >        you want that, and you are done. ]
>>> >
>>> >Vivek
>>> >
>>> >> 
>>> >> Why do we need to include the LID inside the link layer 
>>> address when it
>>> >> is already available in the SLID field? Previous 
>>> discussions indicate
>>> >> that the SLID value works for proxy ARP/bridge implementations.
>>> >> 
>>> >> - Arvind
>>> >> 
>>> >> -----Original Message-----
>>> >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com] 
>>> >> Sent: Wednesday, January 16, 2002 10:29 AM
>>> >> To: kamath@seagullsemi.com
>>> >> Cc: ipoverib@ietf.org
>>> >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>>> >> 
>>> >> Ashwin,
>>> >> Ashwin,
>>> >> 
>>> >>         I agree that you get more flexibility if you 
>>> include the LID.
>>> >>         If we include the GID along with the LID we allow all
>>> >>         implementations the choice to create different 
>>> solutions. We
>>> >>         don't tie them down to one thing. Choosing one or the other
>>> >>         (LID or GID) blocks out various advantages that different
>>> >>         implementations seek. Hence I suggested that let us include
>>> >>         both and move on.
>>> >> 
>>> >> Vivek
>>> >> 
>>> >> <snip>
>>> >> 
>>> >> _______________________________________________
>>> >> IPoverIB mailing list
>>> >> IPoverIB@ietf.org
>>> >> http://www.ietf.org/mailman/listinfo/ipoverib
>>> >> 
>>> >
>>> >
>>> >-- 
>>> >Vivek Kashyap
>>> >Linux Technology Center, IBM
>>> >kashyapv@us.ibm.com 
>>> >vivk@us.ibm.com 
>>> >503 578 3422 (o)
>>> >
>>> >_______________________________________________
>>> >IPoverIB mailing list
>>> >IPoverIB@ietf.org
>>> >http://www.ietf.org/mailman/listinfo/ipoverib
>>> 
>>> 
>>> _______________________________________________
>>> IPoverIB mailing list
>>> IPoverIB@ietf.org
>>> http://www.ietf.org/mailman/listinfo/ipoverib
>>> 


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 16:01:22 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA12675
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 16:01:22 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id QAA24239
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 16:01:25 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA23832;
	Thu, 17 Jan 2002 15:59:52 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA23783
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 15:59:50 -0500 (EST)
Received: from hermes.fm.intel.com (fmr01.intel.com [192.55.52.18])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA12614
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 15:59:45 -0500 (EST)
Received: from petasus.fm.intel.com (petasus.fm.intel.com [10.1.192.37])
	by hermes.fm.intel.com (8.11.6/8.11.6/d: outer.mc,v 1.28 2002/01/02 21:40:45 root Exp $) with ESMTP id g0HKxMt01986
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 20:59:22 GMT
Received: from fmsmsxvs040.fm.intel.com (fmsmsxv040-1.fm.intel.com [132.233.48.108])
	by petasus.fm.intel.com (8.11.6/8.11.6/d: inner.mc,v 1.11 2001/11/09 23:28:01 root Exp $) with SMTP id g0HKxM714025
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 20:59:27 GMT
Received: from fmsmsx29.FM.INTEL.COM ([132.233.42.29])
 by fmsmsxvs040.fm.intel.com (NAVGW 2.5.1.16) with SMTP id M2002011712591207900
 ; Thu, 17 Jan 2002 12:59:12 -0800
Received: by fmsmsx29.fm.intel.com with Internet Mail Service (5.5.2653.19)
	id <C02FQ0GR>; Thu, 17 Jan 2002 12:59:41 -0800
Message-ID: <D9223EB959A5D511A98F00508B68C20C21A852@ORSMSX108>
From: "Larsen, Roy K" <roy.k.larsen@intel.com>
To: "'H.K. Jerry Chu'" <Jerry.Chu@eng.sun.com>
Cc: ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
Date: Thu, 17 Jan 2002 12:59:36 -0800
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="iso-8859-1"
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Jerry,

Yes, this could be a issue for the OS.  But since the dynamic nature of LIDs
is integral to IBA, they are just going to have to find a way to deal with
it.   As I suggested, perhaps with gratuitous ARP.  Anyway, would you agree
it is outside the scope of IPoIB.

Roy

-----Original Message-----
From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
Sent: Thursday, January 17, 2002 12:49 PM
To: roy.k.larsen@intel.com
Cc: ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?


Roy,

Sorry I didn't explain it well. The potential issue I was trying to
point out is inside a local node that has its LID change. The question
is how will the LID change be propergated up to the IP stack. Many IP 
implementations never expect a local NIC's  MAC address can change on
the fly, all by itself. This poses a potential implementation problem.

Jerry

>From: "Larsen, Roy K" <roy.k.larsen@intel.com>
>To: "'H.K. Jerry Chu'" <Jerry.Chu@eng.sun.com>,
arvindm@windows.microsoft.com, 
kashyapv@us.ibm.com
>Cc: ipoverib@ietf.org
>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>Date: Thu, 17 Jan 2002 11:42:20 -0800
>MIME-Version: 1.0
>
>Jerry,
>
>Ok, your concern is a mechanism for notifying a node that a LID change has
>occurred regardless of how the LID was initially obtained.  I'm no SM
expert
>but seem to recall asking my resident expert about this a while back and he
>said there wasn't.  However, if the LID was included in the ARP, wouldn't
>sending a gratuitous ARP when the link comes up solve the problem? 
>
>Roy
>
>-----Original Message-----
>From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
>Sent: Wednesday, January 16, 2002 7:28 PM
>To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com;
>roy.k.larsen@intel.com
>Cc: ipoverib@ietf.org
>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>
>
>Roy,
>
>Thanks for the info. A comment on your statement of volatility of LID
having
>no bearing on whether LID should be included in ARP reply. Yes it does. One
>of the goal of IPoIB is to minimize the changes to the existing IP stack
>implementations. Note that I stated it as a goal, not a constraint. If we
>must do it then we should.
>
>Yes no matter what, LID is cached somewhere and must be flushed. The
>difference
>is whether one must modify the existing stack code to add a facility to
>support
>mac addr change, or do the necessary chore in a complete new IPoIB driver.
>The former requires an upgrade of OS release that can take a long time. The
>latter doesn't have such a problem since it's brand new.
>
>One can also hope that LID flushing through SM can be made more reliable
>than
>gratuitous ARP. But I don't know the SM archtecture enough to make such a
>claim.
>
>To be fair I don't really know how many OSes lack support for mac add
>change. The two that I know, one supports it, one doesn't.
>
>I'm assuming there are solid facility in IBA to support the notification of
>attribute changes to interested parties. Otherwise not only we should
>include
>LID, but also all the rest in the link addr.
>
>Jerry
>
>>From: "Larsen, Roy K" <roy.k.larsen@intel.com>
>>To: "'H.K. Jerry Chu'" <Jerry.Chu@eng.sun.com>,
>arvindm@windows.microsoft.com, kashyapv@us.ibm.com
>>Cc: ipoverib@ietf.org
>>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>>Date: Wed, 16 Jan 2002 18:42:31 -0800
>>MIME-Version: 1.0
>>
>>> -----Original Message-----
>>> From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
>>> Sent: Wednesday, January 16, 2002 6:13 PM
>>> To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com
>>> Cc: ipoverib@ietf.org
>>> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>>> 
>>> 
>>> All the proxy ARP usages that I know of use the proxy's mac 
>>> addr in the
>>> ARP response. I believe the flavor of the proxy ARP you 
>>> alluded below is
>>> rare, and won't work very well anyway if the link address includes any
>>> non-persistent component like LID and QPN because it will 
>>> require another
>>> protocol between the clients and their proxy to constantly update the
>>> proxy database containing clients mac addrs, which may change quite
>>> often.
>>> 
>>> That is also why I wanted to provide an option to not include QPN.
>>> (But I haven't succeeded w/o requiring SIDR or something equivalent.)
>>> 
>>> If LID is included as part of link addr and if it ever changes on
>>> a live node due to subnet merge...etc, the driver must inform its IP
>>> stack about such change. I believe many stacks don't have such a
>>> facility. (FOr most IP stacks the assumption is that its own MAC
>>> addresses just don't change on the fly.) In that case you're totally
>>> hosed. Can someone comment on how likely a LID assignment gets
>>> rescinded?
>>
>>It is possible anytime the link goes down and comes back up again and when
>>the fabic is reconfigured.  This can happen on reboot or simply
>>unplugging/plugging in a link.  It is ultimately a function of the SM and
>>how much trouble the implementation wants to go through to keep LIDs
>>constant.
>>
>>But of course, that has no bearing on whether the LID is contained in the
>>ARP.  You always need the LID.  Unless you want to do a pathrecord lookup
>>each time your implementation is given a packet to send, you're going to
>use
>>a cached LID no matter if it came in the ARP or a pathrecord lookup on the
>>GID.  If it becomes stale, it may take an ARP cache flush timeout to
>>recover.  
>>
>>Roy
>> 
>>> 
>>> Jerry
>>> 
>>> >        A device can always respond for another. The LID of 
>>> the sender
>>> >        is not the same as the LID for which it has 
>>> responded. It also
>>> >        allows any network monitoring/debugging device to work
>>> >        independently. This is the reason why none of the other media
>>> >        does anything like this. You don't pick ethernet source
>>> >        address from the frame but take it from the ARP/ND packet.
>>> >
>>> >        [ If it was ok we wouldn't need anything except QPN/cap_flags
>>> >        - get the LID from response, consult SA using LID for GID if
>>> >        you want that, and you are done. ]
>>> >
>>> >Vivek
>>> >
>>> >> 
>>> >> Why do we need to include the LID inside the link layer 
>>> address when it
>>> >> is already available in the SLID field? Previous 
>>> discussions indicate
>>> >> that the SLID value works for proxy ARP/bridge implementations.
>>> >> 
>>> >> - Arvind
>>> >> 
>>> >> -----Original Message-----
>>> >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com] 
>>> >> Sent: Wednesday, January 16, 2002 10:29 AM
>>> >> To: kamath@seagullsemi.com
>>> >> Cc: ipoverib@ietf.org
>>> >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>>> >> 
>>> >> Ashwin,
>>> >> Ashwin,
>>> >> 
>>> >>         I agree that you get more flexibility if you 
>>> include the LID.
>>> >>         If we include the GID along with the LID we allow all
>>> >>         implementations the choice to create different 
>>> solutions. We
>>> >>         don't tie them down to one thing. Choosing one or the other
>>> >>         (LID or GID) blocks out various advantages that different
>>> >>         implementations seek. Hence I suggested that let us include
>>> >>         both and move on.
>>> >> 
>>> >> Vivek
>>> >> 
>>> >> <snip>
>>> >> 
>>> >> _______________________________________________
>>> >> IPoverIB mailing list
>>> >> IPoverIB@ietf.org
>>> >> http://www.ietf.org/mailman/listinfo/ipoverib
>>> >> 
>>> >
>>> >
>>> >-- 
>>> >Vivek Kashyap
>>> >Linux Technology Center, IBM
>>> >kashyapv@us.ibm.com 
>>> >vivk@us.ibm.com 
>>> >503 578 3422 (o)
>>> >
>>> >_______________________________________________
>>> >IPoverIB mailing list
>>> >IPoverIB@ietf.org
>>> >http://www.ietf.org/mailman/listinfo/ipoverib
>>> 
>>> 
>>> _______________________________________________
>>> IPoverIB mailing list
>>> IPoverIB@ietf.org
>>> http://www.ietf.org/mailman/listinfo/ipoverib
>>> 

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 16:34:31 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA13631
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 16:34:30 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id QAA25691
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 16:34:34 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA25038;
	Thu, 17 Jan 2002 16:28:24 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA25006
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 16:28:22 -0500 (EST)
Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA13392
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 16:28:18 -0500 (EST)
Received: from sunmail1.Sun.COM ([129.145.1.2])
	by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id NAA14441;
	Thu, 17 Jan 2002 13:28:19 -0800 (PST)
Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.87.31])
	by sunmail1.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1-Sun.COM.mod.2) with ESMTP id NAA13455;
	Thu, 17 Jan 2002 13:29:31 -0800 (PST)
Received: from taipei (taipei.Eng.Sun.COM [129.146.85.178])
	by jurassic.eng.sun.com (8.12.2+Sun/8.12.2) with SMTP id g0HLSHqT741486;
	Thu, 17 Jan 2002 13:28:18 -0800 (PST)
Message-Id: <200201172128.g0HLSHqT741486@jurassic.eng.sun.com>
Date: Thu, 17 Jan 2002 13:25:44 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
To: roy.k.larsen@intel.com
Cc: ipoverib@ietf.org
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: SrkybqOqt4/6/hEK80u8NQ==
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.6_06 SunOS 5.8 sun4u sparc 
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

>Jerry,
>
>Yes, this could be a issue for the OS.  But since the dynamic nature of LIDs
>is integral to IBA, they are just going to have to find a way to deal with

And I've been saying this is an IBA issue in general (not just for IPoIB).
So IBA should find a solution for it.

Including the LID in the link addr forces this issue into the IP stack.

>it.   As I suggested, perhaps with gratuitous ARP.

No, forget about gratuitous ARP. The issue I brought up is purely a
issue within the local node that has its own LID changed.

>Anyway, would you agree
>it is outside the scope of IPoIB.

It's an iffy (IMHO). As I've pointed out before, I believe there is a
concensus that IPoIB should fit into the existing IP stack implementations
But that is a "goal", not a constraint. Should we make an exception for
LID?

Not including LID in link addr punts the changing LID problem back to
IB, which I think is the right place for it.

Jerry

>
>Roy
>
>-----Original Message-----
>From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
>Sent: Thursday, January 17, 2002 12:49 PM
>To: roy.k.larsen@intel.com
>Cc: ipoverib@ietf.org
>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>
>
>Roy,
>
>Sorry I didn't explain it well. The potential issue I was trying to
>point out is inside a local node that has its LID change. The question
>is how will the LID change be propergated up to the IP stack. Many IP 
>implementations never expect a local NIC's  MAC address can change on
>the fly, all by itself. This poses a potential implementation problem.
>
>Jerry
>
>>From: "Larsen, Roy K" <roy.k.larsen@intel.com>
>>To: "'H.K. Jerry Chu'" <Jerry.Chu@eng.sun.com>,
>arvindm@windows.microsoft.com, 
>kashyapv@us.ibm.com
>>Cc: ipoverib@ietf.org
>>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>>Date: Thu, 17 Jan 2002 11:42:20 -0800
>>MIME-Version: 1.0
>>
>>Jerry,
>>
>>Ok, your concern is a mechanism for notifying a node that a LID change has
>>occurred regardless of how the LID was initially obtained.  I'm no SM
>expert
>>but seem to recall asking my resident expert about this a while back and he
>>said there wasn't.  However, if the LID was included in the ARP, wouldn't
>>sending a gratuitous ARP when the link comes up solve the problem? 
>>
>>Roy
>>
>>-----Original Message-----
>>From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
>>Sent: Wednesday, January 16, 2002 7:28 PM
>>To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com;
>>roy.k.larsen@intel.com
>>Cc: ipoverib@ietf.org
>>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>>
>>
>>Roy,
>>
>>Thanks for the info. A comment on your statement of volatility of LID
>having
>>no bearing on whether LID should be included in ARP reply. Yes it does. One
>>of the goal of IPoIB is to minimize the changes to the existing IP stack
>>implementations. Note that I stated it as a goal, not a constraint. If we
>>must do it then we should.
>>
>>Yes no matter what, LID is cached somewhere and must be flushed. The
>>difference
>>is whether one must modify the existing stack code to add a facility to
>>support
>>mac addr change, or do the necessary chore in a complete new IPoIB driver.
>>The former requires an upgrade of OS release that can take a long time. The
>>latter doesn't have such a problem since it's brand new.
>>
>>One can also hope that LID flushing through SM can be made more reliable
>>than
>>gratuitous ARP. But I don't know the SM archtecture enough to make such a
>>claim.
>>
>>To be fair I don't really know how many OSes lack support for mac add
>>change. The two that I know, one supports it, one doesn't.
>>
>>I'm assuming there are solid facility in IBA to support the notification of
>>attribute changes to interested parties. Otherwise not only we should
>>include
>>LID, but also all the rest in the link addr.
>>
>>Jerry
>>
>>>From: "Larsen, Roy K" <roy.k.larsen@intel.com>
>>>To: "'H.K. Jerry Chu'" <Jerry.Chu@eng.sun.com>,
>>arvindm@windows.microsoft.com, kashyapv@us.ibm.com
>>>Cc: ipoverib@ietf.org
>>>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>>>Date: Wed, 16 Jan 2002 18:42:31 -0800
>>>MIME-Version: 1.0
>>>
>>>> -----Original Message-----
>>>> From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
>>>> Sent: Wednesday, January 16, 2002 6:13 PM
>>>> To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com
>>>> Cc: ipoverib@ietf.org
>>>> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>>>> 
>>>> 
>>>> All the proxy ARP usages that I know of use the proxy's mac 
>>>> addr in the
>>>> ARP response. I believe the flavor of the proxy ARP you 
>>>> alluded below is
>>>> rare, and won't work very well anyway if the link address includes any
>>>> non-persistent component like LID and QPN because it will 
>>>> require another
>>>> protocol between the clients and their proxy to constantly update the
>>>> proxy database containing clients mac addrs, which may change quite
>>>> often.
>>>> 
>>>> That is also why I wanted to provide an option to not include QPN.
>>>> (But I haven't succeeded w/o requiring SIDR or something equivalent.)
>>>> 
>>>> If LID is included as part of link addr and if it ever changes on
>>>> a live node due to subnet merge...etc, the driver must inform its IP
>>>> stack about such change. I believe many stacks don't have such a
>>>> facility. (FOr most IP stacks the assumption is that its own MAC
>>>> addresses just don't change on the fly.) In that case you're totally
>>>> hosed. Can someone comment on how likely a LID assignment gets
>>>> rescinded?
>>>
>>>It is possible anytime the link goes down and comes back up again and when
>>>the fabic is reconfigured.  This can happen on reboot or simply
>>>unplugging/plugging in a link.  It is ultimately a function of the SM and
>>>how much trouble the implementation wants to go through to keep LIDs
>>>constant.
>>>
>>>But of course, that has no bearing on whether the LID is contained in the
>>>ARP.  You always need the LID.  Unless you want to do a pathrecord lookup
>>>each time your implementation is given a packet to send, you're going to
>>use
>>>a cached LID no matter if it came in the ARP or a pathrecord lookup on the
>>>GID.  If it becomes stale, it may take an ARP cache flush timeout to
>>>recover.  
>>>
>>>Roy
>>> 
>>>> 
>>>> Jerry
>>>> 
>>>> >        A device can always respond for another. The LID of 
>>>> the sender
>>>> >        is not the same as the LID for which it has 
>>>> responded. It also
>>>> >        allows any network monitoring/debugging device to work
>>>> >        independently. This is the reason why none of the other media
>>>> >        does anything like this. You don't pick ethernet source
>>>> >        address from the frame but take it from the ARP/ND packet.
>>>> >
>>>> >        [ If it was ok we wouldn't need anything except QPN/cap_flags
>>>> >        - get the LID from response, consult SA using LID for GID if
>>>> >        you want that, and you are done. ]
>>>> >
>>>> >Vivek
>>>> >
>>>> >> 
>>>> >> Why do we need to include the LID inside the link layer 
>>>> address when it
>>>> >> is already available in the SLID field? Previous 
>>>> discussions indicate
>>>> >> that the SLID value works for proxy ARP/bridge implementations.
>>>> >> 
>>>> >> - Arvind
>>>> >> 
>>>> >> -----Original Message-----
>>>> >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com] 
>>>> >> Sent: Wednesday, January 16, 2002 10:29 AM
>>>> >> To: kamath@seagullsemi.com
>>>> >> Cc: ipoverib@ietf.org
>>>> >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>>>> >> 
>>>> >> Ashwin,
>>>> >> Ashwin,
>>>> >> 
>>>> >>         I agree that you get more flexibility if you 
>>>> include the LID.
>>>> >>         If we include the GID along with the LID we allow all
>>>> >>         implementations the choice to create different 
>>>> solutions. We
>>>> >>         don't tie them down to one thing. Choosing one or the other
>>>> >>         (LID or GID) blocks out various advantages that different
>>>> >>         implementations seek. Hence I suggested that let us include
>>>> >>         both and move on.
>>>> >> 
>>>> >> Vivek
>>>> >> 
>>>> >> <snip>
>>>> >> 
>>>> >> _______________________________________________
>>>> >> IPoverIB mailing list
>>>> >> IPoverIB@ietf.org
>>>> >> http://www.ietf.org/mailman/listinfo/ipoverib
>>>> >> 
>>>> >
>>>> >
>>>> >-- 
>>>> >Vivek Kashyap
>>>> >Linux Technology Center, IBM
>>>> >kashyapv@us.ibm.com 
>>>> >vivk@us.ibm.com 
>>>> >503 578 3422 (o)
>>>> >
>>>> >_______________________________________________
>>>> >IPoverIB mailing list
>>>> >IPoverIB@ietf.org
>>>> >http://www.ietf.org/mailman/listinfo/ipoverib
>>>> 
>>>> 
>>>> _______________________________________________
>>>> IPoverIB mailing list
>>>> IPoverIB@ietf.org
>>>> http://www.ietf.org/mailman/listinfo/ipoverib
>>>> 


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 16:40:42 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA13814
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 16:40:42 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id QAA25847
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 16:40:46 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA25678;
	Thu, 17 Jan 2002 16:34:16 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA25629
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 16:34:14 -0500 (EST)
Received: from pdxpo.dsl-only.net (pdxpo.dsl-only.net [63.105.16.3])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA13621
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 16:34:09 -0500 (EST)
Received: from bonfire (unverified [63.105.18.201]) by pdxpo.dsl-only.net
 (Rockliffe SMTPRA 4.5.4) with SMTP id <B0009229362@pdxpo.dsl-only.net>;
 Thu, 17 Jan 2002 13:32:06 -0800
From: "Bill Strahm" <bill@strahm.net>
To: "Larsen, Roy K" <roy.k.larsen@intel.com>,
        "'H.K. Jerry Chu'" <Jerry.Chu@eng.sun.com>
Cc: <ipoverib@ietf.org>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
Date: Thu, 17 Jan 2002 13:41:21 -0800
Message-ID: <HHEDIJGOEFLPCPNILMPAKEAJCFAA.bill@strahm.net>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0)
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
Importance: Normal
In-Reply-To: <D9223EB959A5D511A98F00508B68C20C21A852@ORSMSX108>
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Roy,

Gratuitous ARP does not help here... I do I send a gratuitous ARP if I can
not change my MAC address in my ARP module...

The easiest solution is to not put LID into the MAC address, then you don't
have to worry about the problem at all on the sender side.  On the receiver
side I believe you have to do the same work regardless...

Bill

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Larsen, Roy K
Sent: Thursday, January 17, 2002 1:00 PM
To: 'H.K. Jerry Chu'
Cc: ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?


Jerry,

Yes, this could be a issue for the OS.  But since the dynamic nature of LIDs
is integral to IBA, they are just going to have to find a way to deal with
it.   As I suggested, perhaps with gratuitous ARP.  Anyway, would you agree
it is outside the scope of IPoIB.

Roy

-----Original Message-----
From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
Sent: Thursday, January 17, 2002 12:49 PM
To: roy.k.larsen@intel.com
Cc: ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?


Roy,

Sorry I didn't explain it well. The potential issue I was trying to
point out is inside a local node that has its LID change. The question
is how will the LID change be propergated up to the IP stack. Many IP
implementations never expect a local NIC's  MAC address can change on
the fly, all by itself. This poses a potential implementation problem.

Jerry

>From: "Larsen, Roy K" <roy.k.larsen@intel.com>
>To: "'H.K. Jerry Chu'" <Jerry.Chu@eng.sun.com>,
arvindm@windows.microsoft.com,
kashyapv@us.ibm.com
>Cc: ipoverib@ietf.org
>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>Date: Thu, 17 Jan 2002 11:42:20 -0800
>MIME-Version: 1.0
>
>Jerry,
>
>Ok, your concern is a mechanism for notifying a node that a LID change has
>occurred regardless of how the LID was initially obtained.  I'm no SM
expert
>but seem to recall asking my resident expert about this a while back and he
>said there wasn't.  However, if the LID was included in the ARP, wouldn't
>sending a gratuitous ARP when the link comes up solve the problem?
>
>Roy
>
>-----Original Message-----
>From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
>Sent: Wednesday, January 16, 2002 7:28 PM
>To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com;
>roy.k.larsen@intel.com
>Cc: ipoverib@ietf.org
>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>
>
>Roy,
>
>Thanks for the info. A comment on your statement of volatility of LID
having
>no bearing on whether LID should be included in ARP reply. Yes it does. One
>of the goal of IPoIB is to minimize the changes to the existing IP stack
>implementations. Note that I stated it as a goal, not a constraint. If we
>must do it then we should.
>
>Yes no matter what, LID is cached somewhere and must be flushed. The
>difference
>is whether one must modify the existing stack code to add a facility to
>support
>mac addr change, or do the necessary chore in a complete new IPoIB driver.
>The former requires an upgrade of OS release that can take a long time. The
>latter doesn't have such a problem since it's brand new.
>
>One can also hope that LID flushing through SM can be made more reliable
>than
>gratuitous ARP. But I don't know the SM archtecture enough to make such a
>claim.
>
>To be fair I don't really know how many OSes lack support for mac add
>change. The two that I know, one supports it, one doesn't.
>
>I'm assuming there are solid facility in IBA to support the notification of
>attribute changes to interested parties. Otherwise not only we should
>include
>LID, but also all the rest in the link addr.
>
>Jerry
>
>>From: "Larsen, Roy K" <roy.k.larsen@intel.com>
>>To: "'H.K. Jerry Chu'" <Jerry.Chu@eng.sun.com>,
>arvindm@windows.microsoft.com, kashyapv@us.ibm.com
>>Cc: ipoverib@ietf.org
>>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>>Date: Wed, 16 Jan 2002 18:42:31 -0800
>>MIME-Version: 1.0
>>
>>> -----Original Message-----
>>> From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
>>> Sent: Wednesday, January 16, 2002 6:13 PM
>>> To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com
>>> Cc: ipoverib@ietf.org
>>> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>>>
>>>
>>> All the proxy ARP usages that I know of use the proxy's mac
>>> addr in the
>>> ARP response. I believe the flavor of the proxy ARP you
>>> alluded below is
>>> rare, and won't work very well anyway if the link address includes any
>>> non-persistent component like LID and QPN because it will
>>> require another
>>> protocol between the clients and their proxy to constantly update the
>>> proxy database containing clients mac addrs, which may change quite
>>> often.
>>>
>>> That is also why I wanted to provide an option to not include QPN.
>>> (But I haven't succeeded w/o requiring SIDR or something equivalent.)
>>>
>>> If LID is included as part of link addr and if it ever changes on
>>> a live node due to subnet merge...etc, the driver must inform its IP
>>> stack about such change. I believe many stacks don't have such a
>>> facility. (FOr most IP stacks the assumption is that its own MAC
>>> addresses just don't change on the fly.) In that case you're totally
>>> hosed. Can someone comment on how likely a LID assignment gets
>>> rescinded?
>>
>>It is possible anytime the link goes down and comes back up again and when
>>the fabic is reconfigured.  This can happen on reboot or simply
>>unplugging/plugging in a link.  It is ultimately a function of the SM and
>>how much trouble the implementation wants to go through to keep LIDs
>>constant.
>>
>>But of course, that has no bearing on whether the LID is contained in the
>>ARP.  You always need the LID.  Unless you want to do a pathrecord lookup
>>each time your implementation is given a packet to send, you're going to
>use
>>a cached LID no matter if it came in the ARP or a pathrecord lookup on the
>>GID.  If it becomes stale, it may take an ARP cache flush timeout to
>>recover.
>>
>>Roy
>>
>>>
>>> Jerry
>>>
>>> >        A device can always respond for another. The LID of
>>> the sender
>>> >        is not the same as the LID for which it has
>>> responded. It also
>>> >        allows any network monitoring/debugging device to work
>>> >        independently. This is the reason why none of the other media
>>> >        does anything like this. You don't pick ethernet source
>>> >        address from the frame but take it from the ARP/ND packet.
>>> >
>>> >        [ If it was ok we wouldn't need anything except QPN/cap_flags
>>> >        - get the LID from response, consult SA using LID for GID if
>>> >        you want that, and you are done. ]
>>> >
>>> >Vivek
>>> >
>>> >>
>>> >> Why do we need to include the LID inside the link layer
>>> address when it
>>> >> is already available in the SLID field? Previous
>>> discussions indicate
>>> >> that the SLID value works for proxy ARP/bridge implementations.
>>> >>
>>> >> - Arvind
>>> >>
>>> >> -----Original Message-----
>>> >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
>>> >> Sent: Wednesday, January 16, 2002 10:29 AM
>>> >> To: kamath@seagullsemi.com
>>> >> Cc: ipoverib@ietf.org
>>> >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>>> >>
>>> >> Ashwin,
>>> >> Ashwin,
>>> >>
>>> >>         I agree that you get more flexibility if you
>>> include the LID.
>>> >>         If we include the GID along with the LID we allow all
>>> >>         implementations the choice to create different
>>> solutions. We
>>> >>         don't tie them down to one thing. Choosing one or the other
>>> >>         (LID or GID) blocks out various advantages that different
>>> >>         implementations seek. Hence I suggested that let us include
>>> >>         both and move on.
>>> >>
>>> >> Vivek
>>> >>
>>> >> <snip>
>>> >>
>>> >> _______________________________________________
>>> >> IPoverIB mailing list
>>> >> IPoverIB@ietf.org
>>> >> http://www.ietf.org/mailman/listinfo/ipoverib
>>> >>
>>> >
>>> >
>>> >--
>>> >Vivek Kashyap
>>> >Linux Technology Center, IBM
>>> >kashyapv@us.ibm.com
>>> >vivk@us.ibm.com
>>> >503 578 3422 (o)
>>> >
>>> >_______________________________________________
>>> >IPoverIB mailing list
>>> >IPoverIB@ietf.org
>>> >http://www.ietf.org/mailman/listinfo/ipoverib
>>>
>>>
>>> _______________________________________________
>>> IPoverIB mailing list
>>> IPoverIB@ietf.org
>>> http://www.ietf.org/mailman/listinfo/ipoverib
>>>

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 17:50:36 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA14914
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 17:50:36 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id RAA28336
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 17:50:38 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA28239;
	Thu, 17 Jan 2002 17:44:45 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA28180
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 17:44:43 -0500 (EST)
Received: from e33.co.us.ibm.com (e33.co.us.ibm.com [32.97.110.131])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA14784;
	Thu, 17 Jan 2002 17:44:39 -0500 (EST)
Received: from westrelay01.boulder.ibm.com (westrelay01.boulder.ibm.com [9.99.140.22])
	by e33.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id RAA78434;
	Thu, 17 Jan 2002 17:41:39 -0500
Received: from d03nm036.boulder.ibm.com (avpilot.boulder.ibm.com [9.17.188.135])
	by westrelay01.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0HMid7113612;
	Thu, 17 Jan 2002 15:44:39 -0700
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Cc: ipoverib@ietf.org, ipoverib-admin@ietf.org, roy.k.larsen@intel.com
X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000
Message-ID: <OFBDB8DFB8.D6A73BFE-ON88256B44.007A8E21@boulder.ibm.com>
From: "Vivek Kashyap" <vivk@us.ibm.com>
Date: Thu, 17 Jan 2002 14:43:18 -0800
X-MIMETrack: Serialize by Router on D03NM036/03/M/IBM(Release 5.0.9 |November 16, 2001) at
 01/17/2002 03:44:39 PM
MIME-Version: 1.0
Content-type: text/plain; charset=us-ascii
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org


Jerry,

I believe Roy has a point. We are dealing with a new technology and need to
work with its features. If the MAC changes the OS has to find ways to deal
with it.

Are you saying that the MAC will change, the OS will not know of it, will
return the GID only and the receiver must therefore go to the SM/SA to get
the LID ? That seems all wrong. If the OS doesn't know what the new LID is
it cannot send the packet at all. An implementation not recognising this
will fail - not just IPoIB but any IB application. The OS knows the LID
changed and all it does is update the necessary table.

BTW,  Roger  has pointed out that the LIDs will hardly ever change so the
concern regarding LID in this case is unfounded. On the contrary the GID
used may change more often based on the proposals seen on this mailing list
- a) use GIDs based on scope b) use non-GUID based GIDs for various reasons
etc. c)  One might also use the chassis' GID or the GID of a group etc. As
far as I can tell a port can have more than one GUID too.  These are more
variable. Any change here is more likely. Since the LID is uniquely tied to
a port but not the GID, return it in the ARP packet and the resolution is
complete.

 LID is the real address uniquely assigned to a port. Not the GID.  You
don't need a GID to talk to another port. You need the LID only. GID is
just an optional item.

[ Now coming to the gratuitous ARP case. If the LID had changed but not the
GID - a gratuitous ARP will succeed only if the receiver actually went to
the SA. This implies that one has to go to the SA on every ARP request one
gets not just when the cache expires. Otherwise such LID changes would
never get through to the peer in time. ]

Vivek
--
Vivek Kashyap
Linux Technology Center, IBM
vivk@us.ibm.com
kashyapv@us.ibm.com
Ph: 503 578 3422 T/L: 775 3422



                                                                                                                                  
                      "H.K. Jerry Chu"                                                                                            
                      <Jerry.Chu@eng.su        To:       roy.k.larsen@intel.com                                                   
                      n.com>                   cc:       ipoverib@ietf.org                                                        
                      Sent by:                 Subject:  RE: [Ipoverib] What's inside an IPoIB link layer address?                
                      ipoverib-admin@ie                                                                                           
                      tf.org                                                                                                      
                                                                                                                                  
                                                                                                                                  
                      01/17/2002 01:25                                                                                            
                      PM                                                                                                          
                      Please respond to                                                                                           
                      "H.K. Jerry Chu"                                                                                            
                                                                                                                                  
                                                                                                                                  



>Jerry,
>
>Yes, this could be a issue for the OS.  But since the dynamic nature of
LIDs
>is integral to IBA, they are just going to have to find a way to deal with

And I've been saying this is an IBA issue in general (not just for IPoIB).
So IBA should find a solution for it.

Including the LID in the link addr forces this issue into the IP stack.

>it.   As I suggested, perhaps with gratuitous ARP.

No, forget about gratuitous ARP. The issue I brought up is purely a
issue within the local node that has its own LID changed.

>Anyway, would you agree
>it is outside the scope of IPoIB.

It's an iffy (IMHO). As I've pointed out before, I believe there is a
concensus that IPoIB should fit into the existing IP stack implementations
But that is a "goal", not a constraint. Should we make an exception for
LID?

Not including LID in link addr punts the changing LID problem back to
IB, which I think is the right place for it.

Jerry

>
>Roy
>
>-----Original Message-----
>From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
>Sent: Thursday, January 17, 2002 12:49 PM
>To: roy.k.larsen@intel.com
>Cc: ipoverib@ietf.org
>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>
>
>Roy,
>
>Sorry I didn't explain it well. The potential issue I was trying to
>point out is inside a local node that has its LID change. The question
>is how will the LID change be propergated up to the IP stack. Many IP
>implementations never expect a local NIC's  MAC address can change on
>the fly, all by itself. This poses a potential implementation problem.
>
>Jerry
>
>>From: "Larsen, Roy K" <roy.k.larsen@intel.com>
>>To: "'H.K. Jerry Chu'" <Jerry.Chu@eng.sun.com>,
>arvindm@windows.microsoft.com,
>kashyapv@us.ibm.com
>>Cc: ipoverib@ietf.org
>>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>>Date: Thu, 17 Jan 2002 11:42:20 -0800
>>MIME-Version: 1.0
>>
>>Jerry,
>>
>>Ok, your concern is a mechanism for notifying a node that a LID change
has
>>occurred regardless of how the LID was initially obtained.  I'm no SM
>expert
>>but seem to recall asking my resident expert about this a while back and
he
>>said there wasn't.  However, if the LID was included in the ARP, wouldn't
>>sending a gratuitous ARP when the link comes up solve the problem?
>>
>>Roy
>>
>>-----Original Message-----
>>From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
>>Sent: Wednesday, January 16, 2002 7:28 PM
>>To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com;
>>roy.k.larsen@intel.com
>>Cc: ipoverib@ietf.org
>>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>>
>>
>>Roy,
>>
>>Thanks for the info. A comment on your statement of volatility of LID
>having
>>no bearing on whether LID should be included in ARP reply. Yes it does.
One
>>of the goal of IPoIB is to minimize the changes to the existing IP stack
>>implementations. Note that I stated it as a goal, not a constraint. If we
>>must do it then we should.
>>
>>Yes no matter what, LID is cached somewhere and must be flushed. The
>>difference
>>is whether one must modify the existing stack code to add a facility to
>>support
>>mac addr change, or do the necessary chore in a complete new IPoIB
driver.
>>The former requires an upgrade of OS release that can take a long time.
The
>>latter doesn't have such a problem since it's brand new.
>>
>>One can also hope that LID flushing through SM can be made more reliable
>>than
>>gratuitous ARP. But I don't know the SM archtecture enough to make such a
>>claim.
>>
>>To be fair I don't really know how many OSes lack support for mac add
>>change. The two that I know, one supports it, one doesn't.
>>
>>I'm assuming there are solid facility in IBA to support the notification
of
>>attribute changes to interested parties. Otherwise not only we should
>>include
>>LID, but also all the rest in the link addr.
>>
>>Jerry
>>
>>>From: "Larsen, Roy K" <roy.k.larsen@intel.com>
>>>To: "'H.K. Jerry Chu'" <Jerry.Chu@eng.sun.com>,
>>arvindm@windows.microsoft.com, kashyapv@us.ibm.com
>>>Cc: ipoverib@ietf.org
>>>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>>>Date: Wed, 16 Jan 2002 18:42:31 -0800
>>>MIME-Version: 1.0
>>>
>>>> -----Original Message-----
>>>> From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
>>>> Sent: Wednesday, January 16, 2002 6:13 PM
>>>> To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com
>>>> Cc: ipoverib@ietf.org
>>>> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>>>>
>>>>
>>>> All the proxy ARP usages that I know of use the proxy's mac
>>>> addr in the
>>>> ARP response. I believe the flavor of the proxy ARP you
>>>> alluded below is
>>>> rare, and won't work very well anyway if the link address includes any
>>>> non-persistent component like LID and QPN because it will
>>>> require another
>>>> protocol between the clients and their proxy to constantly update the
>>>> proxy database containing clients mac addrs, which may change quite
>>>> often.
>>>>
>>>> That is also why I wanted to provide an option to not include QPN.
>>>> (But I haven't succeeded w/o requiring SIDR or something equivalent.)
>>>>
>>>> If LID is included as part of link addr and if it ever changes on
>>>> a live node due to subnet merge...etc, the driver must inform its IP
>>>> stack about such change. I believe many stacks don't have such a
>>>> facility. (FOr most IP stacks the assumption is that its own MAC
>>>> addresses just don't change on the fly.) In that case you're totally
>>>> hosed. Can someone comment on how likely a LID assignment gets
>>>> rescinded?
>>>
>>>It is possible anytime the link goes down and comes back up again and
when
>>>the fabic is reconfigured.  This can happen on reboot or simply
>>>unplugging/plugging in a link.  It is ultimately a function of the SM
and
>>>how much trouble the implementation wants to go through to keep LIDs
>>>constant.
>>>
>>>But of course, that has no bearing on whether the LID is contained in
the
>>>ARP.  You always need the LID.  Unless you want to do a pathrecord
lookup
>>>each time your implementation is given a packet to send, you're going to
>>use
>>>a cached LID no matter if it came in the ARP or a pathrecord lookup on
the
>>>GID.  If it becomes stale, it may take an ARP cache flush timeout to
>>>recover.
>>>
>>>Roy
>>>
>>>>
>>>> Jerry
>>>>
>>>> >        A device can always respond for another. The LID of
>>>> the sender
>>>> >        is not the same as the LID for which it has
>>>> responded. It also
>>>> >        allows any network monitoring/debugging device to work
>>>> >        independently. This is the reason why none of the other media
>>>> >        does anything like this. You don't pick ethernet source
>>>> >        address from the frame but take it from the ARP/ND packet.
>>>> >
>>>> >        [ If it was ok we wouldn't need anything except QPN/cap_flags
>>>> >        - get the LID from response, consult SA using LID for GID if
>>>> >        you want that, and you are done. ]
>>>> >
>>>> >Vivek
>>>> >
>>>> >>
>>>> >> Why do we need to include the LID inside the link layer
>>>> address when it
>>>> >> is already available in the SLID field? Previous
>>>> discussions indicate
>>>> >> that the SLID value works for proxy ARP/bridge implementations.
>>>> >>
>>>> >> - Arvind
>>>> >>
>>>> >> -----Original Message-----
>>>> >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
>>>> >> Sent: Wednesday, January 16, 2002 10:29 AM
>>>> >> To: kamath@seagullsemi.com
>>>> >> Cc: ipoverib@ietf.org
>>>> >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>>>> >>
>>>> >> Ashwin,
>>>> >> Ashwin,
>>>> >>
>>>> >>         I agree that you get more flexibility if you
>>>> include the LID.
>>>> >>         If we include the GID along with the LID we allow all
>>>> >>         implementations the choice to create different
>>>> solutions. We
>>>> >>         don't tie them down to one thing. Choosing one or the other
>>>> >>         (LID or GID) blocks out various advantages that different
>>>> >>         implementations seek. Hence I suggested that let us include
>>>> >>         both and move on.
>>>> >>
>>>> >> Vivek
>>>> >>
>>>> >> <snip>
>>>> >>
>>>> >> _______________________________________________
>>>> >> IPoverIB mailing list
>>>> >> IPoverIB@ietf.org
>>>> >> http://www.ietf.org/mailman/listinfo/ipoverib
>>>> >>
>>>> >
>>>> >
>>>> >--
>>>> >Vivek Kashyap
>>>> >Linux Technology Center, IBM
>>>> >kashyapv@us.ibm.com
>>>> >vivk@us.ibm.com
>>>> >503 578 3422 (o)
>>>> >
>>>> >_______________________________________________
>>>> >IPoverIB mailing list
>>>> >IPoverIB@ietf.org
>>>> >http://www.ietf.org/mailman/listinfo/ipoverib
>>>>
>>>>
>>>> _______________________________________________
>>>> IPoverIB mailing list
>>>> IPoverIB@ietf.org
>>>> http://www.ietf.org/mailman/listinfo/ipoverib
>>>>


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib





_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 18:16:57 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA15265
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 18:16:57 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id SAA29060
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 18:16:59 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA28946;
	Thu, 17 Jan 2002 18:10:43 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA28916
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 18:10:41 -0500 (EST)
Received: from calliope1.fm.intel.com (fmfdns01.fm.intel.com [132.233.247.10])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA15205
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 18:10:37 -0500 (EST)
Received: from fmsmsxvs041.fm.intel.com (fmsmsxv041-1.fm.intel.com [132.233.48.109])
	by calliope1.fm.intel.com (8.9.1a+p1/8.9.1/d: relay.m4,v 1.48 2001/12/13 16:27:50 root Exp $) with SMTP id XAA07407
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 23:10:39 GMT
Received: from FMSMSX017.fm.intel.com ([132.233.42.196])
 by fmsmsxvs041.fm.intel.com (NAVGW 2.5.1.16) with SMTP id M2002011715113531269
 ; Thu, 17 Jan 2002 15:11:35 -0800
Received: by fmsmsx017.fm.intel.com with Internet Mail Service (5.5.2653.19)
	id <C0481VWM>; Thu, 17 Jan 2002 15:10:37 -0800
Message-ID: <D9223EB959A5D511A98F00508B68C20C21A853@ORSMSX108>
From: "Larsen, Roy K" <roy.k.larsen@intel.com>
To: "'Bill Strahm'" <bill@strahm.net>,
        "'H.K. Jerry Chu'" <Jerry.Chu@eng.sun.com>
Cc: ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
Date: Thu, 17 Jan 2002 15:09:22 -0800
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="iso-8859-1"
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Bill and Jerry,

I think I see were we are misunderstand each other.  It is my assumption
that mapping IP address to MAC address from within the OS will be
implementation specific.  That is, what my IPoIB ARP module and link driver
registers as a MAC address to the OS is my private concern.

For example, in Linux, there is a 8 bytes limit to the hardware address.
Since I don't want to be in the OS distribution business, my driver must
live with that.  Needless to say, an IBA address vector aint going to fit in
8 bytes.  All that is required is for me to place something in that field
that will lead me to an internal structure that contains what I need.  It is
at that point I need to make sure I am using a current LID.

The point is, I am not going to be using the LID in registering hardware
address to the OS.  My own or those of remote nodes.  So, it doesn't matter
to the OS if they change.  It will probably be the GUID but that is an
implementation choice for my driver.  Since I need to supply an IPoIB ARP
module, it is easy to inspect all source LIDs received to see if they still
match what I have in my private address vector.  If not, I update it.  The
OS was never involved.

Roy 

-----Original Message-----
From: Bill Strahm [mailto:bill@strahm.net]
Sent: Thursday, January 17, 2002 1:41 PM
To: Larsen, Roy K; 'H.K. Jerry Chu'
Cc: ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?


Roy,

Gratuitous ARP does not help here... I do I send a gratuitous ARP if I can
not change my MAC address in my ARP module...

The easiest solution is to not put LID into the MAC address, then you don't
have to worry about the problem at all on the sender side.  On the receiver
side I believe you have to do the same work regardless...

Bill

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Larsen, Roy K
Sent: Thursday, January 17, 2002 1:00 PM
To: 'H.K. Jerry Chu'
Cc: ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?


Jerry,

Yes, this could be a issue for the OS.  But since the dynamic nature of LIDs
is integral to IBA, they are just going to have to find a way to deal with
it.   As I suggested, perhaps with gratuitous ARP.  Anyway, would you agree
it is outside the scope of IPoIB.

Roy

-----Original Message-----
From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
Sent: Thursday, January 17, 2002 12:49 PM
To: roy.k.larsen@intel.com
Cc: ipoverib@ietf.org
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?


Roy,

Sorry I didn't explain it well. The potential issue I was trying to
point out is inside a local node that has its LID change. The question
is how will the LID change be propergated up to the IP stack. Many IP
implementations never expect a local NIC's  MAC address can change on
the fly, all by itself. This poses a potential implementation problem.

Jerry

>From: "Larsen, Roy K" <roy.k.larsen@intel.com>
>To: "'H.K. Jerry Chu'" <Jerry.Chu@eng.sun.com>,
arvindm@windows.microsoft.com,
kashyapv@us.ibm.com
>Cc: ipoverib@ietf.org
>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>Date: Thu, 17 Jan 2002 11:42:20 -0800
>MIME-Version: 1.0
>
>Jerry,
>
>Ok, your concern is a mechanism for notifying a node that a LID change has
>occurred regardless of how the LID was initially obtained.  I'm no SM
expert
>but seem to recall asking my resident expert about this a while back and he
>said there wasn't.  However, if the LID was included in the ARP, wouldn't
>sending a gratuitous ARP when the link comes up solve the problem?
>
>Roy
>
>-----Original Message-----
>From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
>Sent: Wednesday, January 16, 2002 7:28 PM
>To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com;
>roy.k.larsen@intel.com
>Cc: ipoverib@ietf.org
>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>
>
>Roy,
>
>Thanks for the info. A comment on your statement of volatility of LID
having
>no bearing on whether LID should be included in ARP reply. Yes it does. One
>of the goal of IPoIB is to minimize the changes to the existing IP stack
>implementations. Note that I stated it as a goal, not a constraint. If we
>must do it then we should.
>
>Yes no matter what, LID is cached somewhere and must be flushed. The
>difference
>is whether one must modify the existing stack code to add a facility to
>support
>mac addr change, or do the necessary chore in a complete new IPoIB driver.
>The former requires an upgrade of OS release that can take a long time. The
>latter doesn't have such a problem since it's brand new.
>
>One can also hope that LID flushing through SM can be made more reliable
>than
>gratuitous ARP. But I don't know the SM archtecture enough to make such a
>claim.
>
>To be fair I don't really know how many OSes lack support for mac add
>change. The two that I know, one supports it, one doesn't.
>
>I'm assuming there are solid facility in IBA to support the notification of
>attribute changes to interested parties. Otherwise not only we should
>include
>LID, but also all the rest in the link addr.
>
>Jerry
>
>>From: "Larsen, Roy K" <roy.k.larsen@intel.com>
>>To: "'H.K. Jerry Chu'" <Jerry.Chu@eng.sun.com>,
>arvindm@windows.microsoft.com, kashyapv@us.ibm.com
>>Cc: ipoverib@ietf.org
>>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>>Date: Wed, 16 Jan 2002 18:42:31 -0800
>>MIME-Version: 1.0
>>
>>> -----Original Message-----
>>> From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
>>> Sent: Wednesday, January 16, 2002 6:13 PM
>>> To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com
>>> Cc: ipoverib@ietf.org
>>> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>>>
>>>
>>> All the proxy ARP usages that I know of use the proxy's mac
>>> addr in the
>>> ARP response. I believe the flavor of the proxy ARP you
>>> alluded below is
>>> rare, and won't work very well anyway if the link address includes any
>>> non-persistent component like LID and QPN because it will
>>> require another
>>> protocol between the clients and their proxy to constantly update the
>>> proxy database containing clients mac addrs, which may change quite
>>> often.
>>>
>>> That is also why I wanted to provide an option to not include QPN.
>>> (But I haven't succeeded w/o requiring SIDR or something equivalent.)
>>>
>>> If LID is included as part of link addr and if it ever changes on
>>> a live node due to subnet merge...etc, the driver must inform its IP
>>> stack about such change. I believe many stacks don't have such a
>>> facility. (FOr most IP stacks the assumption is that its own MAC
>>> addresses just don't change on the fly.) In that case you're totally
>>> hosed. Can someone comment on how likely a LID assignment gets
>>> rescinded?
>>
>>It is possible anytime the link goes down and comes back up again and when
>>the fabic is reconfigured.  This can happen on reboot or simply
>>unplugging/plugging in a link.  It is ultimately a function of the SM and
>>how much trouble the implementation wants to go through to keep LIDs
>>constant.
>>
>>But of course, that has no bearing on whether the LID is contained in the
>>ARP.  You always need the LID.  Unless you want to do a pathrecord lookup
>>each time your implementation is given a packet to send, you're going to
>use
>>a cached LID no matter if it came in the ARP or a pathrecord lookup on the
>>GID.  If it becomes stale, it may take an ARP cache flush timeout to
>>recover.
>>
>>Roy
>>
>>>
>>> Jerry
>>>
>>> >        A device can always respond for another. The LID of
>>> the sender
>>> >        is not the same as the LID for which it has
>>> responded. It also
>>> >        allows any network monitoring/debugging device to work
>>> >        independently. This is the reason why none of the other media
>>> >        does anything like this. You don't pick ethernet source
>>> >        address from the frame but take it from the ARP/ND packet.
>>> >
>>> >        [ If it was ok we wouldn't need anything except QPN/cap_flags
>>> >        - get the LID from response, consult SA using LID for GID if
>>> >        you want that, and you are done. ]
>>> >
>>> >Vivek
>>> >
>>> >>
>>> >> Why do we need to include the LID inside the link layer
>>> address when it
>>> >> is already available in the SLID field? Previous
>>> discussions indicate
>>> >> that the SLID value works for proxy ARP/bridge implementations.
>>> >>
>>> >> - Arvind
>>> >>
>>> >> -----Original Message-----
>>> >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
>>> >> Sent: Wednesday, January 16, 2002 10:29 AM
>>> >> To: kamath@seagullsemi.com
>>> >> Cc: ipoverib@ietf.org
>>> >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>>> >>
>>> >> Ashwin,
>>> >> Ashwin,
>>> >>
>>> >>         I agree that you get more flexibility if you
>>> include the LID.
>>> >>         If we include the GID along with the LID we allow all
>>> >>         implementations the choice to create different
>>> solutions. We
>>> >>         don't tie them down to one thing. Choosing one or the other
>>> >>         (LID or GID) blocks out various advantages that different
>>> >>         implementations seek. Hence I suggested that let us include
>>> >>         both and move on.
>>> >>
>>> >> Vivek
>>> >>
>>> >> <snip>
>>> >>
>>> >> _______________________________________________
>>> >> IPoverIB mailing list
>>> >> IPoverIB@ietf.org
>>> >> http://www.ietf.org/mailman/listinfo/ipoverib
>>> >>
>>> >
>>> >
>>> >--
>>> >Vivek Kashyap
>>> >Linux Technology Center, IBM
>>> >kashyapv@us.ibm.com
>>> >vivk@us.ibm.com
>>> >503 578 3422 (o)
>>> >
>>> >_______________________________________________
>>> >IPoverIB mailing list
>>> >IPoverIB@ietf.org
>>> >http://www.ietf.org/mailman/listinfo/ipoverib
>>>
>>>
>>> _______________________________________________
>>> IPoverIB mailing list
>>> IPoverIB@ietf.org
>>> http://www.ietf.org/mailman/listinfo/ipoverib
>>>

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
http://www.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 18:52:07 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA15806
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 18:52:03 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id SAA00276
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 18:52:06 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA00238;
	Thu, 17 Jan 2002 18:50:24 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA00201
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 18:50:22 -0500 (EST)
Received: from mail4.microsoft.com (mail4.microsoft.com [131.107.3.122])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA15798
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 18:50:18 -0500 (EST)
Received: from inet-vrs-04.redmond.corp.microsoft.com ([157.54.8.154]) by mail4.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Thu, 17 Jan 2002 15:49:51 -0800
Received: from 157.54.6.150 by inet-vrs-04.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Thu, 17 Jan 2002 15:49:51 -0800
Received: from red-msg-02.redmond.corp.microsoft.com ([157.54.12.70]) by inet-hub-05.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Thu, 17 Jan 2002 15:49:50 -0800
x-mimeole: Produced By Microsoft Exchange V6.0.5762.3
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
Date: Thu, 17 Jan 2002 15:49:50 -0800
Message-ID: <24A715275661C8428C00432EFCA5CB7C055F9AFC@red-msg-02.redmond.corp.microsoft.com>
Thread-Topic: [Ipoverib] What's inside an IPoIB link layer address?
Thread-Index: AcGfk6mq4rgQqXaoSBWzSnjKEDb80AAHVw2A
From: "Jim Pinkerton" <jpink@microsoft.com>
To: "Roger Ronald" <rronald@VIEO.com>, "Vivek Kashyap" <kashyapv@us.ibm.com>
Cc: <ipoverib@ietf.org>
X-OriginalArrivalTime: 17 Jan 2002 23:49:50.0747 (UTC) FILETIME=[A7459EB0:01C19FB1]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id SAA00202
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit


Roger,

Could you walk through how the LID is preserved across failover of a SA
(i.e. it died, for whatever reason, and the backup SA takes over)? To me
this is a critically important scenario for Data Centers (we both went
through the GSN ARP fiasco, and have battle scars to prove it). It seems
to me that there is no interoperable mechanism to forward the SM/SA
state to the backup system, and thus the LIDs could be potentially
re-assigned as part of the failover? 

The GID prefix though, is generally something the network administrator
has access to (or leave as the default), and thus it is probably
reasonable to expect that the GID is in fact preserved across the
failover?



Jim




> -----Original Message-----
> From: Roger Ronald [mailto:rronald@VIEO.com]
> Sent: Thursday, January 17, 2002 12:14 PM
> To: Vivek Kashyap
> Cc: ipoverib@ietf.org
> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> 
> I think it's important to note that changes in LID should be
infrequent.
> Very infrequent.
> 
> VIEO is one of a small number of companies that has a SM at this time.
> We plan to maintain LID assignments whenever possible. That means
> across reboots, across power fails and across subnet merges (when
> possible).
> 
> There are corner cases where LID may need to change, just like IP
> addresses occasionaly need to change. But these are VERY odd
> events...like the first time merge of fabrics. They won't happen
> hourly, daily, monthly..... They may happen once or twice in a
> fabric. Ever (unless the admin wants to make changes for the heck of
it).
> 
> RR
> 
> 
> ----- Original Message -----
> From: "Vivek Kashyap" <kashyapv@us.ibm.com>
> To: "Bill Strahm" <bill@strahm.net>
> Cc: <ipoverib@ietf.org>
> Sent: Thursday, January 17, 2002 3:27 AM
> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> 
> 
> >
> > >
> > > Just free form thinking here...
> > >
> > > Could we say that GIDs are the equivalent of a Fully Qualified
Domain
> Name
> > > (www.strahm.net for example) and LIDs are the equivalent of IP
> addresses
> > > (63.105.18.42).  Now in many cases IP addresses are quite stable,
and
> do not
> > > change.  However would you ship an application with an IP address
> encoded in
> > > it, or with a FQDN.  It would be of course more efficient to use
the
> IP
> > > address and avoid the DNS lookup (that has to go VERY far away,
and
> can take
> > > quite a while for the first SYN message to get out), however if
the
> address
> > > EVER changes, things become hard.
> > >
> > > Can anyone tell me why this is a bad analogy ???
> >
> >         If it was pure IB then GID to LID would be a similar
analogy.
> >
> >         I'll try taking the your analogy further..a couple of points
> >         though - the SM corresonds to the zone primary but the nodes
> >         themselves corresond to both the secondary and the IP nodes
> >         asking for the resolution. So the analogy is not exactly apt
> >         but I'll try..
> >
> >         If the IP changes the DNS must be updated and looked up. In
> >         this case if LID changes the SM is updated (yes) but it then
> >         requires that the SM be looked up. There is no backlink here
> >         to trigger this lookup. This implies that the full lookup
all
> >         the way to the DNS root must be done - if like DNS it is
done
> >         till the local secondary server ie. referring to the local
> >         cached copy and/or merely re-receiving the peer's GID, the
> >         communication will fail. You must have seen the hell that
> >         breaks loose if any of the zone updates doesn't happen due
to
> >         some error and things are left pointing the wrong way.
> >
> >         Now, this would correspond to the internal LID cache flush -
> >         since the GID is the same but the LID has changed. There is
no
> >         hook to say, hey!, go do a zone transfer i.e. lookup the SM
> >         right now. You need a timer.
> >
> >         Now, you could make this time small - you are going to the
SM
> >         too often because things are not changing that much- make it
> >         long and LID change is surely going to happen at a critical
> >         time. Additionally this is another timer per entry for the
> >         programmers/maintainers/administrators to worry about.
> >
> >         I have all along been arguing against keeping LID out due to
> >         this reason too (code is more complex if no LID is another
> >         reason). It has been my assumption that if GID is used
without
> >         LID then one is forced to go to the SM at every address
> >         resolution. Every arp broadcast will have the whole subnet
> >         scurrying off to the SM to ask the requester's LID. Yes,
some
> >         would mark the entry to be completed but not necessarily - a
> >         thundering herd is likely.
> >
> >         This implies that if we don't use the LID we MUST ALWAYS go
to
> >         the SM for the LID - at every ARP since we don't know if LID
> >         has changed. Of course if the GID changed too then the SM
has
> >         to be consulted anyway.
> >
> >         Also note that if we did have a trap from the SM on every
LID
> >         change it still wouldn't be a good solution - too much
> >         latency; points of synchrnonisation; asynchronous updates;
and
> >         most of all the SM would end up doing nothing else but
issueing
> >         traps to every node that registered with it. IB cannot
mandate
> >         broadcasts/multicast (though making everyone look is bad
too)
> >         because multicasting is optional.
> >
> >         In the case of inclusion of the LID - some of the issues are
> >         the same but a gratuitous ARP solves the problem if the LID
> >         changed. It is an instant zone transfer. One could claim
that
> >         with GID only, on receiving any gratuitous ARP entry, the
node
> >         will go to the SM but that will require specialised coding
to
> >         recognise and handle gratuitous ARP differently than regular
> >         updates. Not a good idea.
> >
> >         Even in the absence of gratuitous ARP the changed LID will
be
> >         known when the ARP is done after a cache expiry if LID is
> >         included. If LID is not included one will get a GID back.
That
> >         will require the expiry of the GID to LID cache before one
> >         goes to the SM.
> >
> >         Inclusion of LID makes things very simple. This is because
LID
> >         is always necessary - can't escape it.
> >
> >         I also like inclusion of LID because it cuts off 2x latency
of
> >         GID to LID lookup in the SM and 4x the RTT and the realted
> >         extra interrupts. Fast failover in data centers is
necessary.
> >
> > Vivek
> >
> >
> >
> > >
> > > Bill
> > >
> > > -----Original Message-----
> > > From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On
> Behalf
> > > Of Larsen, Roy K
> > > Sent: Wednesday, January 16, 2002 6:43 PM
> > > To: 'H.K. Jerry Chu'; arvindm@windows.microsoft.com;
> kashyapv@us.ibm.com
> > > Cc: ipoverib@ietf.org
> > > Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
> > >
> > >
> > > > -----Original Message-----
> > > > From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
> > > > Sent: Wednesday, January 16, 2002 6:13 PM
> > > > To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com
> > > > Cc: ipoverib@ietf.org
> > > > Subject: Re: [Ipoverib] What's inside an IPoIB link layer
address?
> > > >
> > > >
> > > > All the proxy ARP usages that I know of use the proxy's mac
> > > > addr in the
> > > > ARP response. I believe the flavor of the proxy ARP you
> > > > alluded below is
> > > > rare, and won't work very well anyway if the link address
includes
> any
> > > > non-persistent component like LID and QPN because it will
> > > > require another
> > > > protocol between the clients and their proxy to constantly
update
> the
> > > > proxy database containing clients mac addrs, which may change
quite
> > > > often.
> > > >
> > > > That is also why I wanted to provide an option to not include
QPN.
> > > > (But I haven't succeeded w/o requiring SIDR or something
> equivalent.)
> > > >
> > > > If LID is included as part of link addr and if it ever changes
on
> > > > a live node due to subnet merge...etc, the driver must inform
its IP
> > > > stack about such change. I believe many stacks don't have such a
> > > > facility. (FOr most IP stacks the assumption is that its own MAC
> > > > addresses just don't change on the fly.) In that case you're
totally
> > > > hosed. Can someone comment on how likely a LID assignment gets
> > > > rescinded?
> > >
> > > It is possible anytime the link goes down and comes back up again
and
> when
> > > the fabic is reconfigured.  This can happen on reboot or simply
> > > unplugging/plugging in a link.  It is ultimately a function of the
SM
> and
> > > how much trouble the implementation wants to go through to keep
LIDs
> > > constant.
> > >
> > > But of course, that has no bearing on whether the LID is contained
in
> the
> > > ARP.  You always need the LID.  Unless you want to do a pathrecord
> lookup
> > > each time your implementation is given a packet to send, you're
going
> to use
> > > a cached LID no matter if it came in the ARP or a pathrecord
lookup on
> the
> > > GID.  If it becomes stale, it may take an ARP cache flush timeout
to
> > > recover.
> > >
> > > Roy
> > >
> > > >
> > > > Jerry
> > > >
> > > > >        A device can always respond for another. The LID of
> > > > the sender
> > > > >        is not the same as the LID for which it has
> > > > responded. It also
> > > > >        allows any network monitoring/debugging device to work
> > > > >        independently. This is the reason why none of the other
> media
> > > > >        does anything like this. You don't pick ethernet source
> > > > >        address from the frame but take it from the ARP/ND
packet.
> > > > >
> > > > >        [ If it was ok we wouldn't need anything except
> QPN/cap_flags
> > > > >        - get the LID from response, consult SA using LID for
GID
> if
> > > > >        you want that, and you are done. ]
> > > > >
> > > > >Vivek
> > > > >
> > > > >>
> > > > >> Why do we need to include the LID inside the link layer
> > > > address when it
> > > > >> is already available in the SLID field? Previous
> > > > discussions indicate
> > > > >> that the SLID value works for proxy ARP/bridge
implementations.
> > > > >>
> > > > >> - Arvind
> > > > >>
> > > > >> -----Original Message-----
> > > > >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> > > > >> Sent: Wednesday, January 16, 2002 10:29 AM
> > > > >> To: kamath@seagullsemi.com
> > > > >> Cc: ipoverib@ietf.org
> > > > >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer
> address?
> > > > >>
> > > > >> Ashwin,
> > > > >> Ashwin,
> > > > >>
> > > > >>         I agree that you get more flexibility if you
> > > > include the LID.
> > > > >>         If we include the GID along with the LID we allow all
> > > > >>         implementations the choice to create different
> > > > solutions. We
> > > > >>         don't tie them down to one thing. Choosing one or the
> other
> > > > >>         (LID or GID) blocks out various advantages that
different
> > > > >>         implementations seek. Hence I suggested that let us
> include
> > > > >>         both and move on.
> > > > >>
> > > > >> Vivek
> > > > >>
> > > > >> <snip>
> > > > >>
> > > > >> _______________________________________________
> > > > >> IPoverIB mailing list
> > > > >> IPoverIB@ietf.org
> > > > >> http://www.ietf.org/mailman/listinfo/ipoverib
> > > > >>
> > > > >
> > > > >
> > > > >--
> > > > >Vivek Kashyap
> > > > >Linux Technology Center, IBM
> > > > >kashyapv@us.ibm.com
> > > > >vivk@us.ibm.com
> > > > >503 578 3422 (o)
> > > > >
> > > > >_______________________________________________
> > > > >IPoverIB mailing list
> > > > >IPoverIB@ietf.org
> > > > >http://www.ietf.org/mailman/listinfo/ipoverib
> > > >
> > > >
> > > > _______________________________________________
> > > > IPoverIB mailing list
> > > > IPoverIB@ietf.org
> > > > http://www.ietf.org/mailman/listinfo/ipoverib
> > > >
> > >
> > > _______________________________________________
> > > IPoverIB mailing list
> > > IPoverIB@ietf.org
> > > http://www.ietf.org/mailman/listinfo/ipoverib
> > >
> > >
> > > _______________________________________________
> > > IPoverIB mailing list
> > > IPoverIB@ietf.org
> > > http://www.ietf.org/mailman/listinfo/ipoverib
> > >
> >
> >
> > --
> > Vivek Kashyap
> > Linux Technology Center, IBM
> > kashyapv@us.ibm.com
> > vivk@us.ibm.com
> > 503 578 3422 (o)
> >
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> >
> >
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 19:47:27 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA16469
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 19:47:27 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id TAA01964
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 19:47:28 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA01923;
	Thu, 17 Jan 2002 19:46:41 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA01892
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 19:46:39 -0500 (EST)
Received: from mxout2.netvision.net.il (mxout2.netvision.net.il [194.90.9.21])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA16446
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 19:46:36 -0500 (EST)
Received: from vaio505he ([139.92.208.216]) by mxout2.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GQ30079KZGSOY@mxout2.netvision.net.il> for ipoverib@ietf.org;
 Fri, 18 Jan 2002 02:46:06 +0200 (IST)
Date: Fri, 18 Jan 2002 02:46:47 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: [ipoverib] reasons for not using LID ?
To: ipoverib@ietf.org
Reply-to: yaronh@voltaire.com
Message-id: <000701c19fb9$9ca94670$d8d05c8b@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: text/plain; charset=windows-1255
Content-transfer-encoding: 8BIT
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
Content-Transfer-Encoding: 8BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8BIT

By now it is obvious that there are the ones that wants LID's in the ARP
packets

Could the ones that don't give their reasons for why not to
Not just things like “you don't need it”, “we want to save 16-bit”, etc'
But, reasons for what problems could arouse from having it in the packet
I haven't heard any real argument that would make me consider removing it
until now
we then will review them and see if it is possible to deal with those cases,
if not I personally would drop my support in LID's

If people want it for their reasons, the ones that don't also need to do
some explaining

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 20:00:52 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA16587
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 20:00:48 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id UAA02552
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 20:00:49 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA02197;
	Thu, 17 Jan 2002 19:59:34 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA02161
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 19:59:32 -0500 (EST)
Received: from mail1.microsoft.com (mail1.microsoft.com [131.107.3.125])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA16572
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 19:59:29 -0500 (EST)
Received: from inet-vrs-01.redmond.corp.microsoft.com ([157.54.8.27]) by mail1.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Thu, 17 Jan 2002 16:59:00 -0800
Received: from 157.54.8.155 by inet-vrs-01.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Thu, 17 Jan 2002 16:58:59 -0800
Received: from red-imc-04.redmond.corp.microsoft.com ([157.54.2.168]) by inet-hub-04.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Thu, 17 Jan 2002 16:58:59 -0800
Received: from win-imc-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.39]) by red-imc-04.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Thu, 17 Jan 2002 16:58:59 -0800
Received: from win-msg-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.52]) by win-imc-01.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3588.0);
	 Thu, 17 Jan 2002 16:56:53 -0800
Content-Class: urn:content-classes:message
Subject: RE: [ipoverib] reasons for not using LID ?
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
X-MimeOLE: Produced By Microsoft Exchange V6.0.6132.0
Date: Thu, 17 Jan 2002 16:56:55 -0800
Message-ID: <2E33960095B58E40A4D3345AB9F65EC102261873@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
Thread-Topic: [ipoverib] reasons for not using LID ?
Thread-Index: AcGfuVBgpktIKNdwQ0WAcl2LbR9Y+gAAIvxw
From: "Arvind Murching" <arvindm@windows.microsoft.com>
To: <yaronh@voltaire.com>, <ipoverib@ietf.org>
X-OriginalArrivalTime: 18 Jan 2002 00:56:53.0177 (UTC) FILETIME=[04D3BA90:01C19FBB]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id TAA02162
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

The argument should be the other way around, i.e. fields need to justify
their necessity to be part of a protocol.

Some reasons why not:

- LID values cannot be used across IB subnet boundaries
- A LID cannot be used as part of a static ARP entry
- The SLID is already present in case someone needs it

My 2c
- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com] 
Sent: Thursday, January 17, 2002 4:47 PM
To: ipoverib@ietf.org
Subject: [ipoverib] reasons for not using LID ?

By now it is obvious that there are the ones that wants LID's in the ARP
packets

Could the ones that don't give their reasons for why not to
Not just things like "you don't need it", "we want to save 16-bit", etc'
But, reasons for what problems could arouse from having it in the packet
I haven't heard any real argument that would make me consider removing
it
until now
we then will review them and see if it is possible to deal with those
cases,
if not I personally would drop my support in LID's

If people want it for their reasons, the ones that don't also need to do
some explaining

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 20:18:18 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA16817
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 20:18:17 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id UAA03022
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 20:18:19 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA02885;
	Thu, 17 Jan 2002 20:12:56 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA02836
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 20:12:53 -0500 (EST)
Received: from mail3.microsoft.com (mail3.microsoft.com [131.107.3.123])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA16739
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 20:12:51 -0500 (EST)
Received: from INET-VRS-03.redmond.corp.microsoft.com ([157.54.5.27]) by mail3.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Thu, 17 Jan 2002 17:12:22 -0800
Received: from 157.54.8.109 by INET-VRS-03.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Thu, 17 Jan 2002 17:12:21 -0800
Received: from red-msg-02.redmond.corp.microsoft.com ([157.54.12.70]) by inet-hub-02.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Thu, 17 Jan 2002 17:12:22 -0800
x-mimeole: Produced By Microsoft Exchange V6.0.5762.3
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
Date: Thu, 17 Jan 2002 17:12:20 -0800
Message-ID: <24A715275661C8428C00432EFCA5CB7C055F9AFD@red-msg-02.redmond.corp.microsoft.com>
Thread-Topic: [Ipoverib] What's inside an IPoIB link layer address?
Thread-Index: AcGfB0UvmJua7wr4Edao5ABQi2jWzAAecbrgAAR3N1AAB7vhwA==
From: "Jim Pinkerton" <jpink@microsoft.com>
To: "Arvind Murching" <ARVINDM@windows.microsoft.com>, <ipoverib@ietf.org>
X-OriginalArrivalTime: 18 Jan 2002 01:12:22.0194 (UTC) FILETIME=[2E907920:01C19FBD]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id UAA02837
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit


I think it is important to note that the SLID can be retrieved on
receipt of a UD message. 

Please see IB spec 1.0.a, section 11.4.2.1, for the arguments returned
on the "poll for completion" verb. They include (from the list of output
modifiers starting on page 533) a "remote node address and QP". Per
Table 85, for unreliable datagram, the addressing information includes
the SLID, SL, SQP, and DLID path bits.

I'm not a proponent of using the LID, but I did want to point out that
the SLID can be retrieved without putting it into the ARP packet format.



Jim









> -----Original Message-----
> From: Arvind Murching [mailto:arvindm@windows.microsoft.com]
> Sent: Thursday, January 17, 2002 12:17 PM
> To: ipoverib@ietf.org
> Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
> 
> I'm not an IB expert, but I would think that -if- we decided to use
> gratuitious ARP to notify remote nodes of changes, then those remote
> nodes would consult with the SA and re-create their address handles
> using the indicated GID.
> 
> "Optimized/simplified" implementations would pick up new SLIDs from
the
> headers of these gratuitous ARP packets.
> 
> -----Original Message-----
> From: Larsen, Roy K [mailto:roy.k.larsen@intel.com]
> Sent: Thursday, January 17, 2002 11:42 AM
> To: 'H.K. Jerry Chu'; Arvind Murching; kashyapv@us.ibm.com
> Cc: ipoverib@ietf.org
> Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
> 
> Jerry,
> 
> Ok, your concern is a mechanism for notifying a node that a LID change
> has occurred regardless of how the LID was initially obtained.  I'm no
> SM expert but seem to recall asking my resident expert about this a
> while back and he said there wasn't.  However, if the LID was included
> in the ARP, wouldn't sending a gratuitous ARP when the link comes up
> solve the problem?
> 
> Roy
> 
> -----Original Message-----
> From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
> Sent: Wednesday, January 16, 2002 7:28 PM
> To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com;
> roy.k.larsen@intel.com
> Cc: ipoverib@ietf.org
> Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
> 
> 
> Roy,
> 
> Thanks for the info. A comment on your statement of volatility of LID
> having
> no bearing on whether LID should be included in ARP reply. Yes it
does.
> One
> of the goal of IPoIB is to minimize the changes to the existing IP
stack
> implementations. Note that I stated it as a goal, not a constraint. If
> we
> must do it then we should.
> 
> Yes no matter what, LID is cached somewhere and must be flushed. The
> difference
> is whether one must modify the existing stack code to add a facility
to
> support
> mac addr change, or do the necessary chore in a complete new IPoIB
> driver.
> The former requires an upgrade of OS release that can take a long
time.
> The
> latter doesn't have such a problem since it's brand new.
> 
> One can also hope that LID flushing through SM can be made more
reliable
> than
> gratuitous ARP. But I don't know the SM archtecture enough to make
such
> a claim.
> 
> To be fair I don't really know how many OSes lack support for mac add
> change. The two that I know, one supports it, one doesn't.
> 
> I'm assuming there are solid facility in IBA to support the
notification
> of
> attribute changes to interested parties. Otherwise not only we should
> include
> LID, but also all the rest in the link addr.
> 
> Jerry
> 
> >From: "Larsen, Roy K" <roy.k.larsen@intel.com>
> >To: "'H.K. Jerry Chu'" <Jerry.Chu@eng.sun.com>,
> arvindm@windows.microsoft.com, kashyapv@us.ibm.com
> >Cc: ipoverib@ietf.org
> >Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
> >Date: Wed, 16 Jan 2002 18:42:31 -0800
> >MIME-Version: 1.0
> >
> >> -----Original Message-----
> >> From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
> >> Sent: Wednesday, January 16, 2002 6:13 PM
> >> To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com
> >> Cc: ipoverib@ietf.org
> >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> >>
> >>
> >> All the proxy ARP usages that I know of use the proxy's mac
> >> addr in the
> >> ARP response. I believe the flavor of the proxy ARP you
> >> alluded below is
> >> rare, and won't work very well anyway if the link address includes
> any
> >> non-persistent component like LID and QPN because it will
> >> require another
> >> protocol between the clients and their proxy to constantly update
the
> >> proxy database containing clients mac addrs, which may change quite
> >> often.
> >>
> >> That is also why I wanted to provide an option to not include QPN.
> >> (But I haven't succeeded w/o requiring SIDR or something
equivalent.)
> >>
> >> If LID is included as part of link addr and if it ever changes on
> >> a live node due to subnet merge...etc, the driver must inform its
IP
> >> stack about such change. I believe many stacks don't have such a
> >> facility. (FOr most IP stacks the assumption is that its own MAC
> >> addresses just don't change on the fly.) In that case you're
totally
> >> hosed. Can someone comment on how likely a LID assignment gets
> >> rescinded?
> >
> >It is possible anytime the link goes down and comes back up again and
> when
> >the fabic is reconfigured.  This can happen on reboot or simply
> >unplugging/plugging in a link.  It is ultimately a function of the SM
> and
> >how much trouble the implementation wants to go through to keep LIDs
> >constant.
> >
> >But of course, that has no bearing on whether the LID is contained in
> the
> >ARP.  You always need the LID.  Unless you want to do a pathrecord
> lookup
> >each time your implementation is given a packet to send, you're going
> to use
> >a cached LID no matter if it came in the ARP or a pathrecord lookup
on
> the
> >GID.  If it becomes stale, it may take an ARP cache flush timeout to
> >recover.
> >
> >Roy
> >
> >>
> >> Jerry
> >>
> >> >        A device can always respond for another. The LID of
> >> the sender
> >> >        is not the same as the LID for which it has
> >> responded. It also
> >> >        allows any network monitoring/debugging device to work
> >> >        independently. This is the reason why none of the other
> media
> >> >        does anything like this. You don't pick ethernet source
> >> >        address from the frame but take it from the ARP/ND packet.
> >> >
> >> >        [ If it was ok we wouldn't need anything except
> QPN/cap_flags
> >> >        - get the LID from response, consult SA using LID for GID
if
> >> >        you want that, and you are done. ]
> >> >
> >> >Vivek
> >> >
> >> >>
> >> >> Why do we need to include the LID inside the link layer
> >> address when it
> >> >> is already available in the SLID field? Previous
> >> discussions indicate
> >> >> that the SLID value works for proxy ARP/bridge implementations.
> >> >>
> >> >> - Arvind
> >> >>
> >> >> -----Original Message-----
> >> >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> >> >> Sent: Wednesday, January 16, 2002 10:29 AM
> >> >> To: kamath@seagullsemi.com
> >> >> Cc: ipoverib@ietf.org
> >> >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer
address?
> >> >>
> >> >> Ashwin,
> >> >> Ashwin,
> >> >>
> >> >>         I agree that you get more flexibility if you
> >> include the LID.
> >> >>         If we include the GID along with the LID we allow all
> >> >>         implementations the choice to create different
> >> solutions. We
> >> >>         don't tie them down to one thing. Choosing one or the
> other
> >> >>         (LID or GID) blocks out various advantages that
different
> >> >>         implementations seek. Hence I suggested that let us
> include
> >> >>         both and move on.
> >> >>
> >> >> Vivek
> >> >>
> >> >> <snip>
> >> >>
> >> >> _______________________________________________
> >> >> IPoverIB mailing list
> >> >> IPoverIB@ietf.org
> >> >> http://www.ietf.org/mailman/listinfo/ipoverib
> >> >>
> >> >
> >> >
> >> >--
> >> >Vivek Kashyap
> >> >Linux Technology Center, IBM
> >> >kashyapv@us.ibm.com
> >> >vivk@us.ibm.com
> >> >503 578 3422 (o)
> >> >
> >> >_______________________________________________
> >> >IPoverIB mailing list
> >> >IPoverIB@ietf.org
> >> >http://www.ietf.org/mailman/listinfo/ipoverib
> >>
> >>
> >> _______________________________________________
> >> IPoverIB mailing list
> >> IPoverIB@ietf.org
> >> http://www.ietf.org/mailman/listinfo/ipoverib
> >>
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 20:41:40 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA17075
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 20:41:39 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id UAA03841
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 20:41:41 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA03540;
	Thu, 17 Jan 2002 20:30:57 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA03512
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 20:30:56 -0500 (EST)
Received: from mxout2.netvision.net.il (mxout2.netvision.net.il [194.90.9.21])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA16968
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 20:30:52 -0500 (EST)
Received: from vaio505he ([62.0.172.97]) by mxout2.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GQ4007AS1IKR0@mxout2.netvision.net.il> for ipoverib@ietf.org;
 Fri, 18 Jan 2002 03:30:23 +0200 (IST)
Date: Fri, 18 Jan 2002 03:31:03 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: RE: [ipoverib] reasons for not using LID ?
In-reply-to: 
 <2E33960095B58E40A4D3345AB9F65EC102261873@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
To: "'Arvind Murching'" <arvindm@windows.microsoft.com>, yaronh@voltaire.com,
        ipoverib@ietf.org
Reply-to: yaronh@voltaire.com
Message-id: <000001c19fbf$cbff4a40$61ac003e@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7BIT

My remarks attached

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Arvind Murching
Sent: Friday, January 18, 2002 2:57 AM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

The argument should be the other way around, i.e. fields need to justify
their necessity to be part of a protocol.
[YH] I think many arguments where already raised, now it's the opposition
turn to put their arguments

Some reasons why not:

- LID values cannot be used across IB subnet boundaries
[YH] you would any way need to store both LID (Router) and GID (node) in
this case somewhere, you wouldn't go to the SM for any new IP packet wont
you
the routers in this case will any way need to change the LID in the ARP
packets (Proxy ARP and all that)
I believe this case is the one that benefits most from the LID, avoiding the
long and complex multi-subnet resolution process

- A LID cannot be used as part of a static ARP entry
[YH] We didn't say not to have GID, you could still put GID as a static
entry

- The SLID is already present in case someone needs it
[YH] also regular ARP has it, so why not follow that path


My 2c
- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 4:47 PM
To: ipoverib@ietf.org
Subject: [ipoverib] reasons for not using LID ?

By now it is obvious that there are the ones that wants LID's in the ARP
packets

Could the ones that don't give their reasons for why not to
Not just things like "you don't need it", "we want to save 16-bit", etc'
But, reasons for what problems could arouse from having it in the packet
I haven't heard any real argument that would make me consider removing
it
until now
we then will review them and see if it is possible to deal with those
cases,
if not I personally would drop my support in LID's

If people want it for their reasons, the ones that don't also need to do
some explaining

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 20:41:46 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA17091
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 20:41:46 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id UAA03855
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 20:41:48 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA03733;
	Thu, 17 Jan 2002 20:36:25 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA03695
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 20:36:22 -0500 (EST)
Received: from mail1.microsoft.com (mail1.microsoft.com [131.107.3.125])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA17026
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 20:36:20 -0500 (EST)
Received: from inet-vrs-01.redmond.corp.microsoft.com ([157.54.8.27]) by mail1.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Thu, 17 Jan 2002 17:35:51 -0800
Received: from 157.54.8.155 by inet-vrs-01.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Thu, 17 Jan 2002 17:35:49 -0800
Received: from red-imc-02.redmond.corp.microsoft.com ([157.54.9.107]) by inet-hub-04.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Thu, 17 Jan 2002 17:35:50 -0800
Received: from win-imc-02.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.84]) by red-imc-02.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Thu, 17 Jan 2002 17:35:49 -0800
Received: from win-msg-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.52]) by win-imc-02.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3588.0);
	 Thu, 17 Jan 2002 17:33:43 -0800
X-MimeOLE: Produced By Microsoft Exchange V6.0.6132.0
Content-Class: urn:content-classes:message
Subject: RE: [ipoverib] reasons for not using LID ?
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Date: Thu, 17 Jan 2002 17:33:46 -0800
Message-ID: <2E33960095B58E40A4D3345AB9F65EC104F901E6@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
Thread-Topic: [ipoverib] reasons for not using LID ?
Thread-Index: AcGfv2wEvDvT6C8JTUqGk37jyRrP1wAAGWRg
From: "Arvind Murching" <arvindm@windows.microsoft.com>
To: <yaronh@voltaire.com>, <ipoverib@ietf.org>
X-OriginalArrivalTime: 18 Jan 2002 01:33:43.0670 (UTC) FILETIME=[2A623160:01C19FC0]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id UAA03696
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

Below...

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com] 
Sent: Thursday, January 17, 2002 5:31 PM
To: Arvind Murching; yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

My remarks attached

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of
Arvind Murching
Sent: Friday, January 18, 2002 2:57 AM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

The argument should be the other way around, i.e. fields need to justify
their necessity to be part of a protocol.
[YH] I think many arguments where already raised, now it's the
opposition
turn to put their arguments
[Arvind] The need for a redundant field is not obvious.

Some reasons why not:

- LID values cannot be used across IB subnet boundaries
[YH] you would any way need to store both LID (Router) and GID (node) in
this case somewhere, you wouldn't go to the SM for any new IP packet
wont
you
the routers in this case will any way need to change the LID in the ARP
packets (Proxy ARP and all that)
I believe this case is the one that benefits most from the LID, avoiding
the
long and complex multi-subnet resolution process
[Arvind] Consider the most general case of an IB router routing between
IB subnets. This router will not modify an ARP reply packet to insert
its LID into the payload. However, it will insert its SLID into the IB
header of packets that it forwards. Proxy ARP boxes can do the same,
too.

- A LID cannot be used as part of a static ARP entry
[YH] We didn't say not to have GID, you could still put GID as a static
entry

- The SLID is already present in case someone needs it
[YH] also regular ARP has it, so why not follow that path
[Arvind] This is not an argument for adding redundant fields.


My 2c
- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 4:47 PM
To: ipoverib@ietf.org
Subject: [ipoverib] reasons for not using LID ?

By now it is obvious that there are the ones that wants LID's in the ARP
packets

Could the ones that don't give their reasons for why not to
Not just things like "you don't need it", "we want to save 16-bit", etc'
But, reasons for what problems could arouse from having it in the packet
I haven't heard any real argument that would make me consider removing
it
until now
we then will review them and see if it is possible to deal with those
cases,
if not I personally would drop my support in LID's

If people want it for their reasons, the ones that don't also need to do
some explaining

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 20:49:26 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA17215
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 20:49:26 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id UAA04167
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 20:48:28 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA04079;
	Thu, 17 Jan 2002 20:46:45 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA04048
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 20:46:44 -0500 (EST)
Received: from thalia.fm.intel.com (fmfdns02.fm.intel.com [132.233.247.11])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA17157
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 20:46:41 -0500 (EST)
Received: from fmsmsxvs042.fm.intel.com (fmsmsxvs042.fm.intel.com [132.233.42.128])
	by thalia.fm.intel.com (8.9.1a+p1/8.9.1/d: relay.m4,v 1.48 2001/12/13 16:27:50 root Exp $) with SMTP id BAA09523
	for <ipoverib@ietf.org>; Fri, 18 Jan 2002 01:46:40 GMT
Received: from fmsmsx28.fm.intel.com ([132.233.42.28])
 by fmsmsxvs042.fm.intel.com (NAVGW 2.5.1.16) with SMTP id M2002011717484514570
 ; Thu, 17 Jan 2002 17:48:45 -0800
Received: by fmsmsx28.fm.intel.com with Internet Mail Service (5.5.2653.19)
	id <D1006QHX>; Thu, 17 Jan 2002 17:46:38 -0800
Message-ID: <D9223EB959A5D511A98F00508B68C20C21A856@ORSMSX108>
From: "Larsen, Roy K" <roy.k.larsen@intel.com>
To: "'Arvind Murching'" <arvindm@windows.microsoft.com>, yaronh@voltaire.com,
        ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?
Date: Thu, 17 Jan 2002 17:46:32 -0800
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="ISO-8859-1"
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Why do we need to force a one-world view?  The LID advocates believe they
have a marketplace where they can dictate fabric topology and configuration
in such a way that they don't need to consult the SM to resolve an address
if the ARP supplies the LID.  This isn't my marketplace, but I take them at
their word that their needs are valid.  As long as an ARP contains the GID,
my needs are met.  Why can't we enable both and move on!!!!

Roy

-----Original Message-----
From: Arvind Murching [mailto:arvindm@windows.microsoft.com]
Sent: Thursday, January 17, 2002 4:57 PM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?


The argument should be the other way around, i.e. fields need to justify
their necessity to be part of a protocol.

Some reasons why not:

- LID values cannot be used across IB subnet boundaries
- A LID cannot be used as part of a static ARP entry
- The SLID is already present in case someone needs it

My 2c
- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com] 
Sent: Thursday, January 17, 2002 4:47 PM
To: ipoverib@ietf.org
Subject: [ipoverib] reasons for not using LID ?

By now it is obvious that there are the ones that wants LID's in the ARP
packets

Could the ones that don't give their reasons for why not to
Not just things like "you don't need it", "we want to save 16-bit", etc'
But, reasons for what problems could arouse from having it in the packet
I haven't heard any real argument that would make me consider removing
it
until now
we then will review them and see if it is possible to deal with those
cases,
if not I personally would drop my support in LID's

If people want it for their reasons, the ones that don't also need to do
some explaining

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 21:08:16 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA17575
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 21:08:16 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id VAA05080
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 21:08:18 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA04967;
	Thu, 17 Jan 2002 21:01:45 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA04935
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 21:01:43 -0500 (EST)
Received: from pdxpo.dsl-only.net (pdxpo.dsl-only.net [63.105.16.3])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA17401
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 21:01:40 -0500 (EST)
Received: from bonfire (unverified [63.105.18.201]) by pdxpo.dsl-only.net
 (Rockliffe SMTPRA 4.5.4) with SMTP id <B0009235249@pdxpo.dsl-only.net>;
 Thu, 17 Jan 2002 18:00:40 -0800
From: "Bill Strahm" <bill@strahm.net>
To: <yaronh@voltaire.com>, "'Arvind Murching'" <arvindm@windows.microsoft.com>,
        <ipoverib@ietf.org>
Subject: RE: [ipoverib] reasons for not using LID ?
Date: Thu, 17 Jan 2002 18:09:57 -0800
Message-ID: <HHEDIJGOEFLPCPNILMPAAEAOCFAA.bill@strahm.net>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0)
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
Importance: Normal
In-Reply-To: <000301c19fc1$ef6b3690$61ac003e@vaio505he>
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

I am very confused, why will an IB router have to modify an IP ARP packet at
all.  If there is something in the spec that REQUIRES that I think I would
be in favor of ripping it out and fixing this...

Consider IB routers as looking like Ethernet switches, it is just a layer 2
device that sends packets to the correct collision domain.  I see that as
what an IB router is for the IP layer on top of it.  You would have a hard
time convincing me that you want the router sniffing and modifing ARP
packets.  This would be a huge HUGE layer violation.

Now here is your arguement for not including LIDs in the ARP response, I
want a single ARP response for everyone that sends me an ARP request in my
IP subnet.  If this IP subnet crosses an IB subnet, this LID will not be
valid (or the router has to munge it for me)  This seems a lot of work to
get past the case of a simple lookup to the SM that the IBTA Architecture
would have you do when you are communicating with a new endpoint.

Bill

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Yaron Haviv
Sent: Thursday, January 17, 2002 5:46 PM
To: 'Arvind Murching'; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?


Arvind,

The only real argument you make is implementation in Routers
Since we build such boxes, I could tell you dealing with an ARP in the
router in case of IP span IB is complex anyway
Believe me, the simplification you are offering me creates more work than
help inside the router
As you also stated in the passed there are also very little amount of ARP
packets, so replacing it in one or two paces doesn't change anything

Yaron

-----Original Message-----
From: Arvind Murching [mailto:arvindm@windows.microsoft.com]
Sent: Friday, January 18, 2002 3:34 AM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

Below...

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 5:31 PM
To: Arvind Murching; yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

My remarks attached

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of
Arvind Murching
Sent: Friday, January 18, 2002 2:57 AM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

The argument should be the other way around, i.e. fields need to justify
their necessity to be part of a protocol.
[YH] I think many arguments where already raised, now it's the
opposition
turn to put their arguments
[Arvind] The need for a redundant field is not obvious.

Some reasons why not:

- LID values cannot be used across IB subnet boundaries
[YH] you would any way need to store both LID (Router) and GID (node) in
this case somewhere, you wouldn't go to the SM for any new IP packet
wont
you
the routers in this case will any way need to change the LID in the ARP
packets (Proxy ARP and all that)
I believe this case is the one that benefits most from the LID, avoiding
the
long and complex multi-subnet resolution process
[Arvind] Consider the most general case of an IB router routing between
IB subnets. This router will not modify an ARP reply packet to insert
its LID into the payload. However, it will insert its SLID into the IB
header of packets that it forwards. Proxy ARP boxes can do the same,
too.

- A LID cannot be used as part of a static ARP entry
[YH] We didn't say not to have GID, you could still put GID as a static
entry

- The SLID is already present in case someone needs it
[YH] also regular ARP has it, so why not follow that path
[Arvind] This is not an argument for adding redundant fields.


My 2c
- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 4:47 PM
To: ipoverib@ietf.org
Subject: [ipoverib] reasons for not using LID ?

By now it is obvious that there are the ones that wants LID's in the ARP
packets

Could the ones that don't give their reasons for why not to
Not just things like "you don't need it", "we want to save 16-bit", etc'
But, reasons for what problems could arouse from having it in the packet
I haven't heard any real argument that would make me consider removing
it
until now
we then will review them and see if it is possible to deal with those
cases,
if not I personally would drop my support in LID's

If people want it for their reasons, the ones that don't also need to do
some explaining

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 21:42:48 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA19040
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 21:42:43 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id VAA06012
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 21:42:46 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA05932;
	Thu, 17 Jan 2002 21:37:47 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA05898
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 21:37:45 -0500 (EST)
Received: from mxout2.netvision.net.il (mxout2.netvision.net.il [194.90.9.21])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA18933
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 21:37:41 -0500 (EST)
Received: from vaio505he ([62.0.172.97]) by mxout2.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GQ40086T4LX01@mxout2.netvision.net.il> for ipoverib@ietf.org;
 Fri, 18 Jan 2002 04:37:12 +0200 (IST)
Date: Fri, 18 Jan 2002 04:37:51 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: RE: [ipoverib] reasons for not using LID ?
In-reply-to: <HHEDIJGOEFLPCPNILMPAAEAOCFAA.bill@strahm.net>
To: "'Bill Strahm'" <bill@strahm.net>, ipoverib@ietf.org
Reply-to: yaronh@voltaire.com
Message-id: <000b01c19fc9$2115bc40$61ac003e@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7BIT

Bill, the answer to your first question relays in your second one

When IP spans IB, the router receive a packet with the local LID of the
first subnet (even when it is just in the S/DLID) and needs to modify it to
a packet with a local LID of the target subnet
Otherwise it won't get to the target, or returned to the sender
I would be glad to know what is your trick for avoid that
To avoid loops the router will need to implement a proxy ARP, which
maintains the virtualized routing tree
Looking at it like a layer 2 switch is to much of a theoretical abstraction

Answering the 2nd Q, by now you see that the ARP been sent to the target has
the router LID, and when you will reply it will get just to the right place
The end node operation doesn't change in any case

Yaron

-----Original Message-----
From: Bill Strahm [mailto:bill@strahm.net]
Sent: Friday, January 18, 2002 4:10 AM
To: yaronh@voltaire.com; 'Arvind Murching'; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

I am very confused, why will an IB router have to modify an IP ARP packet at
all.  If there is something in the spec that REQUIRES that I think I would
be in favor of ripping it out and fixing this...

Consider IB routers as looking like Ethernet switches, it is just a layer 2
device that sends packets to the correct collision domain.  I see that as
what an IB router is for the IP layer on top of it.  You would have a hard
time convincing me that you want the router sniffing and modifing ARP
packets.  This would be a huge HUGE layer violation.

Now here is your arguement for not including LIDs in the ARP response, I
want a single ARP response for everyone that sends me an ARP request in my
IP subnet.  If this IP subnet crosses an IB subnet, this LID will not be
valid (or the router has to munge it for me)  This seems a lot of work to
get past the case of a simple lookup to the SM that the IBTA Architecture
would have you do when you are communicating with a new endpoint.

Bill

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Yaron Haviv
Sent: Thursday, January 17, 2002 5:46 PM
To: 'Arvind Murching'; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?


Arvind,

The only real argument you make is implementation in Routers
Since we build such boxes, I could tell you dealing with an ARP in the
router in case of IP span IB is complex anyway
Believe me, the simplification you are offering me creates more work than
help inside the router
As you also stated in the passed there are also very little amount of ARP
packets, so replacing it in one or two paces doesn't change anything

Yaron

-----Original Message-----
From: Arvind Murching [mailto:arvindm@windows.microsoft.com]
Sent: Friday, January 18, 2002 3:34 AM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

Below...

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 5:31 PM
To: Arvind Murching; yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

My remarks attached

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of
Arvind Murching
Sent: Friday, January 18, 2002 2:57 AM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

The argument should be the other way around, i.e. fields need to justify
their necessity to be part of a protocol.
[YH] I think many arguments where already raised, now it's the
opposition
turn to put their arguments
[Arvind] The need for a redundant field is not obvious.

Some reasons why not:

- LID values cannot be used across IB subnet boundaries
[YH] you would any way need to store both LID (Router) and GID (node) in
this case somewhere, you wouldn't go to the SM for any new IP packet
wont
you
the routers in this case will any way need to change the LID in the ARP
packets (Proxy ARP and all that)
I believe this case is the one that benefits most from the LID, avoiding
the
long and complex multi-subnet resolution process
[Arvind] Consider the most general case of an IB router routing between
IB subnets. This router will not modify an ARP reply packet to insert
its LID into the payload. However, it will insert its SLID into the IB
header of packets that it forwards. Proxy ARP boxes can do the same,
too.

- A LID cannot be used as part of a static ARP entry
[YH] We didn't say not to have GID, you could still put GID as a static
entry

- The SLID is already present in case someone needs it
[YH] also regular ARP has it, so why not follow that path
[Arvind] This is not an argument for adding redundant fields.


My 2c
- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 4:47 PM
To: ipoverib@ietf.org
Subject: [ipoverib] reasons for not using LID ?

By now it is obvious that there are the ones that wants LID's in the ARP
packets

Could the ones that don't give their reasons for why not to
Not just things like "you don't need it", "we want to save 16-bit", etc'
But, reasons for what problems could arouse from having it in the packet
I haven't heard any real argument that would make me consider removing
it
until now
we then will review them and see if it is possible to deal with those
cases,
if not I personally would drop my support in LID's

If people want it for their reasons, the ones that don't also need to do
some explaining

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 21:45:02 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA17213
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 20:49:26 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id UAA04152
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 20:48:17 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA04035;
	Thu, 17 Jan 2002 20:46:18 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA04004
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 20:46:16 -0500 (EST)
Received: from mxout1.netvision.net.il (mxout1.netvision.net.il [194.90.9.20])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA17153
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 20:46:12 -0500 (EST)
Received: from vaio505he ([62.0.172.97]) by mxout1.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GQ40066U282MB@mxout1.netvision.net.il> for ipoverib@ietf.org;
 Fri, 18 Jan 2002 03:45:43 +0200 (IST)
Date: Fri, 18 Jan 2002 03:46:21 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: RE: [ipoverib] reasons for not using LID ?
In-reply-to: 
 <2E33960095B58E40A4D3345AB9F65EC104F901E6@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
To: "'Arvind Murching'" <arvindm@windows.microsoft.com>, ipoverib@ietf.org
Reply-to: yaronh@voltaire.com
Message-id: <000301c19fc1$ef6b3690$61ac003e@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: multipart/mixed; boundary="Boundary_(ID_IzAG1NDkaHPIIAZiAGQjdw)"
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
X-MS-TNEF-Correlator: 
 00000000D60CDA68838FD111B4D200805F8B14F30700116C80DA5F8FD111B4D200805F8B14F300000001AB620000126C80DA5F8FD111B4D200805F8B14F300000AC90A4C0000
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

This is a multi-part message in MIME format.

--Boundary_(ID_IzAG1NDkaHPIIAZiAGQjdw)
Content-type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7BIT

Arvind,

The only real argument you make is implementation in Routers
Since we build such boxes, I could tell you dealing with an ARP in the
router in case of IP span IB is complex anyway 
Believe me, the simplification you are offering me creates more work than
help inside the router 
As you also stated in the passed there are also very little amount of ARP
packets, so replacing it in one or two paces doesn't change anything 

Yaron

-----Original Message-----
From: Arvind Murching [mailto:arvindm@windows.microsoft.com]
Sent: Friday, January 18, 2002 3:34 AM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

Below...

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 5:31 PM
To: Arvind Murching; yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

My remarks attached

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of
Arvind Murching
Sent: Friday, January 18, 2002 2:57 AM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

The argument should be the other way around, i.e. fields need to justify
their necessity to be part of a protocol.
[YH] I think many arguments where already raised, now it's the
opposition
turn to put their arguments
[Arvind] The need for a redundant field is not obvious.

Some reasons why not:

- LID values cannot be used across IB subnet boundaries
[YH] you would any way need to store both LID (Router) and GID (node) in
this case somewhere, you wouldn't go to the SM for any new IP packet
wont
you
the routers in this case will any way need to change the LID in the ARP
packets (Proxy ARP and all that)
I believe this case is the one that benefits most from the LID, avoiding
the
long and complex multi-subnet resolution process
[Arvind] Consider the most general case of an IB router routing between
IB subnets. This router will not modify an ARP reply packet to insert
its LID into the payload. However, it will insert its SLID into the IB
header of packets that it forwards. Proxy ARP boxes can do the same,
too.

- A LID cannot be used as part of a static ARP entry
[YH] We didn't say not to have GID, you could still put GID as a static
entry

- The SLID is already present in case someone needs it
[YH] also regular ARP has it, so why not follow that path
[Arvind] This is not an argument for adding redundant fields.


My 2c
- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 4:47 PM
To: ipoverib@ietf.org
Subject: [ipoverib] reasons for not using LID ?

By now it is obvious that there are the ones that wants LID's in the ARP
packets

Could the ones that don't give their reasons for why not to
Not just things like "you don't need it", "we want to save 16-bit", etc'
But, reasons for what problems could arouse from having it in the packet
I haven't heard any real argument that would make me consider removing
it
until now
we then will review them and see if it is possible to deal with those
cases,
if not I personally would drop my support in LID's

If people want it for their reasons, the ones that don't also need to do
some explaining

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

--Boundary_(ID_IzAG1NDkaHPIIAZiAGQjdw)
Content-type: application/ms-tnef; name=winmail.dat
Content-disposition: attachment; filename=winmail.dat
Content-Transfer-Encoding: base64

eJ8+IhcBAQaQCAAEAAAAAAABAAEAAQeQBgAIAAAA5wQAAAAAAADrAAEIgAcAGAAAAElQTS5NaWNy
b3NvZnQgTWFpbC5Ob3RlADEIAQ2ABAACAAAAAgACAAEGgAMADgAAANIHAQASAAMALgAAAAUAIgEB
A5AGAEgOAAAtAAAACwACAAEAAAALACMAAAAAAAMAJgAAAAAACwApAAAAAAADAC4AAAAAAAMANgAA
AAAAHgBwAAEAAAAnAAAAW2lwb3ZlcmliXSByZWFzb25zIGZvciBub3QgdXNpbmcgTElEID8AAAIB
cQABAAAAFgAAAAHBn8HuFtiN2BazQEaAsAbDA8fj/eMAAAIBHQwBAAAAGQAAAFNNVFA6WUFST05I
QFZPTFRBSVJFLkNPTQAAAAALAAEOAAAAAEAABg4APEXhwZ/BAQIBCg4BAAAALgAAAAAAAADWDNpo
g4/REbTSAIBfixTzAQARbIDaX4/REbTSAIBfixTzAAAAAatjAAAAAAMAFA4BAAAAAgEJEAEAAABo
CAAAZAgAAAYRAABMWkZ1kwJutAMACgByY3BnMTI14jUDQ3RleAVBAQMB9wcKgAKkA+REYXZpZE8C
gA/zAFAEVk1pByFtsCBGaXgJgBElMgLjCQIAY2gKwHNldDI3BgAGwxElMwRGE8cwIL8IVQeyAoAR
MwjvCfc7GC93DjERIgxgYwBQCwkBZDM2NhZgC6YgBxAQ8G5kDiwKogqECoBUaGUgsQIgbHkgGDAH
QCAKwExndQeAAjAgeQhgIF0AwGseUAQAICBtC1Bl8R9SYXRpAiAgIAOgCACudQ6wFBAddFMLgGMe
UBJ3HlBidQMQZCBzFRrgaCLQbxLAcywgtEkgBaB1IxEOsGwDIC8fogEAB0ALgGcioGl04yNwA5FB
UlAhQiWwHlBnA2AhoiFCY2EUIB5gZpsj8CYwcwqwA6BJQiAiiwWgIHJ4JdF5d2EeoO0ddEIkkAiQ
dh5QB4Aj4P8mggCQIHEGkA3gIPQfogrA/SeSZgZxJWEHgCQQHsEOsNcEIARgLFF3BbBrJnEDkfke
QGxwIUEAkAEAJnoddMZBBCAsA2xzbyMwIOH/CYAmRgqwBBAxgSaBLFEsQvsw8ypgch6gJUACQCCQ
HwC9BGB1H3EnsSYSCrBjIADedCPRMREYMAtRYyVSJaDvIUICIB5RBcB0LhA1EgeR5GRvB5BuJwVA
E+ElYN8y0SlQJbAlUh16WQrAAiDVHXotO0JPBRBnC4Ae4XJNB5BzYTiwO0MddEZ9A2E6HPUF0Ahw
E+AlUltjAMADEHRvOgrAHSJtBkAD8B1Ab3dzLm2vDeADYDEQAYAuKMFdIfW/H2E9gD1AEQApgCPg
SgBwZnUKwB6gMTgj4AHQMKEUUDM6MzQc8E0d1cc/AB+QOkJoQHYG8AGQlRJAZUCyOyAgcG8zgVhp
YkAIkAAwLgWwZzEh9XViagWQQZFSRfU9gFtFtl0esjEQBjECEGkFwG5vBUB1AJAlYUxwSUQgPx16
KhE/4C6fS2A6jzufPK46MyBIEOG+dj6XRG9A3R4wCHBzQfyKN0LlNUNgMSBQQ7n/Pa1FkE//RX9G
j0efSK9Jv/0dg00eogDALjAEICDwAZA/E+AJgEufTK88rlcWLWE/P3ALgFeXPpdgz1fEXU+nA6Aq
EBPwbGYddE9kxQ89rUEPQh9DITI6NTf/Q59WD1cfWC9ZP1pPW18eI+kfF3NoJDNiL1RvEDKhvyKg
KYE6QTRwHVAgIC5q4H9usAiQIxAEIDbwMmIxIGrzb0AhAGZ5HXQmgRJAdDH7N7EAkHQeoHSRcfEK
sTSTPmEyEANgPvAXoUuFW1n+SG4gJAA5Ei5AA4Fy4h801QQgdzKkbB7BZB6hC3DfMlEj4G8AB+Al
oCcEICaB+R10b3BrYHZBIRF1NQhw/yZhN3EhoCZydcF5p3glHQT/biAeMnRDbsJ3cBgxczEAcM8F
QHPTICJvAm9iEPAIYPtAAB16UwNwJqFuVXpAHqDtbwE6Xatvk3YHQApQKKH/AHBvAnHxb0AxgQDQ
QFEEIPsoYSNAYjbwhzFzIgrACJD/f2Z4kh+iLhAkQnliKXJ0Rl8xQC3SBuAlsW+iKCGEKfsl0SMg
R4ySbwABAI0wC4H/dUUokidyMRAHgHpDI+CKN504MmcxIHSRJoJTTYDk/3lxNvAH4CfhNSQddC4Q
AjD/HXQfoXU3JrUgQSZijuYD8P8koYreOHUmgm+iJlUmER106zUljLBQA2B4HqAmEo1S5wdAAyAu
YXQpHXQkAHHw/yo0jsggMXIjNvGbEnHhNvD/K3B6EQRgMUBusANhl9Yj4P0Q4G8RAGZ3fDcXwCVh
jVLtKMZtJEAhAC2IdRgwMRD/CkAhA3eRdhJ/fQhQLxM3Mf8eQZ6TOLA28HsgAyAnZig0/ybFJsIl
UnHwN1AJ4ZtliFb/QABR4SAxJsWWI28CBGF1Af8l1jXiHqA1JHSCLwEEkJL1755SmCSRFQqweRfA
YVBzsPZIP+AqUXJzYQVAliOtVP82cQZBrlsoYB10HkBhUBbR/yexmWadszaBbsEpcAsgqfH/mfgj
k4bSN+EqxBJwKqB1Nfxvb0uMHPBvk4buBCB2+L8xQg3gJgMfYTOgeClXHlD/oACQk19whPN0ghPw
KmGNkf+P9CQkdOEkoX5CjZK54bqW/x10u2mFth4ysXRdAXq1d5D/B5AfYic2j2I24nRCIEGS9f94
gzDzGDAfMAtgBcAmEhPw/8USNZOE1W6xJKB7sZ2zCrB/JbB/fyAygmQDkR8XgPNk76ACgU6DLFwX
MsB1uHEdE/9dr16/Tb9Oz0/fUO9R/1MI+DQ6NGkgU/prX2xtbZ//bq9KH4Tye8IgIoLFnaQymP+d
NbPVKXB6AtzxfAGYbzU0/x16CFEkUuBbN/CQo9NwL1PfdcHbmsfGPvAddE7cYXTC3XjjZwQgJUAg
ASIk0+Vj63RDJaAiI+AiIrHhIouTYb3CMTYtYupzFDBj/icptSGgI+DmbMjSA2ACYP8goCiiipMm
wSeBnuO9sTZI/zHkksgkAL2yODKy0QsglmP/Hszg1CQzH+MtEqU2XKHZAP9md8U2NHEDEXuhkwUv
UwOg/5YjGDAQ8AfRJoESgI1SFCD/IBEnwN50fOEAkO6RdIIlEv8lhCWwQGB8VSdiI9CttSfA+9xS
JABwIcECIJrRivEkM/ZkA2Au4G0eoCNAfNGxAvcDoOGDHXpJs1HEgCCB6vT/tETl/Cq05L4w83RG
N/C4FHuPYrtQeAtTZnc53x16X+8Jbwp/C0qbZVDZAihhPrJfJVIlQDFADFzZbWgz8HAgczovL3cQ
UDEu7dl2Lz6yeVEvDcKOQIDw/i/Y5givE88U3wvfDO8N/y8PDxAfES8SPX0dUB4AQhABAAAAXQAA
ADwyRTMzOTYwMDk1QjU4RTQwQTREMzM0NUFCOUY2NUVDMTA0RjkwMUU2QHdpbi1tc2ctMDEud2lu
Z3JvdXAud2luZGVwbG95Lm50ZGV2Lm1pY3Jvc29mdC5jb20+AAAAAAMACVkBAAAAAgHiZQEAAAAU
AAAAMjJa61un80uWZ+AppchnJAAAWBgCAeNlAQAAABUAAAAUMjJa61un80uWZ+AppchnJAAAWBgA
AAADAAOACCAGAAAAAADAAAAAAAAARgAAAABShQAAjmoBAB4ABIAIIAYAAAAAAMAAAAAAAABGAAAA
AFSFAAABAAAABAAAADkuMAALAAaACCAGAAAAAADAAAAAAAAARgAAAAAGhQAAAAAAAAMAB4AIIAYA
AAAAAMAAAAAAAABGAAAAAAGFAAAAAAAACwAIgAggBgAAAAAAwAAAAAAAAEYAAAAAA4UAAAAAAAAL
AAmACCAGAAAAAADAAAAAAAAARgAAAAAOhQAAAAAAAAMACoAIIAYAAAAAAMAAAAAAAABGAAAAABCF
AAAAAAAAAwALgAggBgAAAAAAwAAAAAAAAEYAAAAAEYUAAAAAAAADAAyACCAGAAAAAADAAAAAAAAA
RgAAAAAYhQAAAAAAAB4ADYAIIAYAAAAAAMAAAAAAAABGAAAAADaFAAABAAAAAQAAAAAAAAAeAA6A
CCAGAAAAAADAAAAAAAAARgAAAAA3hQAAAQAAAAEAAAAAAAAAHgAPgAggBgAAAAAAwAAAAAAAAEYA
AAAAOIUAAAEAAAABAAAAAAAAAEAAG4AIIAYAAAAAAMAAAAAAAABGAAAAAGCFAAAAPKY77///HwsA
Hw4BAAAAAgH4DwEAAAAQAAAAWcsw0bHT0U6F6qveiKrZzAIB+g8BAAAAEAAAANYM2miDj9ERtNIA
gF+LFPMCAfsPAQAAAIwAAAAAAAAAOKG7EAXlEBqhuwgAKypWwgAARU1TTURCLkRMTAAAAAAAAAAA
G1X6IKpmEc2byACqAC/EWgwAAABOVFNFUlZFUgAvbz1Wb2x0YWlyZSBBZHZhbmNlZCBEYXRhIFNl
Y3VyaXR5L291PVZPTFRBSVJFL2NuPVJlY2lwaWVudHMvY249WUFST05IAAMA/g8FAAAAAwANNP0/
AAACARQ0AQAAABAAAABUlKHAKX8QG6WHCAArKiUXHgD6PwEAAAAMAAAAWWFyb24gSGF2aXYAAgF/
AAEAAACNAAAAMDAwMDAwMDBENjBDREE2ODgzOEZEMTExQjREMjAwODA1RjhCMTRGMzA3MDAxMTZD
ODBEQTVGOEZEMTExQjREMjAwODA1RjhCMTRGMzAwMDAwMDAxQUI2MjAwMDAxMjZDODBEQTVGOEZE
MTExQjREMjAwODA1RjhCMTRGMzAwMDAwQUM5MEE0QzAwMDAAAAAAAwAGEHd0Uz4DAAcQAwsAAAMA
EBAAAAAAAwAREAAAAAAeAAgQAQAAAGUAAABBUlZJTkQsVEhFT05MWVJFQUxBUkdVTUVOVFlPVU1B
S0VJU0lNUExFTUVOVEFUSU9OSU5ST1VURVJTU0lOQ0VXRUJVSUxEU1VDSEJPWEVTLElDT1VMRFRF
TExZT1VERUFMSU5HAAAAAPw/

--Boundary_(ID_IzAG1NDkaHPIIAZiAGQjdw)--

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 22:50:53 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA21141
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 22:50:52 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id WAA08923
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 22:50:56 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA08335;
	Thu, 17 Jan 2002 22:47:35 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA08285
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 22:47:33 -0500 (EST)
Received: from zinfandel.topspincom.com ([209.237.59.50])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA21046
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 22:47:27 -0500 (EST)
Received: from anarchy.topspincom.com (lab2.topspincom.com [10.0.0.210]) by zinfandel.topspincom.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13)
	id ZH5HKKH8; Thu, 17 Jan 2002 19:50:49 -0800
Received: (from libor@localhost)
	by anarchy.topspincom.com (8.11.0/8.11.0) id g0I3qdl03595
	for ipoverib@ietf.org; Thu, 17 Jan 2002 19:52:39 -0800
Date: Thu, 17 Jan 2002 19:52:39 -0800
From: Libor Michalek <libor@pobox.com>
To: ipoverib@ietf.org
Subject: Re: [ipoverib] reasons for not using LID ?
Message-ID: <20020117195239.A22989@topspincom.com>
References: <000701c19fb9$9ca94670$d8d05c8b@vaio505he>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <000701c19fb9$9ca94670$d8d05c8b@vaio505he>; from yaronh@voltaire.com on Fri, Jan 18, 2002 at 02:46:47AM +0200
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

On Fri, Jan 18, 2002 at 02:46:47AM +0200, Yaron Haviv wrote:
> By now it is obvious that there are the ones that wants LID's in the ARP
> packets
> 
> Could the ones that don't give their reasons for why not to
> Not just things like 'you don't need it', 'we want to save 16-bit', etc
> But, reasons for what problems could arouse from having it in the packet
> I haven't heard any real argument that would make me consider removing it
> until now
> we then will review them and see if it is possible to deal with those cases,
> if not I personally would drop my support in LID's

  If a single IP subnet spans two IB subnets, (the IP subnet uses them as 
a single layer 2 domain) the LID in an ARP which crosses the IB router
will be incorrect, because the router is an IB router, and does not
necessarily understand IP traffic.

  For example:

  A single IP subnet spans two IB subnets, (subnets X and Y) and there is 
an IB router between the two subnets. Hosts A and B are on the same IP
subnets, but host A is on IB subnet X and host B is on IB subnet Y. Host
A sends an ARP request for host B's address, and host B responds. Host A
will have a LID for host B which is incorrect, because the LID is for a
different IB subnet. What host A needs is the LID of the IB router.

  In this scenario the IB router needs to understand IP ARP packets and
be able to look into each packet to determine if it is an ARP, and
modify the LID. This requires all IB routers to implement proxy ARP. 
This seems to be an unnecessary constraint on IB routers that want to
carry IP traffic.

  The suggestion has been made to include the LID and people can chose
to use it or not. The problem there is if a host driver does use the
LID to communicate on topology described above, it will fail unless the
IB router understands IP. Unless the recipient looks to see if the LID
in the ARP and IB packet SLID match, but this opens problems for legitamate
cases where you want an entity answering ARPs for some other entity.

  Requiring IB routers to understand IP breaks one of the 
mandates/assumptions under which we've been working, and that is to use
all of IB as a strictly layer 2 transport. I favor that we stay with that
course.


-Libor
--
Libor Michalek
libor@topspin.com
Topspin Communications
  


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 23:22:37 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA21632
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 23:22:36 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id XAA10928
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 23:22:40 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id XAA10187;
	Thu, 17 Jan 2002 23:20:38 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id XAA10163
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 23:20:37 -0500 (EST)
Received: from pdxpo.dsl-only.net (pdxpo.dsl-only.net [63.105.16.3])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA21507
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 23:20:33 -0500 (EST)
Received: from bonfire (unverified [63.105.18.201]) by pdxpo.dsl-only.net
 (Rockliffe SMTPRA 4.5.4) with SMTP id <B0009237858@pdxpo.dsl-only.net>;
 Thu, 17 Jan 2002 20:19:36 -0800
From: "Bill Strahm" <bill@strahm.net>
To: <yaronh@voltaire.com>, <ipoverib@ietf.org>
Subject: RE: [ipoverib] reasons for not using LID ?
Date: Thu, 17 Jan 2002 20:28:54 -0800
Message-ID: <HHEDIJGOEFLPCPNILMPAEEBACFAA.bill@strahm.net>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0)
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
Importance: Normal
In-Reply-To: <000b01c19fc9$2115bc40$61ac003e@vaio505he>
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

I do not agree with your analysis in your first answer.  Yes the IB router
would receive the packet on its first interface and forward it on its second
interface replacing the DLID with its own DLID.  This is what a layer 3
router does.

Now look at the Infiniband fabric as a layer 2 for the IP layer 3.  At this
point you want layer 2 semantics.  At this point IB routers look like layer
2 switches and MUST NOT interfere with the layer 3 packets on top (otherwise
it is a layer 3 router, and it might as well do IP routing and we don't have
the subnet spanning problem at all)

The job of an IB router in an infiniband fabric is much simpler if we do not
have LID in the ARP packet at all, and just have the destination node poll
the SM for the LID based on the GID.

Bill

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Yaron Haviv
Sent: Thursday, January 17, 2002 6:38 PM
To: 'Bill Strahm'; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?


Bill, the answer to your first question relays in your second one

When IP spans IB, the router receive a packet with the local LID of the
first subnet (even when it is just in the S/DLID) and needs to modify it to
a packet with a local LID of the target subnet
Otherwise it won't get to the target, or returned to the sender
I would be glad to know what is your trick for avoid that
To avoid loops the router will need to implement a proxy ARP, which
maintains the virtualized routing tree
Looking at it like a layer 2 switch is to much of a theoretical abstraction

Answering the 2nd Q, by now you see that the ARP been sent to the target has
the router LID, and when you will reply it will get just to the right place
The end node operation doesn't change in any case

Yaron

-----Original Message-----
From: Bill Strahm [mailto:bill@strahm.net]
Sent: Friday, January 18, 2002 4:10 AM
To: yaronh@voltaire.com; 'Arvind Murching'; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

I am very confused, why will an IB router have to modify an IP ARP packet at
all.  If there is something in the spec that REQUIRES that I think I would
be in favor of ripping it out and fixing this...

Consider IB routers as looking like Ethernet switches, it is just a layer 2
device that sends packets to the correct collision domain.  I see that as
what an IB router is for the IP layer on top of it.  You would have a hard
time convincing me that you want the router sniffing and modifing ARP
packets.  This would be a huge HUGE layer violation.

Now here is your arguement for not including LIDs in the ARP response, I
want a single ARP response for everyone that sends me an ARP request in my
IP subnet.  If this IP subnet crosses an IB subnet, this LID will not be
valid (or the router has to munge it for me)  This seems a lot of work to
get past the case of a simple lookup to the SM that the IBTA Architecture
would have you do when you are communicating with a new endpoint.

Bill

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Yaron Haviv
Sent: Thursday, January 17, 2002 5:46 PM
To: 'Arvind Murching'; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?


Arvind,

The only real argument you make is implementation in Routers
Since we build such boxes, I could tell you dealing with an ARP in the
router in case of IP span IB is complex anyway
Believe me, the simplification you are offering me creates more work than
help inside the router
As you also stated in the passed there are also very little amount of ARP
packets, so replacing it in one or two paces doesn't change anything

Yaron

-----Original Message-----
From: Arvind Murching [mailto:arvindm@windows.microsoft.com]
Sent: Friday, January 18, 2002 3:34 AM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

Below...

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 5:31 PM
To: Arvind Murching; yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

My remarks attached

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of
Arvind Murching
Sent: Friday, January 18, 2002 2:57 AM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

The argument should be the other way around, i.e. fields need to justify
their necessity to be part of a protocol.
[YH] I think many arguments where already raised, now it's the
opposition
turn to put their arguments
[Arvind] The need for a redundant field is not obvious.

Some reasons why not:

- LID values cannot be used across IB subnet boundaries
[YH] you would any way need to store both LID (Router) and GID (node) in
this case somewhere, you wouldn't go to the SM for any new IP packet
wont
you
the routers in this case will any way need to change the LID in the ARP
packets (Proxy ARP and all that)
I believe this case is the one that benefits most from the LID, avoiding
the
long and complex multi-subnet resolution process
[Arvind] Consider the most general case of an IB router routing between
IB subnets. This router will not modify an ARP reply packet to insert
its LID into the payload. However, it will insert its SLID into the IB
header of packets that it forwards. Proxy ARP boxes can do the same,
too.

- A LID cannot be used as part of a static ARP entry
[YH] We didn't say not to have GID, you could still put GID as a static
entry

- The SLID is already present in case someone needs it
[YH] also regular ARP has it, so why not follow that path
[Arvind] This is not an argument for adding redundant fields.


My 2c
- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 4:47 PM
To: ipoverib@ietf.org
Subject: [ipoverib] reasons for not using LID ?

By now it is obvious that there are the ones that wants LID's in the ARP
packets

Could the ones that don't give their reasons for why not to
Not just things like "you don't need it", "we want to save 16-bit", etc'
But, reasons for what problems could arouse from having it in the packet
I haven't heard any real argument that would make me consider removing
it
until now
we then will review them and see if it is possible to deal with those
cases,
if not I personally would drop my support in LID's

If people want it for their reasons, the ones that don't also need to do
some explaining

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 23:29:55 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA21859
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 23:29:55 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id XAA12275
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 23:29:59 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id XAA12254;
	Thu, 17 Jan 2002 23:28:48 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id XAA12218
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 23:28:46 -0500 (EST)
Received: from e33.co.us.ibm.com (e33.co.us.ibm.com [32.97.110.131])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA21852
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 23:28:42 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e33.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id XAA94512;
	Thu, 17 Jan 2002 23:25:42 -0500
Received: from d03nm036.boulder.ibm.com (avpilot.boulder.ibm.com [9.17.188.135])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0I4ShA41574;
	Thu, 17 Jan 2002 21:28:43 -0700
Subject: RE: [ipoverib] reasons for not using LID ?
To: "Arvind Murching" <arvindm@windows.microsoft.com>
Cc: ipoverib@ietf.org
X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000
Message-ID: <OFC64D721E.D3E1F021-ON88256B45.00063DA4@boulder.ibm.com>
From: "Vivek Kashyap" <vivk@us.ibm.com>
Date: Thu, 17 Jan 2002 20:27:21 -0800
X-MIMETrack: Serialize by Router on D03NM036/03/M/IBM(Release 5.0.9 |November 16, 2001) at
 01/17/2002 09:28:43 PM
MIME-Version: 1.0
Content-type: text/plain; charset=us-ascii
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org


<VK> LID is the only part of the link address that is uniquely tied to a
port.  No communication reaches the port without the LID. It is the
hardware address. GID is not needed for communication unless sending across
the IB subnet which is not possible under IPoIB at present unless you
handcraft the infrastructure.

None of the reasons given below are a problem if the LID is included in the
link address.
<VK>

The argument should be the other way around, i.e. fields need to justify
their necessity to be part of a protocol.

Some reasons why not:

- LID values cannot be used across IB subnet boundaries

<VK> In what way do you see cross-IB subnet communication being used in the
IPoIB context ?  A reason needs to be justified before it can be
considered. What is the advantage you see in IP subnets spanning IB subnets
?

There is no infrastructure supporting it.  There is no illustration of
advantages in IP spanning IB subnets. It will be a long time before IBTA
can duplicate things the way IP has; routing protocols, routing updates,
routing tables, host lookup and updates etc. etc. And finally you will need
to justify IP subnets spanning IB subnets.

The only cross subnet communication possible is while using proxy-ARP. It
works fine with LID in it. Rather including LID is the only way it will
work at present. Use of a random GID and no LID  without the routing
infrastructure in place will cause the node to depend on manual entries.

This reason is therefore invalid.
<VK>

- A LID cannot be used as part of a static ARP entry

<VK>

It is trivial to mark an entry incomplete when read from the static entry
table and update it from the SM if you wish to. Static entries are a
special case and you will handle them specially at startup.  It is not
reason enough to dictate the wire protocol.

BTW, it has been mentioned by SM developers that LID will be a constant. It
has been mentioned by GID proponents that they will use multiple GIDs per
IP address. There can be only one LID per port but N GIDs per port. It is
the LID that is constant and not the GID.

Note that if you think that is important you can always ensure that the
GIDs/LIDs are reserved for the set of nodes. Do it like PVCs in case of IP
over ATM. The SM vendors are going to in general ensure that the LIDs
hardly ever change. The SM can easily be asked to really, more than ever,
look after the static entries.


<VK>




- The SLID is already present in case someone needs it

<VK> It has been pointed out that it doesn't solve every situation. <VK>

Vivek

My 2c
- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 4:47 PM
To: ipoverib@ietf.org
Subject: [ipoverib] reasons for not using LID ?

By now it is obvious that there are the ones that wants LID's in the ARP
packets

Could the ones that don't give their reasons for why not to
Not just things like "you don't need it", "we want to save 16-bit", etc'
But, reasons for what problems could arouse from having it in the packet
I haven't heard any real argument that would make me consider removing
it
until now
we then will review them and see if it is possible to deal with those
cases,
if not I personally would drop my support in LID's

If people want it for their reasons, the ones that don't also need to do
some explaining

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib





_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 23:38:38 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA22352
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 23:38:38 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id XAA12868
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 23:38:42 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id XAA12852;
	Thu, 17 Jan 2002 23:37:44 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id XAA12806
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 23:37:42 -0500 (EST)
Received: from e34.co.us.ibm.com (e34.co.us.ibm.com [32.97.110.132])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA22342;
	Thu, 17 Jan 2002 23:37:37 -0500 (EST)
Received: from westrelay01.boulder.ibm.com (westrelay01.boulder.ibm.com [9.99.140.22])
	by e34.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id XAA17134;
	Thu, 17 Jan 2002 23:34:33 -0500
Received: from d03nm036.boulder.ibm.com (avpilot.boulder.ibm.com [9.17.188.135])
	by westrelay01.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0I4bYo89098;
	Thu, 17 Jan 2002 21:37:34 -0700
Subject: RE: [ipoverib] reasons for not using LID ?
To: "Arvind Murching" <arvindm@windows.microsoft.com>
Cc: ipoverib@ietf.org, ipoverib-admin@ietf.org, yaronh@voltaire.com
X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000
Message-ID: <OFE53AF63E.8651AF7A-ON88256B45.001898EC@boulder.ibm.com>
From: "Vivek Kashyap" <vivk@us.ibm.com>
Date: Thu, 17 Jan 2002 20:36:11 -0800
X-MIMETrack: Serialize by Router on D03NM036/03/M/IBM(Release 5.0.9 |November 16, 2001) at
 01/17/2002 09:37:34 PM
MIME-Version: 1.0
Content-type: text/plain; charset=us-ascii
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org



Below...

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 5:31 PM
To: Arvind Murching; yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

My remarks attached

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of
Arvind Murching
Sent: Friday, January 18, 2002 2:57 AM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

The argument should be the other way around, i.e. fields need to justify
their necessity to be part of a protocol.
[YH] I think many arguments where already raised, now it's the
opposition
turn to put their arguments
[Arvind] The need for a redundant field is not obvious.

<VK> It doesn't become redundant because you want to get it via another
means. Reasons have been given by LID proponents that are very valid.
Reasons given by you against the GID are weak. None of the uses you want
are blocked by including the LID. See the earlier mail in reply to your
mail.
<VK>

Some reasons why not:

- LID values cannot be used across IB subnet boundaries
[YH] you would any way need to store both LID (Router) and GID (node) in
this case somewhere, you wouldn't go to the SM for any new IP packet
wont
you
the routers in this case will any way need to change the LID in the ARP
packets (Proxy ARP and all that)
I believe this case is the one that benefits most from the LID, avoiding
the
long and complex multi-subnet resolution process
[Arvind] Consider the most general case of an IB router routing between
IB subnets. This router will not modify an ARP reply packet to insert
its LID into the payload. However, it will insert its SLID into the IB
header of packets that it forwards. Proxy ARP boxes can do the same,
too.


- A LID cannot be used as part of a static ARP entry
[YH] We didn't say not to have GID, you could still put GID as a static
entry

- The SLID is already present in case someone needs it
[YH] also regular ARP has it, so why not follow that path
[Arvind] This is not an argument for adding redundant fields.


My 2c
- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 4:47 PM
To: ipoverib@ietf.org
Subject: [ipoverib] reasons for not using LID ?

By now it is obvious that there are the ones that wants LID's in the ARP
packets

Could the ones that don't give their reasons for why not to
Not just things like "you don't need it", "we want to save 16-bit", etc'
But, reasons for what problems could arouse from having it in the packet
I haven't heard any real argument that would make me consider removing
it
until now
we then will review them and see if it is possible to deal with those
cases,
if not I personally would drop my support in LID's

If people want it for their reasons, the ones that don't also need to do
some explaining

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib





_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 23:46:29 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA22520
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 23:46:29 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id XAA13032
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 23:46:33 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id XAA13019;
	Thu, 17 Jan 2002 23:45:49 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id XAA12983
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 23:45:46 -0500 (EST)
Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA22514
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 23:45:42 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id WAA22498
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 22:41:29 -0600
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0I4jiE52208
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 23:45:44 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id g0I4jhg07010
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 20:45:43 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0I4jhq23173
	for ipoverib@ietf.org; Thu, 17 Jan 2002 20:45:43 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201180445.g0I4jhq23173@eng4.beaverton.ibm.com>
To: ipoverib@ietf.org
Date: Thu, 17 Jan 2002 20:45:42 -0800 (PST)
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Subject: [Ipoverib] encapsulation issues
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit


> > 
> > 
> > 2. 	I believe based on the recent set of mails that the consensus no
> >         longer favours inclusion of the 'ethertype'. Is there a
> >         disagreement ?
> 
> There were many that said that an Ethertype was desirable and useful in
> their implementations.  Your counter argument was that it was not needed
> because it could be inferred through other means so why add an additional
> mechanism.  If you would allow us to use the same logic used in #1 above
> ("the architecture not forcing a limited solution on any set of
> aspirations"), then I would like an Ethertype defined as part of the
> payload.  As noted earlier, the inclusion of an Ethertype does not handicap
> or restrict any solution.
> 
> Roy

        I know that many found it desirable in the beginning but it
        appeared to me that people had changed their mind.

        The reason ethertype is unnecessary is that it doesn't buy us
        the solution it is supposed to due to the requirement that
        every other IB protocol follows the same rules. IPoIB has no
        control over that.

Vivek


> 
> > 
> > 
> > 3. 	As many have noted in the last couple of days that using IB
> >         over non-UD modes or utilising QPNs mapped to specific
> >         endpoints is desirable. A complete, interoperable solution has
> >         been presented in
> >         draft-kashyap-ipoib-advanced-capabilities-00.txt. 
> > 
> > 	Please review and comment. 
> > 
> > Vivek
> > 
> > 
> > 	
> > 
> > -- 
> > Vivek Kashyap
> > Linux Technology Center, IBM
> > kashyapv@us.ibm.com 
> > vivk@us.ibm.com 
> > 503 578 3422 (o)
> > 
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> > 
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)
-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 17 23:49:53 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA22558
	for <ipoverib-archive@odin.ietf.org>; Thu, 17 Jan 2002 23:49:53 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id XAA13116
	for ipoverib-archive@odin.ietf.org; Thu, 17 Jan 2002 23:49:57 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id XAA13088;
	Thu, 17 Jan 2002 23:47:00 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id XAA13048
	for <ipoverib@optimus.ietf.org>; Thu, 17 Jan 2002 23:46:58 -0500 (EST)
Received: from e33.co.us.ibm.com (e33.co.us.ibm.com [32.97.110.131])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA22534;
	Thu, 17 Jan 2002 23:46:53 -0500 (EST)
Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23])
	by e33.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id XAA20176;
	Thu, 17 Jan 2002 23:43:50 -0500
Received: from d03nm036.boulder.ibm.com (avpilot.boulder.ibm.com [9.17.188.135])
	by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0I4kpi31884;
	Thu, 17 Jan 2002 21:46:51 -0700
Subject: RE: [ipoverib] reasons for not using LID ?
To: "Larsen, Roy K" <roy.k.larsen@intel.com>
Cc: "'Arvind Murching'" <arvindm@windows.microsoft.com>, ipoverib@ietf.org,
        ipoverib-admin@ietf.org, yaronh@voltaire.com
X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000
Message-ID: <OF89F10560.DD939CFE-ON88256B45.0019547D@boulder.ibm.com>
From: "Vivek Kashyap" <vivk@us.ibm.com>
Date: Thu, 17 Jan 2002 20:45:28 -0800
X-MIMETrack: Serialize by Router on D03NM036/03/M/IBM(Release 5.0.9 |November 16, 2001) at
 01/17/2002 09:46:51 PM
MIME-Version: 1.0
Content-type: text/plain; charset=us-ascii
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org



Why do we need to force a one-world view?  The LID advocates believe they
have a marketplace where they can dictate fabric topology and configuration
in such a way that they don't need to consult the SM to resolve an address
if the ARP supplies the LID.  This isn't my marketplace, but I take them at
their word that their needs are valid.  As long as an ARP contains the GID,
my needs are met.  Why can't we enable both and move on!!!!

Roy

<VK>
I totally agree with the sentiment. The architecture needs to provide a
solution that is valid for all solutions possible as long as they don't
break interoperability. The packet can include both the LID and the GID and
things will work fine for both camps.

<VK>

-----Original Message-----
From: Arvind Murching [mailto:arvindm@windows.microsoft.com]
Sent: Thursday, January 17, 2002 4:57 PM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?


The argument should be the other way around, i.e. fields need to justify
their necessity to be part of a protocol.

Some reasons why not:

- LID values cannot be used across IB subnet boundaries
- A LID cannot be used as part of a static ARP entry
- The SLID is already present in case someone needs it

My 2c
- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 4:47 PM
To: ipoverib@ietf.org
Subject: [ipoverib] reasons for not using LID ?

By now it is obvious that there are the ones that wants LID's in the ARP
packets

Could the ones that don't give their reasons for why not to
Not just things like "you don't need it", "we want to save 16-bit", etc'
But, reasons for what problems could arouse from having it in the packet
I haven't heard any real argument that would make me consider removing
it
until now
we then will review them and see if it is possible to deal with those
cases,
if not I personally would drop my support in LID's

If people want it for their reasons, the ones that don't also need to do
some explaining

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib





_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Fri Jan 18 00:36:28 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA23214
	for <ipoverib-archive@odin.ietf.org>; Fri, 18 Jan 2002 00:36:28 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id AAA14850
	for ipoverib-archive@odin.ietf.org; Fri, 18 Jan 2002 00:36:31 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id AAA14814;
	Fri, 18 Jan 2002 00:33:27 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id AAA14782
	for <ipoverib@optimus.ietf.org>; Fri, 18 Jan 2002 00:33:25 -0500 (EST)
Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA23187
	for <ipoverib@ietf.org>; Fri, 18 Jan 2002 00:33:22 -0500 (EST)
Received: from westrelay01.boulder.ibm.com (westrelay01.boulder.ibm.com [9.99.140.22])
	by e31.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id AAA94298
	for <ipoverib@ietf.org>; Fri, 18 Jan 2002 00:30:21 -0500
Received: from d03nm036.boulder.ibm.com (avpilot.boulder.ibm.com [9.17.188.135])
	by westrelay01.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0I5XMo60582
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 22:33:22 -0700
Subject: RE: [ipoverib] reasons for not using LID ?
To: ipoverib@ietf.org
X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000
Message-ID: <OF28B27E32.4291E62C-ON88256B45.001E1DA6@boulder.ibm.com>
From: "Vivek Kashyap" <vivk@us.ibm.com>
Date: Thu, 17 Jan 2002 21:31:59 -0800
X-MIMETrack: Serialize by Router on D03NM036/03/M/IBM(Release 5.0.9 |November 16, 2001) at
 01/17/2002 10:33:21 PM
MIME-Version: 1.0
Content-type: text/plain; charset=us-ascii
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org


My comments in <VK> below.


The argument should be the other way around, i.e. fields need to justify
their necessity to be part of a protocol.
[YH] I think many arguments where already raised, now it's the
opposition
turn to put their arguments
[Arvind] The need for a redundant field is not obvious.

<VK> It doesn't become redundant because you want to get it via another
means. Reasons have been given by LID proponents that are very valid.
Reasons given by you against the LID are weak. None of the uses you want
are blocked by including the LID. See the earlier mail in reply to your
mail.
<VK>

Some reasons why not:

- LID values cannot be used across IB subnet boundaries
[YH] you would any way need to store both LID (Router) and GID (node) in
this case somewhere, you wouldn't go to the SM for any new IP packet
wont
you
the routers in this case will any way need to change the LID in the ARP
packets (Proxy ARP and all that)
I believe this case is the one that benefits most from the LID, avoiding
the
long and complex multi-subnet resolution process
[Arvind] Consider the most general case of an IB router routing between
IB subnets. This router will not modify an ARP reply packet to insert
its LID into the payload. However, it will insert its SLID into the IB
header of packets that it forwards. Proxy ARP boxes can do the same,
too.


- A LID cannot be used as part of a static ARP entry
[YH] We didn't say not to have GID, you could still put GID as a static
entry

- The SLID is already present in case someone needs it
[YH] also regular ARP has it, so why not follow that path
[Arvind] This is not an argument for adding redundant fields.


My 2c
- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 4:47 PM
To: ipoverib@ietf.org
Subject: [ipoverib] reasons for not using LID ?

By now it is obvious that there are the ones that wants LID's in the ARP
packets

Could the ones that don't give their reasons for why not to
Not just things like "you don't need it", "we want to save 16-bit", etc'
But, reasons for what problems could arouse from having it in the packet
I haven't heard any real argument that would make me consider removing
it
until now
we then will review them and see if it is possible to deal with those
cases,
if not I personally would drop my support in LID's

If people want it for their reasons, the ones that don't also need to do
some explaining

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib





_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib





_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Fri Jan 18 00:39:21 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA23294
	for <ipoverib-archive@odin.ietf.org>; Fri, 18 Jan 2002 00:39:21 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id AAA14981
	for ipoverib-archive@odin.ietf.org; Fri, 18 Jan 2002 00:39:23 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id AAA14908;
	Fri, 18 Jan 2002 00:36:49 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id AAA14867
	for <ipoverib@optimus.ietf.org>; Fri, 18 Jan 2002 00:36:47 -0500 (EST)
Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA23229
	for <ipoverib@ietf.org>; Fri, 18 Jan 2002 00:36:43 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e31.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id AAA118916
	for <ipoverib@ietf.org>; Fri, 18 Jan 2002 00:33:43 -0500
Received: from d03nm036.boulder.ibm.com (avpilot.boulder.ibm.com [9.17.188.135])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0I5ahA83786
	for <ipoverib@ietf.org>; Thu, 17 Jan 2002 22:36:43 -0700
Subject: RE: [ipoverib] reasons for not using LID ?
To: ipoverib@ietf.org
X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000
Message-ID: <OFC46EA45A.6D89C2E3-ON88256B45.001A30D6@boulder.ibm.com>
From: "Vivek Kashyap" <vivk@us.ibm.com>
Date: Thu, 17 Jan 2002 21:35:20 -0800
X-MIMETrack: Serialize by Router on D03NM036/03/M/IBM(Release 5.0.9 |November 16, 2001) at
 01/17/2002 10:36:43 PM
MIME-Version: 1.0
Content-type: text/plain; charset=us-ascii
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org


<VK> Before we go this path we need to know why we need to support IPoIB
subnets spanning IB subnets. We know that IB architecture is lacking a lot
many things before it can even support such non-routing protocol based
communications i.e. IB routers themselves are not fully specified, P_Key
sharing across IB subnets is not done et.c .etc. IB subnets provide
isolation. Why should they be joined together in IPoIB ? .

[ BTW, even if this is done cross-subnet ARP can work as explained below
and in more detail earlier. However, the WG needn't consider IPoIB subnets
spanning IB subnets unless shown some valid reason for it. ]

<VK>

I am very confused, why will an IB router have to modify an IP ARP packet
at
all.  If there is something in the spec that REQUIRES that I think I would
be in favor of ripping it out and fixing this...

Consider IB routers as looking like Ethernet switches, it is just a layer 2
device that sends packets to the correct collision domain.  I see that as
what an IB router is for the IP layer on top of it.  You would have a hard
time convincing me that you want the router sniffing and modifing ARP
packets.  This would be a huge HUGE layer violation.

Now here is your arguement for not including LIDs in the ARP response, I
want a single ARP response for everyone that sends me an ARP request in my
IP subnet.  If this IP subnet crosses an IB subnet, this LID will not be
valid (or the router has to munge it for me)  This seems a lot of work to
get past the case of a simple lookup to the SM that the IBTA Architecture
would have you do when you are communicating with a new endpoint.

<VK> The SM cannot help you here. It knows nothing of the GID from another
subnet. For that you need to access the routing infrastructure. Considering
the hypothetical case when such an IPoIB subnet exists the ARP module will
have to get the LID from the routing infrastructure. If the LID is included
it will overwrite it with that supplied from the routing infrastructure on
noting a 'foreign' GID-prefix. It will do the same when only the GID is
used. So there is no difference.

In the case of within an IB subnet, the only type of subnet possible, the
non-LID version will have a different step to follow (go to the SA) but the
contains-LID option won't have to do anything. The contains-LID option
doesn't stop anyone from going to the SA though.

<VK>

Bill

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Yaron Haviv
Sent: Thursday, January 17, 2002 5:46 PM
To: 'Arvind Murching'; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?


Arvind,

The only real argument you make is implementation in Routers
Since we build such boxes, I could tell you dealing with an ARP in the
router in case of IP span IB is complex anyway
Believe me, the simplification you are offering me creates more work than
help inside the router
As you also stated in the passed there are also very little amount of ARP
packets, so replacing it in one or two paces doesn't change anything

Yaron

-----Original Message-----
From: Arvind Murching [mailto:arvindm@windows.microsoft.com]
Sent: Friday, January 18, 2002 3:34 AM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

Below...

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 5:31 PM
To: Arvind Murching; yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

My remarks attached

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of
Arvind Murching
Sent: Friday, January 18, 2002 2:57 AM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

The argument should be the other way around, i.e. fields need to justify
their necessity to be part of a protocol.
[YH] I think many arguments where already raised, now it's the
opposition
turn to put their arguments
[Arvind] The need for a redundant field is not obvious.

Some reasons why not:

- LID values cannot be used across IB subnet boundaries
[YH] you would any way need to store both LID (Router) and GID (node) in
this case somewhere, you wouldn't go to the SM for any new IP packet
wont
you
the routers in this case will any way need to change the LID in the ARP
packets (Proxy ARP and all that)
I believe this case is the one that benefits most from the LID, avoiding
the
long and complex multi-subnet resolution process
[Arvind] Consider the most general case of an IB router routing between
IB subnets. This router will not modify an ARP reply packet to insert
its LID into the payload. However, it will insert its SLID into the IB
header of packets that it forwards. Proxy ARP boxes can do the same,
too.

- A LID cannot be used as part of a static ARP entry
[YH] We didn't say not to have GID, you could still put GID as a static
entry

- The SLID is already present in case someone needs it
[YH] also regular ARP has it, so why not follow that path
[Arvind] This is not an argument for adding redundant fields.


My 2c
- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 4:47 PM
To: ipoverib@ietf.org
Subject: [ipoverib] reasons for not using LID ?

By now it is obvious that there are the ones that wants LID's in the ARP
packets

Could the ones that don't give their reasons for why not to
Not just things like "you don't need it", "we want to save 16-bit", etc'
But, reasons for what problems could arouse from having it in the packet
I haven't heard any real argument that would make me consider removing
it
until now
we then will review them and see if it is possible to deal with those
cases,
if not I personally would drop my support in LID's

If people want it for their reasons, the ones that don't also need to do
some explaining

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib





_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Fri Jan 18 00:54:13 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA23450
	for <ipoverib-archive@odin.ietf.org>; Fri, 18 Jan 2002 00:54:13 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id AAA15375
	for ipoverib-archive@odin.ietf.org; Fri, 18 Jan 2002 00:54:13 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id AAA15287;
	Fri, 18 Jan 2002 00:51:18 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id AAA15234
	for <ipoverib@optimus.ietf.org>; Fri, 18 Jan 2002 00:51:15 -0500 (EST)
Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA23410
	for <ipoverib@ietf.org>; Fri, 18 Jan 2002 00:51:13 -0500 (EST)
Received: from westrelay01.boulder.ibm.com (westrelay01.boulder.ibm.com [9.99.140.22])
	by e31.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id AAA136650;
	Fri, 18 Jan 2002 00:48:12 -0500
Received: from d03nm036.boulder.ibm.com (avpilot.boulder.ibm.com [9.17.188.135])
	by westrelay01.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0I5pDo111656;
	Thu, 17 Jan 2002 22:51:13 -0700
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address? (SA down)
To: "Jim Pinkerton" <jpink@microsoft.com>
Cc: ipoverib@ietf.org
X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000
Message-ID: <OF27274595.0E6E736F-ON88256B45.001EC98C@boulder.ibm.com>
From: "Vivek Kashyap" <vivk@us.ibm.com>
Date: Thu, 17 Jan 2002 21:49:46 -0800
X-MIMETrack: Serialize by Router on D03NM036/03/M/IBM(Release 5.0.9 |November 16, 2001) at
 01/17/2002 10:51:13 PM
MIME-Version: 1.0
Content-type: text/plain; charset=us-ascii
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org


<VK>
Jim,
      How do you propose that the SA, if it cannot preserve the LIDs, keep
the GIDs recorded rightly ? GID prefix may live but what about the many
GIDs assigned to each port ? Additionally will it selectively keep all the
MGIDs in the database so that our subnet survives but loose the LIDs ? I
would expect a backup SA to always be able to share the database. At
minimum the backup can collect the existing information and build its
database and assign to only those that are unassigned. If that can't be
provided and the crash of an SA causes all of IB activity to come crashing
down (LIDs/GIDs lost - all SRP connections/IP/SDP connections down)  no one
will buy such an SM/SA. If IB cannot stay up through the crash of an SA
then IB won't survive either.

Vivek
<VK>

Roger,

Could you walk through how the LID is preserved across failover of a SA
(i.e. it died, for whatever reason, and the backup SA takes over)? To me
this is a critically important scenario for Data Centers (we both went
through the GSN ARP fiasco, and have battle scars to prove it). It seems
to me that there is no interoperable mechanism to forward the SM/SA
state to the backup system, and thus the LIDs could be potentially
re-assigned as part of the failover?

The GID prefix though, is generally something the network administrator
has access to (or leave as the default), and thus it is probably
reasonable to expect that the GID is in fact preserved across the
failover?



Jim




> -----Original Message-----
> From: Roger Ronald [mailto:rronald@VIEO.com]
> Sent: Thursday, January 17, 2002 12:14 PM
> To: Vivek Kashyap
> Cc: ipoverib@ietf.org
> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>
> I think it's important to note that changes in LID should be
infrequent.
> Very infrequent.
>
> VIEO is one of a small number of companies that has a SM at this time.
> We plan to maintain LID assignments whenever possible. That means
> across reboots, across power fails and across subnet merges (when
> possible).
>
> There are corner cases where LID may need to change, just like IP
> addresses occasionaly need to change. But these are VERY odd
> events...like the first time merge of fabrics. They won't happen
> hourly, daily, monthly..... They may happen once or twice in a
> fabric. Ever (unless the admin wants to make changes for the heck of
it).
>
> RR
>
>
> ----- Original Message -----
> From: "Vivek Kashyap" <kashyapv@us.ibm.com>
> To: "Bill Strahm" <bill@strahm.net>
> Cc: <ipoverib@ietf.org>
> Sent: Thursday, January 17, 2002 3:27 AM
> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>
>
> >
> > >
> > > Just free form thinking here...
> > >
> > > Could we say that GIDs are the equivalent of a Fully Qualified
Domain
> Name
> > > (www.strahm.net for example) and LIDs are the equivalent of IP
> addresses
> > > (63.105.18.42).  Now in many cases IP addresses are quite stable,
and
> do not
> > > change.  However would you ship an application with an IP address
> encoded in
> > > it, or with a FQDN.  It would be of course more efficient to use
the
> IP
> > > address and avoid the DNS lookup (that has to go VERY far away,
and
> can take
> > > quite a while for the first SYN message to get out), however if
the
> address
> > > EVER changes, things become hard.
> > >
> > > Can anyone tell me why this is a bad analogy ???
> >
> >         If it was pure IB then GID to LID would be a similar
analogy.
> >
> >         I'll try taking the your analogy further..a couple of points
> >         though - the SM corresonds to the zone primary but the nodes
> >         themselves corresond to both the secondary and the IP nodes
> >         asking for the resolution. So the analogy is not exactly apt
> >         but I'll try..
> >
> >         If the IP changes the DNS must be updated and looked up. In
> >         this case if LID changes the SM is updated (yes) but it then
> >         requires that the SM be looked up. There is no backlink here
> >         to trigger this lookup. This implies that the full lookup
all
> >         the way to the DNS root must be done - if like DNS it is
done
> >         till the local secondary server ie. referring to the local
> >         cached copy and/or merely re-receiving the peer's GID, the
> >         communication will fail. You must have seen the hell that
> >         breaks loose if any of the zone updates doesn't happen due
to
> >         some error and things are left pointing the wrong way.
> >
> >         Now, this would correspond to the internal LID cache flush -
> >         since the GID is the same but the LID has changed. There is
no
> >         hook to say, hey!, go do a zone transfer i.e. lookup the SM
> >         right now. You need a timer.
> >
> >         Now, you could make this time small - you are going to the
SM
> >         too often because things are not changing that much- make it
> >         long and LID change is surely going to happen at a critical
> >         time. Additionally this is another timer per entry for the
> >         programmers/maintainers/administrators to worry about.
> >
> >         I have all along been arguing against keeping LID out due to
> >         this reason too (code is more complex if no LID is another
> >         reason). It has been my assumption that if GID is used
without
> >         LID then one is forced to go to the SM at every address
> >         resolution. Every arp broadcast will have the whole subnet
> >         scurrying off to the SM to ask the requester's LID. Yes,
some
> >         would mark the entry to be completed but not necessarily - a
> >         thundering herd is likely.
> >
> >         This implies that if we don't use the LID we MUST ALWAYS go
to
> >         the SM for the LID - at every ARP since we don't know if LID
> >         has changed. Of course if the GID changed too then the SM
has
> >         to be consulted anyway.
> >
> >         Also note that if we did have a trap from the SM on every
LID
> >         change it still wouldn't be a good solution - too much
> >         latency; points of synchrnonisation; asynchronous updates;
and
> >         most of all the SM would end up doing nothing else but
issueing
> >         traps to every node that registered with it. IB cannot
mandate
> >         broadcasts/multicast (though making everyone look is bad
too)
> >         because multicasting is optional.
> >
> >         In the case of inclusion of the LID - some of the issues are
> >         the same but a gratuitous ARP solves the problem if the LID
> >         changed. It is an instant zone transfer. One could claim
that
> >         with GID only, on receiving any gratuitous ARP entry, the
node
> >         will go to the SM but that will require specialised coding
to
> >         recognise and handle gratuitous ARP differently than regular
> >         updates. Not a good idea.
> >
> >         Even in the absence of gratuitous ARP the changed LID will
be
> >         known when the ARP is done after a cache expiry if LID is
> >         included. If LID is not included one will get a GID back.
That
> >         will require the expiry of the GID to LID cache before one
> >         goes to the SM.
> >
> >         Inclusion of LID makes things very simple. This is because
LID
> >         is always necessary - can't escape it.
> >
> >         I also like inclusion of LID because it cuts off 2x latency
of
> >         GID to LID lookup in the SM and 4x the RTT and the realted
> >         extra interrupts. Fast failover in data centers is
necessary.
> >
> > Vivek
> >
> >
> >
> > >
> > > Bill
> > >
> > > -----Original Message-----
> > > From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On
> Behalf
> > > Of Larsen, Roy K
> > > Sent: Wednesday, January 16, 2002 6:43 PM
> > > To: 'H.K. Jerry Chu'; arvindm@windows.microsoft.com;
> kashyapv@us.ibm.com
> > > Cc: ipoverib@ietf.org
> > > Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
> > >
> > >
> > > > -----Original Message-----
> > > > From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
> > > > Sent: Wednesday, January 16, 2002 6:13 PM
> > > > To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com
> > > > Cc: ipoverib@ietf.org
> > > > Subject: Re: [Ipoverib] What's inside an IPoIB link layer
address?
> > > >
> > > >
> > > > All the proxy ARP usages that I know of use the proxy's mac
> > > > addr in the
> > > > ARP response. I believe the flavor of the proxy ARP you
> > > > alluded below is
> > > > rare, and won't work very well anyway if the link address
includes
> any
> > > > non-persistent component like LID and QPN because it will
> > > > require another
> > > > protocol between the clients and their proxy to constantly
update
> the
> > > > proxy database containing clients mac addrs, which may change
quite
> > > > often.
> > > >
> > > > That is also why I wanted to provide an option to not include
QPN.
> > > > (But I haven't succeeded w/o requiring SIDR or something
> equivalent.)
> > > >
> > > > If LID is included as part of link addr and if it ever changes
on
> > > > a live node due to subnet merge...etc, the driver must inform
its IP
> > > > stack about such change. I believe many stacks don't have such a
> > > > facility. (FOr most IP stacks the assumption is that its own MAC
> > > > addresses just don't change on the fly.) In that case you're
totally
> > > > hosed. Can someone comment on how likely a LID assignment gets
> > > > rescinded?
> > >
> > > It is possible anytime the link goes down and comes back up again
and
> when
> > > the fabic is reconfigured.  This can happen on reboot or simply
> > > unplugging/plugging in a link.  It is ultimately a function of the
SM
> and
> > > how much trouble the implementation wants to go through to keep
LIDs
> > > constant.
> > >
> > > But of course, that has no bearing on whether the LID is contained
in
> the
> > > ARP.  You always need the LID.  Unless you want to do a pathrecord
> lookup
> > > each time your implementation is given a packet to send, you're
going
> to use
> > > a cached LID no matter if it came in the ARP or a pathrecord
lookup on
> the
> > > GID.  If it becomes stale, it may take an ARP cache flush timeout
to
> > > recover.
> > >
> > > Roy
> > >
> > > >
> > > > Jerry
> > > >
> > > > >        A device can always respond for another. The LID of
> > > > the sender
> > > > >        is not the same as the LID for which it has
> > > > responded. It also
> > > > >        allows any network monitoring/debugging device to work
> > > > >        independently. This is the reason why none of the other
> media
> > > > >        does anything like this. You don't pick ethernet source
> > > > >        address from the frame but take it from the ARP/ND
packet.
> > > > >
> > > > >        [ If it was ok we wouldn't need anything except
> QPN/cap_flags
> > > > >        - get the LID from response, consult SA using LID for
GID
> if
> > > > >        you want that, and you are done. ]
> > > > >
> > > > >Vivek
> > > > >
> > > > >>
> > > > >> Why do we need to include the LID inside the link layer
> > > > address when it
> > > > >> is already available in the SLID field? Previous
> > > > discussions indicate
> > > > >> that the SLID value works for proxy ARP/bridge
implementations.
> > > > >>
> > > > >> - Arvind
> > > > >>
> > > > >> -----Original Message-----
> > > > >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> > > > >> Sent: Wednesday, January 16, 2002 10:29 AM
> > > > >> To: kamath@seagullsemi.com
> > > > >> Cc: ipoverib@ietf.org
> > > > >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer
> address?
> > > > >>
> > > > >> Ashwin,
> > > > >> Ashwin,
> > > > >>
> > > > >>         I agree that you get more flexibility if you
> > > > include the LID.
> > > > >>         If we include the GID along with the LID we allow all
> > > > >>         implementations the choice to create different
> > > > solutions. We
> > > > >>         don't tie them down to one thing. Choosing one or the
> other
> > > > >>         (LID or GID) blocks out various advantages that
different
> > > > >>         implementations seek. Hence I suggested that let us
> include
> > > > >>         both and move on.
> > > > >>
> > > > >> Vivek
> > > > >>
> > > > >> <snip>
> > > > >>
> > > > >> _______________________________________________
> > > > >> IPoverIB mailing list
> > > > >> IPoverIB@ietf.org
> > > > >> http://www.ietf.org/mailman/listinfo/ipoverib
> > > > >>
> > > > >
> > > > >
> > > > >--
> > > > >Vivek Kashyap
> > > > >Linux Technology Center, IBM
> > > > >kashyapv@us.ibm.com
> > > > >vivk@us.ibm.com
> > > > >503 578 3422 (o)
> > > > >
> > > > >_______________________________________________
> > > > >IPoverIB mailing list
> > > > >IPoverIB@ietf.org
> > > > >http://www.ietf.org/mailman/listinfo/ipoverib
> > > >
> > > >
> > > > _______________________________________________
> > > > IPoverIB mailing list
> > > > IPoverIB@ietf.org
> > > > http://www.ietf.org/mailman/listinfo/ipoverib
> > > >
> > >
> > > _______________________________________________
> > > IPoverIB mailing list
> > > IPoverIB@ietf.org
> > > http://www.ietf.org/mailman/listinfo/ipoverib
> > >
> > >
> > > _______________________________________________
> > > IPoverIB mailing list
> > > IPoverIB@ietf.org
> > > http://www.ietf.org/mailman/listinfo/ipoverib
> > >
> >
> >
> > --
> > Vivek Kashyap
> > Linux Technology Center, IBM
> > kashyapv@us.ibm.com
> > vivk@us.ibm.com
> > 503 578 3422 (o)
> >
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> >
> >
>
>
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> http://www.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib





_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Fri Jan 18 07:59:09 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA09043
	for <ipoverib-archive@odin.ietf.org>; Fri, 18 Jan 2002 07:59:09 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id HAA10410
	for ipoverib-archive@odin.ietf.org; Fri, 18 Jan 2002 07:59:12 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id HAA10296;
	Fri, 18 Jan 2002 07:53:15 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id HAA10262
	for <ipoverib@optimus.ietf.org>; Fri, 18 Jan 2002 07:53:10 -0500 (EST)
Received: from mxout1.netvision.net.il (mxout1.netvision.net.il [194.90.9.20])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA08887
	for <ipoverib@ietf.org>; Fri, 18 Jan 2002 07:53:03 -0500 (EST)
Received: from vaio505he ([139.92.208.56]) by mxout1.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GQ4009DYX3HFV@mxout1.netvision.net.il> for ipoverib@ietf.org;
 Fri, 18 Jan 2002 14:52:32 +0200 (IST)
Date: Fri, 18 Jan 2002 14:53:08 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: RE: [ipoverib] reasons for not using LID ?
In-reply-to: <HHEDIJGOEFLPCPNILMPAEEBACFAA.bill@strahm.net>
To: "'Bill Strahm'" <bill@strahm.net>, ipoverib@ietf.org,
        "'Libor Michalek'" <libor@pobox.com>
Reply-to: yaronh@voltaire.com
Message-id: <001501c1a01f$161156f0$61ac003e@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7BIT

Bill,

First when forwarding packets from one subnet to another, the LRH header
needs to be reconstructed
This involved replacing both the DLID & SLID in the new header, so we do
deal with the packet
And we do have the LID'd in the packet LRH header anyway
And we don't keep the same LID's in a way that the destination will get a
LID which belongs to a different subnet

You assume that the IB routers works like a layer 2 Ethernet switch
Unfortunately this is not the case, since as mike noted I couldn't reveal
the content of the LWG discussions I cannot go into it
Lets just say as one of the people active in the router sub-group that it is
totally different and a lot of things still needs to be determined
I would choose at least initially to implement something like Proxy ARP to
deal with the Issues
One difference I could point out is that the Ethernet spanning tree is a
self-learning mechanism where as IB routing is not
The Proxy ARP mechanism compensates for that fact

I'm trying to stay away from the theory and see how today I would implement
those things the most efficient way
When digging into it and going over all the potential problems, I would
personally like to avoid the SM and especially when going to a different
subnet
And not just because of the few extra milliseconds delay, and SM overhead,
but because I know it could work in all the scenarios

One thing worth digging into is how to enable avoiding the SM using the SLID
and other extra information that will enable detecting when that LID is
invalid, plus maybe the Valid LID, this way if/when IB routing will be
different it might answer your desires too

Yaron

 From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Bill Strahm
Sent: Friday, January 18, 2002 6:29 AM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

I do not agree with your analysis in your first answer.  Yes the IB router
would receive the packet on its first interface and forward it on its second
interface replacing the DLID with its own DLID.  This is what a layer 3
router does.

Now look at the Infiniband fabric as a layer 2 for the IP layer 3.  At this
point you want layer 2 semantics.  At this point IB routers look like layer
2 switches and MUST NOT interfere with the layer 3 packets on top (otherwise
it is a layer 3 router, and it might as well do IP routing and we don't have
the subnet spanning problem at all)

The job of an IB router in an infiniband fabric is much simpler if we do not
have LID in the ARP packet at all, and just have the destination node poll
the SM for the LID based on the GID.

Bill

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Yaron Haviv
Sent: Thursday, January 17, 2002 6:38 PM
To: 'Bill Strahm'; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?


Bill, the answer to your first question relays in your second one

When IP spans IB, the router receive a packet with the local LID of the
first subnet (even when it is just in the S/DLID) and needs to modify it to
a packet with a local LID of the target subnet
Otherwise it won't get to the target, or returned to the sender
I would be glad to know what is your trick for avoid that
To avoid loops the router will need to implement a proxy ARP, which
maintains the virtualized routing tree
Looking at it like a layer 2 switch is to much of a theoretical abstraction

Answering the 2nd Q, by now you see that the ARP been sent to the target has
the router LID, and when you will reply it will get just to the right place
The end node operation doesn't change in any case

Yaron

-----Original Message-----
From: Bill Strahm [mailto:bill@strahm.net]
Sent: Friday, January 18, 2002 4:10 AM
To: yaronh@voltaire.com; 'Arvind Murching'; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

I am very confused, why will an IB router have to modify an IP ARP packet at
all.  If there is something in the spec that REQUIRES that I think I would
be in favor of ripping it out and fixing this...

Consider IB routers as looking like Ethernet switches, it is just a layer 2
device that sends packets to the correct collision domain.  I see that as
what an IB router is for the IP layer on top of it.  You would have a hard
time convincing me that you want the router sniffing and modifing ARP
packets.  This would be a huge HUGE layer violation.

Now here is your arguement for not including LIDs in the ARP response, I
want a single ARP response for everyone that sends me an ARP request in my
IP subnet.  If this IP subnet crosses an IB subnet, this LID will not be
valid (or the router has to munge it for me)  This seems a lot of work to
get past the case of a simple lookup to the SM that the IBTA Architecture
would have you do when you are communicating with a new endpoint.

Bill

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Yaron Haviv
Sent: Thursday, January 17, 2002 5:46 PM
To: 'Arvind Murching'; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?


Arvind,

The only real argument you make is implementation in Routers
Since we build such boxes, I could tell you dealing with an ARP in the
router in case of IP span IB is complex anyway
Believe me, the simplification you are offering me creates more work than
help inside the router
As you also stated in the passed there are also very little amount of ARP
packets, so replacing it in one or two paces doesn't change anything

Yaron

-----Original Message-----
From: Arvind Murching [mailto:arvindm@windows.microsoft.com]
Sent: Friday, January 18, 2002 3:34 AM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

Below...

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 5:31 PM
To: Arvind Murching; yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

My remarks attached

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of
Arvind Murching
Sent: Friday, January 18, 2002 2:57 AM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

The argument should be the other way around, i.e. fields need to justify
their necessity to be part of a protocol.
[YH] I think many arguments where already raised, now it's the
opposition
turn to put their arguments
[Arvind] The need for a redundant field is not obvious.

Some reasons why not:

- LID values cannot be used across IB subnet boundaries
[YH] you would any way need to store both LID (Router) and GID (node) in
this case somewhere, you wouldn't go to the SM for any new IP packet
wont
you
the routers in this case will any way need to change the LID in the ARP
packets (Proxy ARP and all that)
I believe this case is the one that benefits most from the LID, avoiding
the
long and complex multi-subnet resolution process
[Arvind] Consider the most general case of an IB router routing between
IB subnets. This router will not modify an ARP reply packet to insert
its LID into the payload. However, it will insert its SLID into the IB
header of packets that it forwards. Proxy ARP boxes can do the same,
too.

- A LID cannot be used as part of a static ARP entry
[YH] We didn't say not to have GID, you could still put GID as a static
entry

- The SLID is already present in case someone needs it
[YH] also regular ARP has it, so why not follow that path
[Arvind] This is not an argument for adding redundant fields.


My 2c
- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 4:47 PM
To: ipoverib@ietf.org
Subject: [ipoverib] reasons for not using LID ?

By now it is obvious that there are the ones that wants LID's in the ARP
packets

Could the ones that don't give their reasons for why not to
Not just things like "you don't need it", "we want to save 16-bit", etc'
But, reasons for what problems could arouse from having it in the packet
I haven't heard any real argument that would make me consider removing
it
until now
we then will review them and see if it is possible to deal with those
cases,
if not I personally would drop my support in LID's

If people want it for their reasons, the ones that don't also need to do
some explaining

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Fri Jan 18 09:45:38 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA12995
	for <ipoverib-archive@odin.ietf.org>; Fri, 18 Jan 2002 09:45:38 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id JAA15108
	for ipoverib-archive@odin.ietf.org; Fri, 18 Jan 2002 09:45:40 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id JAA14955;
	Fri, 18 Jan 2002 09:38:39 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id JAA14905
	for <ipoverib@optimus.ietf.org>; Fri, 18 Jan 2002 09:38:36 -0500 (EST)
Received: from ztxmail05.ztx.compaq.com (ztxmail05.ztx.compaq.com [161.114.1.209])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA12698
	for <ipoverib@ietf.org>; Fri, 18 Jan 2002 09:38:32 -0500 (EST)
Received: by ztxmail05.ztx.compaq.com (Postfix, from userid 12345)
	id 09677431B; Fri, 18 Jan 2002 08:38:04 -0600 (CST)
Received: from cceexg11.americas.cpqcorp.net (cceexg11.americas.cpqcorp.net [16.110.250.125])
	by ztxmail05.ztx.compaq.com (Postfix) with ESMTP
	id F071F41F1; Fri, 18 Jan 2002 08:38:03 -0600 (CST)
Received: from cceexc18.americas.cpqcorp.net ([16.110.250.64]) by cceexg11.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966);
	 Fri, 18 Jan 2002 08:38:03 -0600
X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="windows-1255"
Subject: RE: [ipoverib] reasons for not using LID ?
Date: Fri, 18 Jan 2002 08:37:12 -0600
Message-ID: <4D027986353D1341ADA4F2A4F8A170760E3175@cceexc18.americas.cpqcorp.net>
Thread-Topic: [ipoverib] reasons for not using LID ?
Thread-Index: AcGfuaANTvuihY/bQT+vzYpdtPA3tQAbVpjg
From: "Culley, Paul" <Paul.Culley@compaq.com>
To: <yaronh@voltaire.com>, <ipoverib@ietf.org>
X-OriginalArrivalTime: 18 Jan 2002 14:38:03.0732 (UTC) FILETIME=[BC5DC940:01C1A02D]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id JAA14906
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

1) a LID in the body of the of the ARP is insufficient for IB routing,
you need the LID/SL pair in order to actually get a packet anywhere.  IB
provides NO WAY to select an SL, based only on the LID, unless the ARP
requestor already has that LID and an SL cached or is willing to look it
up with the SA.  Since one of the reasons for including the LID at all
is to avoid this lookup, the solution is not complete without SL.
[Please don't respond that some implementation specific mechanism can be
chosen to select SL; there is no way for an endnode implementation to
determine the SA policies for SL usage at this time, so it is not
possible for the endnode to reliably chose an SL without asking the SA
(or caching a LID/SL derived from the SA).

2) LID in the body of the ARP will be invalid if the ARP crosses IB
subnets.  This has been discussed a lot already, but the following
observations seem pertinent:
 a) IB routers will happen someday, it would be nice to not have to
  go back to this process to fix ARP when it does happen.
 b) The IB Management group currently intends to support cross
  subnet operation for IB via use of the same SA PathRecord Lookup
  mechanism as is used for local operations.  This is primarily so
  that early endnode implementations can operate without changes.
  It is possible that other mechanisms may be defined as
  optimizations for routers, but that is at best an early concept. If
  we support a GID lookup for the ARP packets (and either drop body
  LIDs, use IB Header SLIDs, or indicate when to ignore them) ARP and
  IP can work across IB subnet boundaries.
 c) It is certain that IB routers will modify IB LRH headers;
  choosing new values for SLID/DLID/SL at subnet boundaries.  And a
  requirement on IB subnets and routers is that a unicast packet's
  SLID/SL can be used to return a packet to the source allows using
  SLID/SL in any case where the source of the ARP is also the place
  comm packets go.
 d) It seems a significant violation of layering to require an IB
  router to understand an IP ARP, just so that it can modify a LID in
  the body of the packet.

3) As it happens, it is perfectly legal for an IB fabric to use
different LID addresses to talk to the same device from different other
devices, that is one of the reasons that we allow up to 2^LMC addresses
for each device.  (That's right, there are potentially up to 128 LID
addresses for each port!)  And not all LID/SL addresses need be
reachable from all other devices.  This means that taking a LID from the
ARP Request as a way to communicate, may not work.  And it is not
currently possible for an endnode to determine in a defined way that the
SA uses this type of policy, so endnodes have NO architected WAY to tell
when to ignore the LID and go for GID on their own.

As an aside, this means that gratuitous multicast ARP and snooping the
multicast ARP requests is only good for the GID.  An endnode might
perform the snoop, but would have to keep track of the fact that the
LID/SL were not valid, probably somewhere in the ARP module.

My preference is to include only the GID in the body of the ARP request
or response, along with a bit that indicates when the IB Header's
LRH:SLID/SL is known to be invalid by the sender.  This bit would always
be set for multicast ARP requests, and usually reset for unicast ARP
responses.  It can be set in the ARP response if the responder KNOWs
that it is responding for some other port on the IB fabric.

Given this method, the majority of the time the endnodes could get
valid, best effort, LID/SL combinations directly from the ARP response
packets, and cache them.  The endnode could also snoop the ARP requests
or gratuitous ARPs for GIDs, and perform SA lookups when it needed to
send to one of the devices which only had the GID in the table.

Paul R. Culley
Compaq Sr. Fellow
281-514-5543


-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 6:47 PM
To: ipoverib@ietf.org
Subject: [ipoverib] reasons for not using LID ?


By now it is obvious that there are the ones that wants LID's in the ARP
packets

Could the ones that don't give their reasons for why not to
Not just things like “you don't need it”, “we want to save 16-bit”, etc'
But, reasons for what problems could arouse from having it in the packet
I haven't heard any real argument that would make me consider removing
it
until now
we then will review them and see if it is possible to deal with those
cases,
if not I personally would drop my support in LID's

If people want it for their reasons, the ones that don't also need to do
some explaining

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Fri Jan 18 09:45:39 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA13005
	for <ipoverib-archive@odin.ietf.org>; Fri, 18 Jan 2002 09:45:38 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id JAA15122
	for ipoverib-archive@odin.ietf.org; Fri, 18 Jan 2002 09:45:40 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id JAA14782;
	Fri, 18 Jan 2002 09:33:44 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id JAA14752
	for <ipoverib@optimus.ietf.org>; Fri, 18 Jan 2002 09:33:42 -0500 (EST)
Received: from vieo.com (vieo.com [216.30.79.131])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA12538
	for <ipoverib@ietf.org>; Fri, 18 Jan 2002 09:33:39 -0500 (EST)
Received: (from root@localhost)
	by vieo.com (8.11.2/8.11.2) id g0IEXBh36666
	for ipoverib@ietf.org; Fri, 18 Jan 2002 08:33:11 -0600 (CST)
	(envelope-from rronald@VIEO.com)
Received: from Fishhead (dhcp-10-2-15-151.vieo.com [10.2.15.151])
	by vieo.com (8.11.2/8.11.2) with SMTP id g0IEX6c36539;
	Fri, 18 Jan 2002 08:33:06 -0600 (CST)
	(envelope-from rronald@VIEO.com)
Message-ID: <003c01c1a02d$024b3420$970f020a@Fishhead>
From: "Roger Ronald" <rronald@VIEO.com>
To: "Jim Pinkerton" <jpink@microsoft.com>,
        "Vivek Kashyap" <kashyapv@us.ibm.com>
Cc: <ipoverib@ietf.org>, "Jeff Young" <jsy@VIEO.com>
References: <24A715275661C8428C00432EFCA5CB7C055F9AFC@red-msg-02.redmond.corp.microsoft.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
Date: Fri, 18 Jan 2002 08:32:50 -0600
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 5.50.4133.2400
X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400
X-scanner: scanned by Inflex 0.1.5c+ on vieo.com
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

I should probably have Jeff Young answer this, but he's been
holed up in a Denver hotel room...having gotten a "bug" while
traveling to the Mgmt Working Group meeting.

I can take a shot.

I believe that your statement noting the "lack of an interoperable mechanism"
to replicate state information in the current IBTA spec is correct. I think
that the Mgmt working group is trying to do some new work in this area, but
I am not sure if that will resolve the problem. My ignorance here is a blessing 
since we aren't supposed to discuss IBTA work in progress.   :-)

Personally, I don't see the lack of an "interoperable method" as a big problem.
Big data centers aren't (IMHO) going to spend millions on IB enabled gear
and then allow the default priorities of the equipment to select a Subnet Manager.
Subnet management will be important and the data center will configure
their system to use the SM(s) that they select as best....based on performance,
reliability, compliance, and (most relevant) high availability. Since 
consistent performance and high availability are unlikely to be exactly
equal from different vendors, my theory is that you can count on the data
center manager picking exactly ONE vendor's SM(s).

Given that, all that your problem needs is a vendor solution to a high
availability subnet. With any solution (vendor or interoperable spec),
LIDs can be maintained just as well as a GID prefix can. The only 
time I would expect these (both LID and GID prefix) to ever change
would be based on admin intervention or a "first time" subnet merge.

I don't know about other vendors (thanks for being such a great straight man),
but VIEO does plan to make the SM function highly available along with providing
persistancy for important data including LIDs and GID prefixes.

RR
----- Original Message ----- 
From: "Jim Pinkerton" <jpink@microsoft.com>
To: "Roger Ronald" <rronald@vieo.com>; "Vivek Kashyap" <kashyapv@us.ibm.com>
Cc: <ipoverib@ietf.org>
Sent: Thursday, January 17, 2002 5:49 PM
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?


> 
> Roger,
> 
> Could you walk through how the LID is preserved across failover of a SA
> (i.e. it died, for whatever reason, and the backup SA takes over)? To me
> this is a critically important scenario for Data Centers (we both went
> through the GSN ARP fiasco, and have battle scars to prove it). It seems
> to me that there is no interoperable mechanism to forward the SM/SA
> state to the backup system, and thus the LIDs could be potentially
> re-assigned as part of the failover? 
> 
> The GID prefix though, is generally something the network administrator
> has access to (or leave as the default), and thus it is probably
> reasonable to expect that the GID is in fact preserved across the
> failover?
> 
> 
> 
> Jim
> 
> 
> 
> 
> > -----Original Message-----
> > From: Roger Ronald [mailto:rronald@VIEO.com]
> > Sent: Thursday, January 17, 2002 12:14 PM
> > To: Vivek Kashyap
> > Cc: ipoverib@ietf.org
> > Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> > 
> > I think it's important to note that changes in LID should be
> infrequent.
> > Very infrequent.
> > 
> > VIEO is one of a small number of companies that has a SM at this time.
> > We plan to maintain LID assignments whenever possible. That means
> > across reboots, across power fails and across subnet merges (when
> > possible).
> > 
> > There are corner cases where LID may need to change, just like IP
> > addresses occasionaly need to change. But these are VERY odd
> > events...like the first time merge of fabrics. They won't happen
> > hourly, daily, monthly..... They may happen once or twice in a
> > fabric. Ever (unless the admin wants to make changes for the heck of
> it).
> > 
> > RR
> > 
> > 
> > ----- Original Message -----
> > From: "Vivek Kashyap" <kashyapv@us.ibm.com>
> > To: "Bill Strahm" <bill@strahm.net>
> > Cc: <ipoverib@ietf.org>
> > Sent: Thursday, January 17, 2002 3:27 AM
> > Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> > 
> > 
> > >
> > > >
> > > > Just free form thinking here...
> > > >
> > > > Could we say that GIDs are the equivalent of a Fully Qualified
> Domain
> > Name
> > > > (www.strahm.net for example) and LIDs are the equivalent of IP
> > addresses
> > > > (63.105.18.42).  Now in many cases IP addresses are quite stable,
> and
> > do not
> > > > change.  However would you ship an application with an IP address
> > encoded in
> > > > it, or with a FQDN.  It would be of course more efficient to use
> the
> > IP
> > > > address and avoid the DNS lookup (that has to go VERY far away,
> and
> > can take
> > > > quite a while for the first SYN message to get out), however if
> the
> > address
> > > > EVER changes, things become hard.
> > > >
> > > > Can anyone tell me why this is a bad analogy ???
> > >
> > >         If it was pure IB then GID to LID would be a similar
> analogy.
> > >
> > >         I'll try taking the your analogy further..a couple of points
> > >         though - the SM corresonds to the zone primary but the nodes
> > >         themselves corresond to both the secondary and the IP nodes
> > >         asking for the resolution. So the analogy is not exactly apt
> > >         but I'll try..
> > >
> > >         If the IP changes the DNS must be updated and looked up. In
> > >         this case if LID changes the SM is updated (yes) but it then
> > >         requires that the SM be looked up. There is no backlink here
> > >         to trigger this lookup. This implies that the full lookup
> all
> > >         the way to the DNS root must be done - if like DNS it is
> done
> > >         till the local secondary server ie. referring to the local
> > >         cached copy and/or merely re-receiving the peer's GID, the
> > >         communication will fail. You must have seen the hell that
> > >         breaks loose if any of the zone updates doesn't happen due
> to
> > >         some error and things are left pointing the wrong way.
> > >
> > >         Now, this would correspond to the internal LID cache flush -
> > >         since the GID is the same but the LID has changed. There is
> no
> > >         hook to say, hey!, go do a zone transfer i.e. lookup the SM
> > >         right now. You need a timer.
> > >
> > >         Now, you could make this time small - you are going to the
> SM
> > >         too often because things are not changing that much- make it
> > >         long and LID change is surely going to happen at a critical
> > >         time. Additionally this is another timer per entry for the
> > >         programmers/maintainers/administrators to worry about.
> > >
> > >         I have all along been arguing against keeping LID out due to
> > >         this reason too (code is more complex if no LID is another
> > >         reason). It has been my assumption that if GID is used
> without
> > >         LID then one is forced to go to the SM at every address
> > >         resolution. Every arp broadcast will have the whole subnet
> > >         scurrying off to the SM to ask the requester's LID. Yes,
> some
> > >         would mark the entry to be completed but not necessarily - a
> > >         thundering herd is likely.
> > >
> > >         This implies that if we don't use the LID we MUST ALWAYS go
> to
> > >         the SM for the LID - at every ARP since we don't know if LID
> > >         has changed. Of course if the GID changed too then the SM
> has
> > >         to be consulted anyway.
> > >
> > >         Also note that if we did have a trap from the SM on every
> LID
> > >         change it still wouldn't be a good solution - too much
> > >         latency; points of synchrnonisation; asynchronous updates;
> and
> > >         most of all the SM would end up doing nothing else but
> issueing
> > >         traps to every node that registered with it. IB cannot
> mandate
> > >         broadcasts/multicast (though making everyone look is bad
> too)
> > >         because multicasting is optional.
> > >
> > >         In the case of inclusion of the LID - some of the issues are
> > >         the same but a gratuitous ARP solves the problem if the LID
> > >         changed. It is an instant zone transfer. One could claim
> that
> > >         with GID only, on receiving any gratuitous ARP entry, the
> node
> > >         will go to the SM but that will require specialised coding
> to
> > >         recognise and handle gratuitous ARP differently than regular
> > >         updates. Not a good idea.
> > >
> > >         Even in the absence of gratuitous ARP the changed LID will
> be
> > >         known when the ARP is done after a cache expiry if LID is
> > >         included. If LID is not included one will get a GID back.
> That
> > >         will require the expiry of the GID to LID cache before one
> > >         goes to the SM.
> > >
> > >         Inclusion of LID makes things very simple. This is because
> LID
> > >         is always necessary - can't escape it.
> > >
> > >         I also like inclusion of LID because it cuts off 2x latency
> of
> > >         GID to LID lookup in the SM and 4x the RTT and the realted
> > >         extra interrupts. Fast failover in data centers is
> necessary.
> > >
> > > Vivek
> > >
> > >
> > >
> > > >
> > > > Bill
> > > >
> > > > -----Original Message-----
> > > > From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On
> > Behalf
> > > > Of Larsen, Roy K
> > > > Sent: Wednesday, January 16, 2002 6:43 PM
> > > > To: 'H.K. Jerry Chu'; arvindm@windows.microsoft.com;
> > kashyapv@us.ibm.com
> > > > Cc: ipoverib@ietf.org
> > > > Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
> > > > > Sent: Wednesday, January 16, 2002 6:13 PM
> > > > > To: arvindm@windows.microsoft.com; kashyapv@us.ibm.com
> > > > > Cc: ipoverib@ietf.org
> > > > > Subject: Re: [Ipoverib] What's inside an IPoIB link layer
> address?
> > > > >
> > > > >
> > > > > All the proxy ARP usages that I know of use the proxy's mac
> > > > > addr in the
> > > > > ARP response. I believe the flavor of the proxy ARP you
> > > > > alluded below is
> > > > > rare, and won't work very well anyway if the link address
> includes
> > any
> > > > > non-persistent component like LID and QPN because it will
> > > > > require another
> > > > > protocol between the clients and their proxy to constantly
> update
> > the
> > > > > proxy database containing clients mac addrs, which may change
> quite
> > > > > often.
> > > > >
> > > > > That is also why I wanted to provide an option to not include
> QPN.
> > > > > (But I haven't succeeded w/o requiring SIDR or something
> > equivalent.)
> > > > >
> > > > > If LID is included as part of link addr and if it ever changes
> on
> > > > > a live node due to subnet merge...etc, the driver must inform
> its IP
> > > > > stack about such change. I believe many stacks don't have such a
> > > > > facility. (FOr most IP stacks the assumption is that its own MAC
> > > > > addresses just don't change on the fly.) In that case you're
> totally
> > > > > hosed. Can someone comment on how likely a LID assignment gets
> > > > > rescinded?
> > > >
> > > > It is possible anytime the link goes down and comes back up again
> and
> > when
> > > > the fabic is reconfigured.  This can happen on reboot or simply
> > > > unplugging/plugging in a link.  It is ultimately a function of the
> SM
> > and
> > > > how much trouble the implementation wants to go through to keep
> LIDs
> > > > constant.
> > > >
> > > > But of course, that has no bearing on whether the LID is contained
> in
> > the
> > > > ARP.  You always need the LID.  Unless you want to do a pathrecord
> > lookup
> > > > each time your implementation is given a packet to send, you're
> going
> > to use
> > > > a cached LID no matter if it came in the ARP or a pathrecord
> lookup on
> > the
> > > > GID.  If it becomes stale, it may take an ARP cache flush timeout
> to
> > > > recover.
> > > >
> > > > Roy
> > > >
> > > > >
> > > > > Jerry
> > > > >
> > > > > >        A device can always respond for another. The LID of
> > > > > the sender
> > > > > >        is not the same as the LID for which it has
> > > > > responded. It also
> > > > > >        allows any network monitoring/debugging device to work
> > > > > >        independently. This is the reason why none of the other
> > media
> > > > > >        does anything like this. You don't pick ethernet source
> > > > > >        address from the frame but take it from the ARP/ND
> packet.
> > > > > >
> > > > > >        [ If it was ok we wouldn't need anything except
> > QPN/cap_flags
> > > > > >        - get the LID from response, consult SA using LID for
> GID
> > if
> > > > > >        you want that, and you are done. ]
> > > > > >
> > > > > >Vivek
> > > > > >
> > > > > >>
> > > > > >> Why do we need to include the LID inside the link layer
> > > > > address when it
> > > > > >> is already available in the SLID field? Previous
> > > > > discussions indicate
> > > > > >> that the SLID value works for proxy ARP/bridge
> implementations.
> > > > > >>
> > > > > >> - Arvind
> > > > > >>
> > > > > >> -----Original Message-----
> > > > > >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> > > > > >> Sent: Wednesday, January 16, 2002 10:29 AM
> > > > > >> To: kamath@seagullsemi.com
> > > > > >> Cc: ipoverib@ietf.org
> > > > > >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer
> > address?
> > > > > >>
> > > > > >> Ashwin,
> > > > > >> Ashwin,
> > > > > >>
> > > > > >>         I agree that you get more flexibility if you
> > > > > include the LID.
> > > > > >>         If we include the GID along with the LID we allow all
> > > > > >>         implementations the choice to create different
> > > > > solutions. We
> > > > > >>         don't tie them down to one thing. Choosing one or the
> > other
> > > > > >>         (LID or GID) blocks out various advantages that
> different
> > > > > >>         implementations seek. Hence I suggested that let us
> > include
> > > > > >>         both and move on.
> > > > > >>
> > > > > >> Vivek
> > > > > >>
> > > > > >> <snip>
> > > > > >>
> > > > > >> _______________________________________________
> > > > > >> IPoverIB mailing list
> > > > > >> IPoverIB@ietf.org
> > > > > >> http://www.ietf.org/mailman/listinfo/ipoverib
> > > > > >>
> > > > > >
> > > > > >
> > > > > >--
> > > > > >Vivek Kashyap
> > > > > >Linux Technology Center, IBM
> > > > > >kashyapv@us.ibm.com
> > > > > >vivk@us.ibm.com
> > > > > >503 578 3422 (o)
> > > > > >
> > > > > >_______________________________________________
> > > > > >IPoverIB mailing list
> > > > > >IPoverIB@ietf.org
> > > > > >http://www.ietf.org/mailman/listinfo/ipoverib
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > IPoverIB mailing list
> > > > > IPoverIB@ietf.org
> > > > > http://www.ietf.org/mailman/listinfo/ipoverib
> > > > >
> > > >
> > > > _______________________________________________
> > > > IPoverIB mailing list
> > > > IPoverIB@ietf.org
> > > > http://www.ietf.org/mailman/listinfo/ipoverib
> > > >
> > > >
> > > > _______________________________________________
> > > > IPoverIB mailing list
> > > > IPoverIB@ietf.org
> > > > http://www.ietf.org/mailman/listinfo/ipoverib
> > > >
> > >
> > >
> > > --
> > > Vivek Kashyap
> > > Linux Technology Center, IBM
> > > kashyapv@us.ibm.com
> > > vivk@us.ibm.com
> > > 503 578 3422 (o)
> > >
> > > _______________________________________________
> > > IPoverIB mailing list
> > > IPoverIB@ietf.org
> > > http://www.ietf.org/mailman/listinfo/ipoverib
> > >
> > >
> > 
> > 
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > http://www.ietf.org/mailman/listinfo/ipoverib
> 
> 


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Fri Jan 18 11:57:51 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA18643
	for <ipoverib-archive@odin.ietf.org>; Fri, 18 Jan 2002 11:57:51 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id LAA22495
	for ipoverib-archive@odin.ietf.org; Fri, 18 Jan 2002 11:57:54 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id LAA21657;
	Fri, 18 Jan 2002 11:33:50 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id LAA21626
	for <ipoverib@optimus.ietf.org>; Fri, 18 Jan 2002 11:33:48 -0500 (EST)
Received: from pdxpo.dsl-only.net (pdxpo.dsl-only.net [63.105.16.3])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA17592
	for <ipoverib@ietf.org>; Fri, 18 Jan 2002 11:33:44 -0500 (EST)
Received: from bonfire (unverified [63.105.18.201]) by pdxpo.dsl-only.net
 (Rockliffe SMTPRA 4.5.4) with SMTP id <B0009247400@pdxpo.dsl-only.net>;
 Fri, 18 Jan 2002 08:32:47 -0800
From: "Bill Strahm" <bill@strahm.net>
To: <yaronh@voltaire.com>, <ipoverib@ietf.org>,
        "'Libor Michalek'" <libor@pobox.com>
Subject: RE: [ipoverib] reasons for not using LID ?
Date: Fri, 18 Jan 2002 08:42:10 -0800
Message-ID: <HHEDIJGOEFLPCPNILMPAOECLCFAA.bill@strahm.net>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0)
In-reply-to: <001501c1a01f$161156f0$61ac003e@vaio505he>
X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
Importance: Normal
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Question for you Yaron,

For you ARP Proxy (on the IB Router)...

How will you detect ARP packets, given that the ARP response is Unicast back
to the sender.  The response can be on any QPN going to any destination.  I
would be scared of using pattern matching because then it starts to
potentially interfere with other protocols that are sending random binary
data, and also that would require a filter on every packet coming across the
link (at 12x rates even)

This is a hard problem that can easily be solved by a simple GID->LID lookup
at the SM.  Either that or limit IP subnets to a single IB subnet and do IP
routing to cross IB subnets, then the IB router just has to handle IP
routing which is a simple problem especially in a static environment.

Bill

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Yaron Haviv
Sent: Friday, January 18, 2002 4:53 AM
To: 'Bill Strahm'; ipoverib@ietf.org; 'Libor Michalek'
Subject: RE: [ipoverib] reasons for not using LID ?


Bill,

First when forwarding packets from one subnet to another, the LRH header
needs to be reconstructed
This involved replacing both the DLID & SLID in the new header, so we do
deal with the packet
And we do have the LID'd in the packet LRH header anyway
And we don't keep the same LID's in a way that the destination will get a
LID which belongs to a different subnet

You assume that the IB routers works like a layer 2 Ethernet switch
Unfortunately this is not the case, since as mike noted I couldn't reveal
the content of the LWG discussions I cannot go into it
Lets just say as one of the people active in the router sub-group that it is
totally different and a lot of things still needs to be determined
I would choose at least initially to implement something like Proxy ARP to
deal with the Issues
One difference I could point out is that the Ethernet spanning tree is a
self-learning mechanism where as IB routing is not
The Proxy ARP mechanism compensates for that fact

I'm trying to stay away from the theory and see how today I would implement
those things the most efficient way
When digging into it and going over all the potential problems, I would
personally like to avoid the SM and especially when going to a different
subnet
And not just because of the few extra milliseconds delay, and SM overhead,
but because I know it could work in all the scenarios

One thing worth digging into is how to enable avoiding the SM using the SLID
and other extra information that will enable detecting when that LID is
invalid, plus maybe the Valid LID, this way if/when IB routing will be
different it might answer your desires too

Yaron

 From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Bill Strahm
Sent: Friday, January 18, 2002 6:29 AM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

I do not agree with your analysis in your first answer.  Yes the IB router
would receive the packet on its first interface and forward it on its second
interface replacing the DLID with its own DLID.  This is what a layer 3
router does.

Now look at the Infiniband fabric as a layer 2 for the IP layer 3.  At this
point you want layer 2 semantics.  At this point IB routers look like layer
2 switches and MUST NOT interfere with the layer 3 packets on top (otherwise
it is a layer 3 router, and it might as well do IP routing and we don't have
the subnet spanning problem at all)

The job of an IB router in an infiniband fabric is much simpler if we do not
have LID in the ARP packet at all, and just have the destination node poll
the SM for the LID based on the GID.

Bill

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Yaron Haviv
Sent: Thursday, January 17, 2002 6:38 PM
To: 'Bill Strahm'; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?


Bill, the answer to your first question relays in your second one

When IP spans IB, the router receive a packet with the local LID of the
first subnet (even when it is just in the S/DLID) and needs to modify it to
a packet with a local LID of the target subnet
Otherwise it won't get to the target, or returned to the sender
I would be glad to know what is your trick for avoid that
To avoid loops the router will need to implement a proxy ARP, which
maintains the virtualized routing tree
Looking at it like a layer 2 switch is to much of a theoretical abstraction

Answering the 2nd Q, by now you see that the ARP been sent to the target has
the router LID, and when you will reply it will get just to the right place
The end node operation doesn't change in any case

Yaron

-----Original Message-----
From: Bill Strahm [mailto:bill@strahm.net]
Sent: Friday, January 18, 2002 4:10 AM
To: yaronh@voltaire.com; 'Arvind Murching'; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

I am very confused, why will an IB router have to modify an IP ARP packet at
all.  If there is something in the spec that REQUIRES that I think I would
be in favor of ripping it out and fixing this...

Consider IB routers as looking like Ethernet switches, it is just a layer 2
device that sends packets to the correct collision domain.  I see that as
what an IB router is for the IP layer on top of it.  You would have a hard
time convincing me that you want the router sniffing and modifing ARP
packets.  This would be a huge HUGE layer violation.

Now here is your arguement for not including LIDs in the ARP response, I
want a single ARP response for everyone that sends me an ARP request in my
IP subnet.  If this IP subnet crosses an IB subnet, this LID will not be
valid (or the router has to munge it for me)  This seems a lot of work to
get past the case of a simple lookup to the SM that the IBTA Architecture
would have you do when you are communicating with a new endpoint.

Bill

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Yaron Haviv
Sent: Thursday, January 17, 2002 5:46 PM
To: 'Arvind Murching'; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?


Arvind,

The only real argument you make is implementation in Routers
Since we build such boxes, I could tell you dealing with an ARP in the
router in case of IP span IB is complex anyway
Believe me, the simplification you are offering me creates more work than
help inside the router
As you also stated in the passed there are also very little amount of ARP
packets, so replacing it in one or two paces doesn't change anything

Yaron

-----Original Message-----
From: Arvind Murching [mailto:arvindm@windows.microsoft.com]
Sent: Friday, January 18, 2002 3:34 AM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

Below...

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 5:31 PM
To: Arvind Murching; yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

My remarks attached

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of
Arvind Murching
Sent: Friday, January 18, 2002 2:57 AM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

The argument should be the other way around, i.e. fields need to justify
their necessity to be part of a protocol.
[YH] I think many arguments where already raised, now it's the
opposition
turn to put their arguments
[Arvind] The need for a redundant field is not obvious.

Some reasons why not:

- LID values cannot be used across IB subnet boundaries
[YH] you would any way need to store both LID (Router) and GID (node) in
this case somewhere, you wouldn't go to the SM for any new IP packet
wont
you
the routers in this case will any way need to change the LID in the ARP
packets (Proxy ARP and all that)
I believe this case is the one that benefits most from the LID, avoiding
the
long and complex multi-subnet resolution process
[Arvind] Consider the most general case of an IB router routing between
IB subnets. This router will not modify an ARP reply packet to insert
its LID into the payload. However, it will insert its SLID into the IB
header of packets that it forwards. Proxy ARP boxes can do the same,
too.

- A LID cannot be used as part of a static ARP entry
[YH] We didn't say not to have GID, you could still put GID as a static
entry

- The SLID is already present in case someone needs it
[YH] also regular ARP has it, so why not follow that path
[Arvind] This is not an argument for adding redundant fields.


My 2c
- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 4:47 PM
To: ipoverib@ietf.org
Subject: [ipoverib] reasons for not using LID ?

By now it is obvious that there are the ones that wants LID's in the ARP
packets

Could the ones that don't give their reasons for why not to
Not just things like "you don't need it", "we want to save 16-bit", etc'
But, reasons for what problems could arouse from having it in the packet
I haven't heard any real argument that would make me consider removing
it
until now
we then will review them and see if it is possible to deal with those
cases,
if not I personally would drop my support in LID's

If people want it for their reasons, the ones that don't also need to do
some explaining

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Fri Jan 18 13:16:43 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA21802
	for <ipoverib-archive@odin.ietf.org>; Fri, 18 Jan 2002 13:16:41 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id NAA27374
	for ipoverib-archive@odin.ietf.org; Fri, 18 Jan 2002 13:16:44 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA27035;
	Fri, 18 Jan 2002 13:02:51 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA27006
	for <ipoverib@optimus.ietf.org>; Fri, 18 Jan 2002 13:02:50 -0500 (EST)
Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA21220
	for <ipoverib@ietf.org>; Fri, 18 Jan 2002 13:02:47 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id LAA79112;
	Fri, 18 Jan 2002 11:58:32 -0600
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0II2l1181512;
	Fri, 18 Jan 2002 13:02:47 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0II2jV16209;
	Fri, 18 Jan 2002 10:02:45 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0II2iI17782;
	Fri, 18 Jan 2002 10:02:44 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201181802.g0II2iI17782@eng4.beaverton.ibm.com>
Subject: Re: [ipoverib] reasons for not using LID ?
To: Paul.Culley@compaq.com (Culley, Paul)
Date: Fri, 18 Jan 2002 10:02:43 -0800 (PST)
Cc: yaronh@voltaire.com, ipoverib@ietf.org
In-Reply-To: <4D027986353D1341ADA4F2A4F8A170760E3175@cceexc18.americas.cpqcorp.net> from "Culley, Paul" at Jan 18, 2002 07:37:12 AM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

> 
> 1) a LID in the body of the of the ARP is insufficient for IB routing,
> you need the LID/SL pair in order to actually get a packet anywhere.  IB
> provides NO WAY to select an SL, based only on the LID, unless the ARP
> requestor already has that LID and an SL cached or is willing to look it
> up with the SA.  Since one of the reasons for including the LID at all
> is to avoid this lookup, the solution is not complete without SL.
> [Please don't respond that some implementation specific mechanism can be
> chosen to select SL; there is no way for an endnode implementation to
> determine the SA policies for SL usage at this time, so it is not
> possible for the endnode to reliably chose an SL without asking the SA
> (or caching a LID/SL derived from the SA).

        SL can be determined based on workload characteristics. It is
        not tied to LID. An implementation is free to go to the SA or
        determine it on such a mapping. The IB spec leaves it upto the
        imlementation to make the choice - and some of us have. 


> 
> 2) LID in the body of the ARP will be invalid if the ARP crosses IB
> subnets.  This has been discussed a lot already, but the following
> observations seem pertinent:
>  a) IB routers will happen someday, it would be nice to not have to
>   go back to this process to fix ARP when it does happen.

        As noted in replies to the above 'reason' there is nothing to
        fix. Still nothing has been specified when this will happen
        and why will it even be needed.

	ARP across IB subnet:

        On receiving an ARP request the receiving node will check the
        GID-prefix. If it is of the remote node it asks the routing
        infrastructure for the LID. If the GID is local it just uses
        it. It is very simple.

>  b) The IB Management group currently intends to support cross
>   subnet operation for IB via use of the same SA PathRecord Lookup
>   mechanism as is used for local operations.  This is primarily so
>   that early endnode implementations can operate without changes.
>   It is possible that other mechanisms may be defined as
>   optimizations for routers, but that is at best an early concept. If
>   we support a GID lookup for the ARP packets (and either drop body
>   LIDs, use IB Header SLIDs, or indicate when to ignore them) ARP and
>   IP can work across IB subnet boundaries.

        Lot of speculation. Out of scope. We need to work with what is
        there and is likely to be present for a long time. A short
        incomplete list of IBTA to do:

		First routers need to be fully defined. 
		Then multicast needs to fleshed out.
        	Partitioning across subnets has to be figured out. 
		SA/SM coordination has to be specified. 
		Routing protocol has to be specified.
		Node to Routing infrastructure interaction has to be specified
		SA to Router interaction


	Then the usefulness of such a setup has to be justified. 

	[ Note that ARP including LID will work very simply as noted above.]



>  c) It is certain that IB routers will modify IB LRH headers;
>   choosing new values for SLID/DLID/SL at subnet boundaries.  And a
>   requirement on IB subnets and routers is that a unicast packet's
>   SLID/SL can be used to return a packet to the source allows using
>   SLID/SL in any case where the source of the ARP is also the place
>   comm packets go.
>  d) It seems a significant violation of layering to require an IB
>   router to understand an IP ARP, just so that it can modify a LID in
>   the body of the packet.

        On receiving an ARP request the node will check the
        GID-prefix. If it is of the remote node it asks the routing
        infrastructure for the LID. If the GID is local it just uses
        it. It is very simple.

> 
> 3) As it happens, it is perfectly legal for an IB fabric to use
> different LID addresses to talk to the same device from different other
> devices, that is one of the reasons that we allow up to 2^LMC addresses
> for each device.  (That's right, there are potentially up to 128 LID
> addresses for each port!)  And not all LID/SL addresses need be
> reachable from all other devices.  This means that taking a LID from the
> ARP Request as a way to communicate, may not work.  And it is not
> currently possible for an endnode to determine in a defined way that the
> SA uses this type of policy, so endnodes have NO architected WAY to tell
> when to ignore the LID and go for GID on their own.

There is no architected way to choose which of the LMC bits to pick.
If multiple paths are registered at the SA a GID lookup will return a
set of them. It is then upto the responder to pick the path it
chooses. which one will it choose ? how? It is a lot better for the
server, which setup the paths for a reason, specifies the one it wants
used. It is *better* to use LIDs in this case.

Include both GID and LID. It allows all implementations to work.

Vivek
-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Fri Jan 18 13:36:32 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA22584
	for <ipoverib-archive@odin.ietf.org>; Fri, 18 Jan 2002 13:36:27 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id NAA28535
	for ipoverib-archive@odin.ietf.org; Fri, 18 Jan 2002 13:36:30 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA27711;
	Fri, 18 Jan 2002 13:22:29 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA27681
	for <ipoverib@optimus.ietf.org>; Fri, 18 Jan 2002 13:22:28 -0500 (EST)
Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA22082
	for <ipoverib@ietf.org>; Fri, 18 Jan 2002 13:22:25 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id MAA33576;
	Fri, 18 Jan 2002 12:18:09 -0600
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0IIMM1150892;
	Fri, 18 Jan 2002 13:22:22 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0IIMJV16843;
	Fri, 18 Jan 2002 10:22:19 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0IIMIf19680;
	Fri, 18 Jan 2002 10:22:18 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201181822.g0IIMIf19680@eng4.beaverton.ibm.com>
Subject: Re: [ipoverib] reasons for not using LID ?
To: bill@strahm.net (Bill Strahm)
Date: Fri, 18 Jan 2002 10:22:17 -0800 (PST)
Cc: yaronh@voltaire.com, ipoverib@ietf.org, libor@pobox.com ('Libor Michalek')
In-Reply-To: <HHEDIJGOEFLPCPNILMPAOECLCFAA.bill@strahm.net> from "Bill Strahm" at Jan 18, 2002 07:42:10 AM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Bill, Yaron,

	There are two situations 
		- proxy-ARP for a node on another subnet
		- span IP subnets across IB subnets

1. 	proxy-ARP for a node on another subnet works very easily. 
        You return the LID of the local IB router and the GID of the
        remote IB node. Note this is possible with the current
        specification. If only the GID is included excluding the LID
        this is not possible since you can't lookup the remote node's
        GID with the SA. You need to include both. [ SA to SA
        coordination within an IB subnet isn't specified what of
        across the subnet. ]

        Note that the ARP packet is fielded by the IPoIB device
        logically/physically associated with the IB router. The ARP
        packet itself does not travel beyond the IB subnet.


2. 	For IP subnets to span IB subnets there is too much 
        speculation and no justification. First of all we don't know
        why we need it. We don't know when it will be possible. We
        don't know if it will be possible. It is a complicated issue -
        too many things have to be updated in IB architecutre.

        Even for argument's sake lets take it that it was possible for
        IP subnets to span IB subnets. The solution is very simple
        when LID is included. And it is compatible with the solution
        in 1. above.

        The receiving node's ARP module looks at the GID-prefix. If
        the prefix is of another subnet then the routing
        infrastructure is looked up and the router's LID is recorded
        otherwise the LID in the packet is recorded.


I find it odd that we are discussing 'IP subnets spanning IB subnets'
but not proxy-ARP that can work, happen and maybe is all that will
ever be needed or possible(if IBTA doesn't solve routing/cross subnet
issues). 


Vivek

	
> 
> Question for you Yaron,
> 
> For you ARP Proxy (on the IB Router)...
> 
> How will you detect ARP packets, given that the ARP response is Unicast back
> to the sender.  The response can be on any QPN going to any destination.  I
> would be scared of using pattern matching because then it starts to
> potentially interfere with other protocols that are sending random binary
> data, and also that would require a filter on every packet coming across the
> link (at 12x rates even)
> 
> This is a hard problem that can easily be solved by a simple GID->LID lookup
> at the SM.  Either that or limit IP subnets to a single IB subnet and do IP
> routing to cross IB subnets, then the IB router just has to handle IP
> routing which is a simple problem especially in a static environment.
> 
> Bill
> 
> -----Original Message-----
> From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
> Of Yaron Haviv
> Sent: Friday, January 18, 2002 4:53 AM
> To: 'Bill Strahm'; ipoverib@ietf.org; 'Libor Michalek'
> Subject: RE: [ipoverib] reasons for not using LID ?
> 
> 
> Bill,
> 
> First when forwarding packets from one subnet to another, the LRH header
> needs to be reconstructed
> This involved replacing both the DLID & SLID in the new header, so we do
> deal with the packet
> And we do have the LID'd in the packet LRH header anyway
> And we don't keep the same LID's in a way that the destination will get a
> LID which belongs to a different subnet
> 
> You assume that the IB routers works like a layer 2 Ethernet switch
> Unfortunately this is not the case, since as mike noted I couldn't reveal
> the content of the LWG discussions I cannot go into it
> Lets just say as one of the people active in the router sub-group that it is
> totally different and a lot of things still needs to be determined
> I would choose at least initially to implement something like Proxy ARP to
> deal with the Issues
> One difference I could point out is that the Ethernet spanning tree is a
> self-learning mechanism where as IB routing is not
> The Proxy ARP mechanism compensates for that fact
> 
> I'm trying to stay away from the theory and see how today I would implement
> those things the most efficient way
> When digging into it and going over all the potential problems, I would
> personally like to avoid the SM and especially when going to a different
> subnet
> And not just because of the few extra milliseconds delay, and SM overhead,
> but because I know it could work in all the scenarios
> 
> One thing worth digging into is how to enable avoiding the SM using the SLID
> and other extra information that will enable detecting when that LID is
> invalid, plus maybe the Valid LID, this way if/when IB routing will be
> different it might answer your desires too
> 
> Yaron
> 
>  From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
> Bill Strahm
> Sent: Friday, January 18, 2002 6:29 AM
> To: yaronh@voltaire.com; ipoverib@ietf.org
> Subject: RE: [ipoverib] reasons for not using LID ?
> 
> I do not agree with your analysis in your first answer.  Yes the IB router
> would receive the packet on its first interface and forward it on its second
> interface replacing the DLID with its own DLID.  This is what a layer 3
> router does.
> 
> Now look at the Infiniband fabric as a layer 2 for the IP layer 3.  At this
> point you want layer 2 semantics.  At this point IB routers look like layer
> 2 switches and MUST NOT interfere with the layer 3 packets on top (otherwise
> it is a layer 3 router, and it might as well do IP routing and we don't have
> the subnet spanning problem at all)
> 
> The job of an IB router in an infiniband fabric is much simpler if we do not
> have LID in the ARP packet at all, and just have the destination node poll
> the SM for the LID based on the GID.
> 
> Bill
> 
> -----Original Message-----
> From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
> Of Yaron Haviv
> Sent: Thursday, January 17, 2002 6:38 PM
> To: 'Bill Strahm'; ipoverib@ietf.org
> Subject: RE: [ipoverib] reasons for not using LID ?
> 
> 
> Bill, the answer to your first question relays in your second one
> 
> When IP spans IB, the router receive a packet with the local LID of the
> first subnet (even when it is just in the S/DLID) and needs to modify it to
> a packet with a local LID of the target subnet
> Otherwise it won't get to the target, or returned to the sender
> I would be glad to know what is your trick for avoid that
> To avoid loops the router will need to implement a proxy ARP, which
> maintains the virtualized routing tree
> Looking at it like a layer 2 switch is to much of a theoretical abstraction
> 
> Answering the 2nd Q, by now you see that the ARP been sent to the target has
> the router LID, and when you will reply it will get just to the right place
> The end node operation doesn't change in any case
> 
> Yaron
> 
> -----Original Message-----
> From: Bill Strahm [mailto:bill@strahm.net]
> Sent: Friday, January 18, 2002 4:10 AM
> To: yaronh@voltaire.com; 'Arvind Murching'; ipoverib@ietf.org
> Subject: RE: [ipoverib] reasons for not using LID ?
> 
> I am very confused, why will an IB router have to modify an IP ARP packet at
> all.  If there is something in the spec that REQUIRES that I think I would
> be in favor of ripping it out and fixing this...
> 
> Consider IB routers as looking like Ethernet switches, it is just a layer 2
> device that sends packets to the correct collision domain.  I see that as
> what an IB router is for the IP layer on top of it.  You would have a hard
> time convincing me that you want the router sniffing and modifing ARP
> packets.  This would be a huge HUGE layer violation.
> 
> Now here is your arguement for not including LIDs in the ARP response, I
> want a single ARP response for everyone that sends me an ARP request in my
> IP subnet.  If this IP subnet crosses an IB subnet, this LID will not be
> valid (or the router has to munge it for me)  This seems a lot of work to
> get past the case of a simple lookup to the SM that the IBTA Architecture
> would have you do when you are communicating with a new endpoint.
> 
> Bill
> 
> -----Original Message-----
> From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
> Of Yaron Haviv
> Sent: Thursday, January 17, 2002 5:46 PM
> To: 'Arvind Murching'; ipoverib@ietf.org
> Subject: RE: [ipoverib] reasons for not using LID ?
> 
> 
> Arvind,
> 
> The only real argument you make is implementation in Routers
> Since we build such boxes, I could tell you dealing with an ARP in the
> router in case of IP span IB is complex anyway
> Believe me, the simplification you are offering me creates more work than
> help inside the router
> As you also stated in the passed there are also very little amount of ARP
> packets, so replacing it in one or two paces doesn't change anything
> 
> Yaron
> 
> -----Original Message-----
> From: Arvind Murching [mailto:arvindm@windows.microsoft.com]
> Sent: Friday, January 18, 2002 3:34 AM
> To: yaronh@voltaire.com; ipoverib@ietf.org
> Subject: RE: [ipoverib] reasons for not using LID ?
> 
> Below...
> 
> -----Original Message-----
> From: Yaron Haviv [mailto:yaronh@voltaire.com]
> Sent: Thursday, January 17, 2002 5:31 PM
> To: Arvind Murching; yaronh@voltaire.com; ipoverib@ietf.org
> Subject: RE: [ipoverib] reasons for not using LID ?
> 
> My remarks attached
> 
> -----Original Message-----
> From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
> Of
> Arvind Murching
> Sent: Friday, January 18, 2002 2:57 AM
> To: yaronh@voltaire.com; ipoverib@ietf.org
> Subject: RE: [ipoverib] reasons for not using LID ?
> 
> The argument should be the other way around, i.e. fields need to justify
> their necessity to be part of a protocol.
> [YH] I think many arguments where already raised, now it's the
> opposition
> turn to put their arguments
> [Arvind] The need for a redundant field is not obvious.
> 
> Some reasons why not:
> 
> - LID values cannot be used across IB subnet boundaries
> [YH] you would any way need to store both LID (Router) and GID (node) in
> this case somewhere, you wouldn't go to the SM for any new IP packet
> wont
> you
> the routers in this case will any way need to change the LID in the ARP
> packets (Proxy ARP and all that)
> I believe this case is the one that benefits most from the LID, avoiding
> the
> long and complex multi-subnet resolution process
> [Arvind] Consider the most general case of an IB router routing between
> IB subnets. This router will not modify an ARP reply packet to insert
> its LID into the payload. However, it will insert its SLID into the IB
> header of packets that it forwards. Proxy ARP boxes can do the same,
> too.
> 
> - A LID cannot be used as part of a static ARP entry
> [YH] We didn't say not to have GID, you could still put GID as a static
> entry
> 
> - The SLID is already present in case someone needs it
> [YH] also regular ARP has it, so why not follow that path
> [Arvind] This is not an argument for adding redundant fields.
> 
> 
> My 2c
> - Arvind
> 
> -----Original Message-----
> From: Yaron Haviv [mailto:yaronh@voltaire.com]
> Sent: Thursday, January 17, 2002 4:47 PM
> To: ipoverib@ietf.org
> Subject: [ipoverib] reasons for not using LID ?
> 
> By now it is obvious that there are the ones that wants LID's in the ARP
> packets
> 
> Could the ones that don't give their reasons for why not to
> Not just things like "you don't need it", "we want to save 16-bit", etc'
> But, reasons for what problems could arouse from having it in the packet
> I haven't heard any real argument that would make me consider removing
> it
> until now
> we then will review them and see if it is possible to deal with those
> cases,
> if not I personally would drop my support in LID's
> 
> If people want it for their reasons, the ones that don't also need to do
> some explaining
> 
> Yaron
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Fri Jan 18 14:24:54 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA24141
	for <ipoverib-archive@odin.ietf.org>; Fri, 18 Jan 2002 14:24:54 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id OAA00329
	for ipoverib-archive@odin.ietf.org; Fri, 18 Jan 2002 14:24:57 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA29868;
	Fri, 18 Jan 2002 14:05:38 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA29833
	for <ipoverib@optimus.ietf.org>; Fri, 18 Jan 2002 14:05:36 -0500 (EST)
Received: from omegaband.com (faust.seagullsemi.com [206.196.68.34])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA23759
	for <ipoverib@ietf.org>; Fri, 18 Jan 2002 14:05:32 -0500 (EST)
Received: from seahorsepc [207.8.52.226] by omegaband.com with ESMTP
  (SMTPD32-7.04) id A05E6A6B014C; Fri, 18 Jan 2002 12:58:38 -0600
From: "Ashwin Kamath" <kamath@omegaband.com>
To: <ipoverib@ietf.org>
Date: Fri, 18 Jan 2002 13:03:45 -0600
Message-ID: <IFEKJBDLPFOEGGPIIGMFGEFACAAA.kamath@omegaband.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0)
Importance: Normal
In-Reply-To: <OF27274595.0E6E736F-ON88256B45.001EC98C@boulder.ibm.com>
X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400
Content-Transfer-Encoding: 7bit
Subject: [Ipoverib] Interoperability
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

 There are two major implementation camps, with SA and without SA. Is it the
goal of the architecture to be complete enough to deal with interoperability
between implementations? Is interop supposed to be gauranteed if two
distinct implementations adhere to the specs?
 In my opinion, it must be complete enough that two distinct
implementations, must interoperate if they adhere to IpoIB spec. Comments...

-Ashwin.


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 18 14:51:57 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA24924
	for <ipoverib-archive@odin.ietf.org>; Fri, 18 Jan 2002 14:51:57 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id OAA01401
	for ipoverib-archive@odin.ietf.org; Fri, 18 Jan 2002 14:52:00 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA00871;
	Fri, 18 Jan 2002 14:31:12 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA00839
	for <ipoverib@optimus.ietf.org>; Fri, 18 Jan 2002 14:31:07 -0500 (EST)
Received: from omegaband.com (faust.seagullsemi.com [206.196.68.34])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA24286
	for <ipoverib@ietf.org>; Fri, 18 Jan 2002 14:31:03 -0500 (EST)
Received: from seahorsepc [207.8.52.226] by omegaband.com with ESMTP
  (SMTPD32-7.04) id A65A218E0146; Fri, 18 Jan 2002 13:24:10 -0600
From: "Ashwin Kamath" <kamath@omegaband.com>
To: <ipoverib@ietf.org>
Subject: RE: [Ipoverib] Interoperability
Date: Fri, 18 Jan 2002 13:29:17 -0600
Message-ID: <IFEKJBDLPFOEGGPIIGMFCEFBCAAA.kamath@omegaband.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0)
Importance: Normal
In-Reply-To: <004101c1a055$59cce8b0$a40f020a@Fishhead>
X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Roger,
 Yes, my phrasing is incorrect. Without the SA did not mean that the SA did
not exist in the subnet, or did not have a part to play in the definition of
addresses. So is the interop question a moot question?

-Ashwin.

-----Original Message-----
From: Roger Ronald [mailto:rronald@VIEO.com]
Sent: Friday, January 18, 2002 1:22 PM
To: Ashwin Kamath
Subject: Re: [Ipoverib] Interoperability


I wouldn't agree with your phrasing.

The two camps are
1) Those who believe that the SA should be consulted directly based
on a single GID passed in the ARP request; and
2) Those who believe that other information may be directly passed
from the ARP requestee to the requestor.

In both cases, there is no doubt that the SA was involved in originally
defining the addresses that will be used. It's merely a question about
the wisdom of allowing an intermediary to resend that information.

RR

----- Original Message -----
From: "Ashwin Kamath" <kamath@omegaband.com>
To: <ipoverib@ietf.org>
Sent: Friday, January 18, 2002 1:03 PM
Subject: [Ipoverib] Interoperability


> There are two major implementation camps, with SA and without SA. Is it
the
> goal of the architecture to be complete enough to deal with
interoperability
> between implementations? Is interop supposed to be gauranteed if two
> distinct implementations adhere to the specs?
>  In my opinion, it must be complete enough that two distinct
> implementations, must interoperate if they adhere to IpoIB spec.
Comments...
>
> -Ashwin.
>
>
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
>
>




_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 18 15:14:07 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA25582
	for <ipoverib-archive@odin.ietf.org>; Fri, 18 Jan 2002 15:14:02 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id PAA02540
	for ipoverib-archive@odin.ietf.org; Fri, 18 Jan 2002 15:14:06 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA01672;
	Fri, 18 Jan 2002 14:55:01 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA01646
	for <ipoverib@ns.ietf.org>; Fri, 18 Jan 2002 14:54:59 -0500 (EST)
Received: from ztxmail03.ztx.compaq.com (ztxmail03.ztx.compaq.com [161.114.1.207])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA25025
	for <ipoverib@ietf.org>; Fri, 18 Jan 2002 14:54:55 -0500 (EST)
Received: by ztxmail03.ztx.compaq.com (Postfix, from userid 12345)
	id A8C992EED; Fri, 18 Jan 2002 13:54:23 -0600 (CST)
Received: from cceexg13.americas.cpqcorp.net (cceexg13.americas.cpqcorp.net [16.110.250.119])
	by ztxmail03.ztx.compaq.com (Postfix) with ESMTP
	id 9F7AC1724; Fri, 18 Jan 2002 13:54:23 -0600 (CST)
Received: from cceexc18.americas.cpqcorp.net ([16.110.250.64]) by cceexg13.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966);
	 Fri, 18 Jan 2002 13:54:23 -0600
X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="US-ASCII"
Subject: RE: [Ipoverib] Interoperability
Date: Fri, 18 Jan 2002 13:54:22 -0600
Message-ID: <4D027986353D1341ADA4F2A4F8A170760E3176@cceexc18.americas.cpqcorp.net>
Thread-Topic: [Ipoverib] Interoperability
Thread-Index: AcGgUyVfPFifld3dR7+IDfM4TsgZ7gAAheRA
From: "Culley, Paul" <Paul.Culley@compaq.com>
To: "Ashwin Kamath" <kamath@omegaband.com>, <ipoverib@ietf.org>
X-OriginalArrivalTime: 18 Jan 2002 19:54:23.0494 (UTC) FILETIME=[ED2FA260:01C1A059]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id OAA01647
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

These are key questions, and a source of some of the confusion on this
reflector.  One or two members seem to think that an endnode can choose
LID and SL values in some fashion without going to the SA (or having the
information derived from the SA).  While this can certainly be made to
work, if a vendor provides both the SA and the endnodes, it will not
work across every possible implementation of the SM/SA.

The only interface architected by IB that is guaranteed to return paths
suitable to get the data across, is the PathRecord query.  Between that
and requirements on SM that any given SLID/SL/DLID set work in both
directions, you have just about got all the useful defined mechanisms
that an endnode can use.

As has been pointed out, any device can give any single other device the
information it has on a path between them, but that information must
include all three elements SLID/SL/DLID, one way or the other.  Changing
any one element risks that the packets will be dropped either going or
returning with the inverted SLID/DLID.

As it stands now, using an SGID and DGID, with TClass=0 and FlowLabel=0
will also provide an interoperable query for PathRecord within the
subnet.

It is the goal of most of the IB members to make IB routing work with
the same interfaces, and the same way, although that work is far from
done.  If we pull this off, then endnodes designed before IB routers
become available will still work across subnets.  To this end I am
warning about practices that won't work, like getting LIDs from the body
of the packet, where they are not valid if they cross an IB subnet
boundary, in the hope that IPoIB stuff will work when IB routers become
available.

I certainly hope it is the goal of the IPoIB group to enable
interoperability between endnode devices (supporting IPoIB) and SAs, IB
routers etc. that don't know about IP.  This means adding no new
constraints on the IB infrastructure, only to the endnodes or new IP
routers attached to the IB fabric.

Paul R. Culley
Compaq Sr. Fellow
281-514-5543


-----Original Message-----
From: Ashwin Kamath [mailto:kamath@omegaband.com]
Sent: Friday, January 18, 2002 1:04 PM
To: ipoverib@ietf.org
Subject: [Ipoverib] Interoperability


 There are two major implementation camps, with SA and without SA. Is it
the
goal of the architecture to be complete enough to deal with
interoperability
between implementations? Is interop supposed to be gauranteed if two
distinct implementations adhere to the specs?
 In my opinion, it must be complete enough that two distinct
implementations, must interoperate if they adhere to IpoIB spec.
Comments...

-Ashwin.


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 18 15:37:23 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA26153
	for <ipoverib-archive@odin.ietf.org>; Fri, 18 Jan 2002 15:37:19 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id PAA03660
	for ipoverib-archive@odin.ietf.org; Fri, 18 Jan 2002 15:37:20 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA02609;
	Fri, 18 Jan 2002 15:15:28 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA02579
	for <ipoverib@ns.ietf.org>; Fri, 18 Jan 2002 15:15:26 -0500 (EST)
Received: from vieo.com (vieo.com [216.30.79.131])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA25606
	for <ipoverib@ietf.org>; Fri, 18 Jan 2002 15:15:21 -0500 (EST)
Received: (from root@localhost)
	by vieo.com (8.11.2/8.11.2) id g0IKErW23063
	for ipoverib@ietf.org; Fri, 18 Jan 2002 14:14:53 -0600 (CST)
	(envelope-from rronald@VIEO.com)
Received: from Fishhead (dhcp-10-2-15-164.vieo.com [10.2.15.164])
	by vieo.com (8.11.2/8.11.2) with SMTP id g0IKEpc22981;
	Fri, 18 Jan 2002 14:14:51 -0600 (CST)
	(envelope-from rronald@VIEO.com)
Message-ID: <007b01c1a05c$c050f110$a40f020a@Fishhead>
From: "Roger Ronald" <rronald@VIEO.com>
To: "Ashwin Kamath" <kamath@omegaband.com>, <ipoverib@ietf.org>
References: <IFEKJBDLPFOEGGPIIGMFCEFBCAAA.kamath@omegaband.com>
Subject: Re: [Ipoverib] Interoperability
Date: Fri, 18 Jan 2002 14:14:36 -0600
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 5.50.4133.2400
X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400
X-scanner: scanned by Inflex 0.1.5c+ on vieo.com
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

No, your question is a very important one. 

My opinion is that if the protocol allows two different 
processes for address resolution, they should both
interoperate if the protocol is to be judged successful.

Just to speculate, it may be that the two methods quickly converge
and work the same....or the two methods may have circumstances
where rules must be added to prevent different answers.
There will be quite a bit of analysis needed to answer the question,
but we obviously have an interested and vocal community to do 
the work. :-)

RR
----- Original Message ----- 
From: "Ashwin Kamath" <kamath@omegaband.com>
To: <ipoverib@ietf.org>
Sent: Friday, January 18, 2002 1:29 PM
Subject: RE: [Ipoverib] Interoperability


> Roger,
>  Yes, my phrasing is incorrect. Without the SA did not mean that the SA did
> not exist in the subnet, or did not have a part to play in the definition of
> addresses. So is the interop question a moot question?
> 
> -Ashwin.
> 
> -----Original Message-----
> From: Roger Ronald [mailto:rronald@VIEO.com]
> Sent: Friday, January 18, 2002 1:22 PM
> To: Ashwin Kamath
> Subject: Re: [Ipoverib] Interoperability
> 
> 
> I wouldn't agree with your phrasing.
> 
> The two camps are
> 1) Those who believe that the SA should be consulted directly based
> on a single GID passed in the ARP request; and
> 2) Those who believe that other information may be directly passed
> from the ARP requestee to the requestor.
> 
> In both cases, there is no doubt that the SA was involved in originally
> defining the addresses that will be used. It's merely a question about
> the wisdom of allowing an intermediary to resend that information.
> 
> RR
> 
> ----- Original Message -----
> From: "Ashwin Kamath" <kamath@omegaband.com>
> To: <ipoverib@ietf.org>
> Sent: Friday, January 18, 2002 1:03 PM
> Subject: [Ipoverib] Interoperability
> 
> 
> > There are two major implementation camps, with SA and without SA. Is it
> the
> > goal of the architecture to be complete enough to deal with
> interoperability
> > between implementations? Is interop supposed to be gauranteed if two
> > distinct implementations adhere to the specs?
> >  In my opinion, it must be complete enough that two distinct
> > implementations, must interoperate if they adhere to IpoIB spec.
> Comments...
> >
> > -Ashwin.
> >
> >
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > https://www1.ietf.org/mailman/listinfo/ipoverib
> >
> >
> 
> 
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 
> 


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 18 17:05:14 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA28764
	for <ipoverib-archive@odin.ietf.org>; Fri, 18 Jan 2002 17:05:14 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id RAA07272
	for ipoverib-archive@odin.ietf.org; Fri, 18 Jan 2002 17:05:16 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA06319;
	Fri, 18 Jan 2002 16:53:31 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA06283
	for <ipoverib@ns.ietf.org>; Fri, 18 Jan 2002 16:53:28 -0500 (EST)
Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA28546
	for <ipoverib@ietf.org>; Fri, 18 Jan 2002 16:53:25 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e31.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id QAA86144;
	Fri, 18 Jan 2002 16:50:24 -0500
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0ILrFD92674;
	Fri, 18 Jan 2002 14:53:15 -0700
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0ILrAV23396;
	Fri, 18 Jan 2002 13:53:10 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0ILr8H06022;
	Fri, 18 Jan 2002 13:53:08 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201182153.g0ILr8H06022@eng4.beaverton.ibm.com>
Subject: Re: [ipoverib] reasons for not using LID ?
To: yaronh@voltaire.com
Date: Fri, 18 Jan 2002 13:53:07 -0800 (PST)
Cc: Paul.Culley@COMPAQ.com ('Culley Paul'), ipoverib@ietf.org,
        bill@strahm.net ('Bill Strahm'), kashyapv@us.ibm.com ('Vivek Kashyap')
In-Reply-To: <002001c1a065$d34351b0$61ac003e@vaio505he> from "Yaron Haviv" at Jan 18, 2002 10:19:28 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Yaron,

	personal mail. 

        what? not the clean approach ? how ? I thought I presented how
        the inclusion of LID is cleaner.

Vivek

Vivek

> 
> This is a multi-part message in MIME format.
> 
> --Boundary_(ID_FXr1pBXMduEyHZhYqhe+Zw)
> Content-type: text/plain; charset=windows-1255
> Content-Transfer-Encoding: 7BIT
> 
> Bill, Paul
> 
> I would like to conclude that discussion with out adding any counter
> arguments
> I agree that going the path of LID in the payload is not the clean approach,
> as both Paul & Bill presented 
> My understanding is that It could be made to work and solve other
> multi-subnet issues that exists today
> 
> I'm personally willing to go for Paul's suggestion and use SLID, plus add a
> way to indicate when to ignore it 
> I hope that Vivek would agree to it too, and we could go to the next topic
> on the agenda 
> 
> Paul, we need to see what we need to do in MSO to improve the multi-subnet
> path discovery mechanism 
> 
> Yaron
> 
> 
> --Boundary_(ID_FXr1pBXMduEyHZhYqhe+Zw)
> Content-type: application/ms-tnef; name=winmail.dat
> Content-disposition: attachment; filename=winmail.dat
> Content-Transfer-Encoding: base64
> 
> eJ8+IiEVAQaQCAAEAAAAAAABAAEAAQeQBgAIAAAA5wQAAAAAAADrAAEIgAcAGAAAAElQTS5NaWNy
> b3NvZnQgTWFpbC5Ob3RlADEIAQ2ABAACAAAAAgACAAEGgAMADgAAANIHAQASABcAEwAAAAUAGwEB
> A5AGAAgIAAAsAAAACwACAAEAAAALACMAAAAAAAMAJgAAAAAACwApAAAAAAADAC4AAAAAAAMANgAA
> AAAAHgBwAAEAAAAnAAAAW2lwb3ZlcmliXSByZWFzb25zIGZvciBub3QgdXNpbmcgTElEID8AAAIB
> cQABAAAAFgAAAAHBoGXP84B+bada8klnrhit06R7hEMAAAIBHQwBAAAAGQAAAFNNVFA6WUFST05I
> QFZPTFRBSVJFLkNPTQAAAAALAAEOAAAAAEAABg4A+gS/ZaDBAQIBCg4BAAAALgAAAAAAAADWDNpo
> g4/REbTSAIBfixTzAQARbIDaX4/REbTSAIBfixTzAAAAAatjAAAAAAMAFA4BAAAAAgEJEAEAAABi
> AgAAXgIAAGYDAABMWkZ1/brFCocACgENA0N0ZXh0Aff/AqQD5AXrAoMAUALzBrQCgyYyA8UCAGNo
> CsBzZWB0MTc3IAcTAoB9/wqACM8J2QKACoQLNxLCAdAEIEIDEGwsIFBhHnUVQAqyCoUKgEkgdwEI
> YGxkIGxpa2WQIHRvIAWgbmMKQC8BABowE3AFQGQEAGN1rQQQaQIgGYBpGwAgCGBJBUBhZBtQbmcc
> gG46eRphdQIwBJAcgHJn3nUHgAIwEzAZFmEJwhr0fGdvHMIbABogCrAcImZgIExJRCALgB/lec8U
> 8ByQIOAEIG5vBUAf8sUaoGUDkWFwcANgANB6aBgwYQQgBuAcIRhSIP4mF+MgIBVgE6AdcRnQGJXu
> TR0gHWAEgXMBkCYgHMK7IeEbA0kFQB1BGcFiGiDPAMAa0hpQGZByaxzxGdC6cwbwdhogI9EdkW0Y
> cIB0aS1zdWJuE7D/IdEqYAeRGwMPAAQAHjAaMeZkIXAYnSdtICAmQQIg/QdAbB0gA/AYEB+zGlAf
> kG4gAhAFwBhSJwQgKmBnfmcHkCowG9EpAhuQGiBT/yCxGDALUBuQHIIcgBmAIXC3GjILgBtQYxsg
> GiB3IAD7IQEyQWciEBVgIOAFQRkWzGhvLXAa9FZpKXAo4M8ZlB7lMkEiMW9vI3EpEf53InEZoy7R
> GkEf8iqQDxD9GjFwMpAcQCEEHuAJ8Cwg/yVmGJUYUhgwN1EqkCVBGkFvE6Ay0hshO1lkMkIF0U/9
> MiNtIxEpcR/yKgsgMxtSvT5Rch0gB4ATYQMAcy1QXRicWQrAAiAYnH1DUAAAHgBCEAEAAABHAAAA
> PDREMDI3OTg2MzUzRDEzNDFBREE0RjJBNEY4QTE3MDc2MEUzMTc1QGNjZWV4YzE4LmFtZXJpY2Fz
> LmNwcWNvcnAubmV0PgAAAwAJWQEAAAACAeJlAQAAABQAAAAyMlrrW6fzS5Zn4CmlyGckAABYyAIB
> 42UBAAAAFQAAABQyMlrrW6fzS5Zn4CmlyGckAABYyAAAAAMAA4AIIAYAAAAAAMAAAAAAAABGAAAA
> AFKFAACOagEAHgAEgAggBgAAAAAAwAAAAAAAAEYAAAAAVIUAAAEAAAAEAAAAOS4wAAsABoAIIAYA
> AAAAAMAAAAAAAABGAAAAAAaFAAAAAAAAAwAHgAggBgAAAAAAwAAAAAAAAEYAAAAAAYUAAAAAAAAL
> AAiACCAGAAAAAADAAAAAAAAARgAAAAADhQAAAAAAAAsACYAIIAYAAAAAAMAAAAAAAABGAAAAAA6F
> AAAAAAAAAwAKgAggBgAAAAAAwAAAAAAAAEYAAAAAEIUAAAAAAAADAAuACCAGAAAAAADAAAAAAAAA
> RgAAAAARhQAAAAAAAAMADIAIIAYAAAAAAMAAAAAAAABGAAAAABiFAAAAAAAAHgANgAggBgAAAAAA
> wAAAAAAAAEYAAAAANoUAAAEAAAABAAAAAAAAAB4ADoAIIAYAAAAAAMAAAAAAAABGAAAAADeFAAAB
> AAAAAQAAAAAAAAAeAA+ACCAGAAAAAADAAAAAAAAARgAAAAA4hQAAAQAAAAEAAAAAAAAACwAfDgEA
> AAACAfgPAQAAABAAAABZyzDRsdPRToXqq96IqtnMAgH6DwEAAAAQAAAA1gzaaIOP0RG00gCAX4sU
> 8wIB+w8BAAAAjAAAAAAAAAA4obsQBeUQGqG7CAArKlbCAABFTVNNREIuRExMAAAAAAAAAAAbVfog
> qmYRzZvIAKoAL8RaDAAAAE5UU0VSVkVSAC9vPVZvbHRhaXJlIEFkdmFuY2VkIERhdGEgU2VjdXJp
> dHkvb3U9Vk9MVEFJUkUvY249UmVjaXBpZW50cy9jbj1ZQVJPTkgAAwD+DwUAAAADAA00/T8AAAIB
> FDQBAAAAEAAAAFSUocApfxAbpYcIACsqJRceAPo/AQAAAAwAAABZYXJvbiBIYXZpdgACAX8AAQAA
> AI0AAAAwMDAwMDAwMEQ2MENEQTY4ODM4RkQxMTFCNEQyMDA4MDVGOEIxNEYzMDcwMDExNkM4MERB
> NUY4RkQxMTFCNEQyMDA4MDVGOEIxNEYzMDAwMDAwMDFBQjYyMDAwMDEyNkM4MERBNUY4RkQxMTFC
> NEQyMDA4MDVGOEIxNEYzMDAwMDBBQzkwQTU1MDAwMAAAAAADAAYQw5fPEgMABxDoAQAAAwAQEAAA
> AAADABEQAAAAAB4ACBABAAAAZQAAAEJJTEwsUEFVTElXT1VMRExJS0VUT0NPTkNMVURFVEhBVERJ
> U0NVU1NJT05XSVRIT1VUQURESU5HQU5ZQ09VTlRFUkFSR1VNRU5UU0lBR1JFRVRIQVRHT0lOR1RI
> RVBBVEhPRkwAAAAAv8o=
> 
> --Boundary_(ID_FXr1pBXMduEyHZhYqhe+Zw)--
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 18 17:36:40 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA29407
	for <ipoverib-archive@odin.ietf.org>; Fri, 18 Jan 2002 17:36:39 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id RAA08076
	for ipoverib-archive@odin.ietf.org; Fri, 18 Jan 2002 17:36:42 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA05113;
	Fri, 18 Jan 2002 16:19:32 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA05058
	for <ipoverib@ns.ietf.org>; Fri, 18 Jan 2002 16:19:26 -0500 (EST)
Received: from mxout2.netvision.net.il (mxout2.netvision.net.il [194.90.9.21])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA27354
	for <ipoverib@ietf.org>; Fri, 18 Jan 2002 16:19:22 -0500 (EST)
Received: from vaio505he ([139.92.208.8]) by mxout2.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GQ500E2FKJFWA@mxout2.netvision.net.il> for ipoverib@ietf.org;
 Fri, 18 Jan 2002 23:18:54 +0200 (IST)
Date: Fri, 18 Jan 2002 23:19:28 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: RE: [ipoverib] reasons for not using LID ?
In-reply-to: 
 <4D027986353D1341ADA4F2A4F8A170760E3175@cceexc18.americas.cpqcorp.net>
To: "'Culley, Paul'" <Paul.Culley@COMPAQ.com>, yaronh@voltaire.com,
        ipoverib@ietf.org, "'Bill Strahm'" <bill@strahm.net>,
        "'Vivek Kashyap'" <kashyapv@us.ibm.com>
Reply-to: yaronh@voltaire.com
Message-id: <002001c1a065$d34351b0$61ac003e@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: multipart/mixed; boundary="Boundary_(ID_FXr1pBXMduEyHZhYqhe+Zw)"
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
X-MS-TNEF-Correlator: 
 00000000D60CDA68838FD111B4D200805F8B14F30700116C80DA5F8FD111B4D200805F8B14F300000001AB620000126C80DA5F8FD111B4D200805F8B14F300000AC90A550000
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

This is a multi-part message in MIME format.

--Boundary_(ID_FXr1pBXMduEyHZhYqhe+Zw)
Content-type: text/plain; charset=windows-1255
Content-Transfer-Encoding: 7BIT

Bill, Paul

I would like to conclude that discussion with out adding any counter
arguments
I agree that going the path of LID in the payload is not the clean approach,
as both Paul & Bill presented 
My understanding is that It could be made to work and solve other
multi-subnet issues that exists today

I'm personally willing to go for Paul's suggestion and use SLID, plus add a
way to indicate when to ignore it 
I hope that Vivek would agree to it too, and we could go to the next topic
on the agenda 

Paul, we need to see what we need to do in MSO to improve the multi-subnet
path discovery mechanism 

Yaron


--Boundary_(ID_FXr1pBXMduEyHZhYqhe+Zw)
Content-type: application/ms-tnef; name=winmail.dat
Content-disposition: attachment; filename=winmail.dat
Content-Transfer-Encoding: base64

eJ8+IiEVAQaQCAAEAAAAAAABAAEAAQeQBgAIAAAA5wQAAAAAAADrAAEIgAcAGAAAAElQTS5NaWNy
b3NvZnQgTWFpbC5Ob3RlADEIAQ2ABAACAAAAAgACAAEGgAMADgAAANIHAQASABcAEwAAAAUAGwEB
A5AGAAgIAAAsAAAACwACAAEAAAALACMAAAAAAAMAJgAAAAAACwApAAAAAAADAC4AAAAAAAMANgAA
AAAAHgBwAAEAAAAnAAAAW2lwb3ZlcmliXSByZWFzb25zIGZvciBub3QgdXNpbmcgTElEID8AAAIB
cQABAAAAFgAAAAHBoGXP84B+bada8klnrhit06R7hEMAAAIBHQwBAAAAGQAAAFNNVFA6WUFST05I
QFZPTFRBSVJFLkNPTQAAAAALAAEOAAAAAEAABg4A+gS/ZaDBAQIBCg4BAAAALgAAAAAAAADWDNpo
g4/REbTSAIBfixTzAQARbIDaX4/REbTSAIBfixTzAAAAAatjAAAAAAMAFA4BAAAAAgEJEAEAAABi
AgAAXgIAAGYDAABMWkZ1/brFCocACgENA0N0ZXh0Aff/AqQD5AXrAoMAUALzBrQCgyYyA8UCAGNo
CsBzZWB0MTc3IAcTAoB9/wqACM8J2QKACoQLNxLCAdAEIEIDEGwsIFBhHnUVQAqyCoUKgEkgdwEI
YGxkIGxpa2WQIHRvIAWgbmMKQC8BABowE3AFQGQEAGN1rQQQaQIgGYBpGwAgCGBJBUBhZBtQbmcc
gG46eRphdQIwBJAcgHJn3nUHgAIwEzAZFmEJwhr0fGdvHMIbABogCrAcImZgIExJRCALgB/lec8U
8ByQIOAEIG5vBUAf8sUaoGUDkWFwcANgANB6aBgwYQQgBuAcIRhSIP4mF+MgIBVgE6AdcRnQGJXu
TR0gHWAEgXMBkCYgHMK7IeEbA0kFQB1BGcFiGiDPAMAa0hpQGZByaxzxGdC6cwbwdhogI9EdkW0Y
cIB0aS1zdWJuE7D/IdEqYAeRGwMPAAQAHjAaMeZkIXAYnSdtICAmQQIg/QdAbB0gA/AYEB+zGlAf
kG4gAhAFwBhSJwQgKmBnfmcHkCowG9EpAhuQGiBT/yCxGDALUBuQHIIcgBmAIXC3GjILgBtQYxsg
GiB3IAD7IQEyQWciEBVgIOAFQRkWzGhvLXAa9FZpKXAo4M8ZlB7lMkEiMW9vI3EpEf53InEZoy7R
GkEf8iqQDxD9GjFwMpAcQCEEHuAJ8Cwg/yVmGJUYUhgwN1EqkCVBGkFvE6Ay0hshO1lkMkIF0U/9
MiNtIxEpcR/yKgsgMxtSvT5Rch0gB4ATYQMAcy1QXRicWQrAAiAYnH1DUAAAHgBCEAEAAABHAAAA
PDREMDI3OTg2MzUzRDEzNDFBREE0RjJBNEY4QTE3MDc2MEUzMTc1QGNjZWV4YzE4LmFtZXJpY2Fz
LmNwcWNvcnAubmV0PgAAAwAJWQEAAAACAeJlAQAAABQAAAAyMlrrW6fzS5Zn4CmlyGckAABYyAIB
42UBAAAAFQAAABQyMlrrW6fzS5Zn4CmlyGckAABYyAAAAAMAA4AIIAYAAAAAAMAAAAAAAABGAAAA
AFKFAACOagEAHgAEgAggBgAAAAAAwAAAAAAAAEYAAAAAVIUAAAEAAAAEAAAAOS4wAAsABoAIIAYA
AAAAAMAAAAAAAABGAAAAAAaFAAAAAAAAAwAHgAggBgAAAAAAwAAAAAAAAEYAAAAAAYUAAAAAAAAL
AAiACCAGAAAAAADAAAAAAAAARgAAAAADhQAAAAAAAAsACYAIIAYAAAAAAMAAAAAAAABGAAAAAA6F
AAAAAAAAAwAKgAggBgAAAAAAwAAAAAAAAEYAAAAAEIUAAAAAAAADAAuACCAGAAAAAADAAAAAAAAA
RgAAAAARhQAAAAAAAAMADIAIIAYAAAAAAMAAAAAAAABGAAAAABiFAAAAAAAAHgANgAggBgAAAAAA
wAAAAAAAAEYAAAAANoUAAAEAAAABAAAAAAAAAB4ADoAIIAYAAAAAAMAAAAAAAABGAAAAADeFAAAB
AAAAAQAAAAAAAAAeAA+ACCAGAAAAAADAAAAAAAAARgAAAAA4hQAAAQAAAAEAAAAAAAAACwAfDgEA
AAACAfgPAQAAABAAAABZyzDRsdPRToXqq96IqtnMAgH6DwEAAAAQAAAA1gzaaIOP0RG00gCAX4sU
8wIB+w8BAAAAjAAAAAAAAAA4obsQBeUQGqG7CAArKlbCAABFTVNNREIuRExMAAAAAAAAAAAbVfog
qmYRzZvIAKoAL8RaDAAAAE5UU0VSVkVSAC9vPVZvbHRhaXJlIEFkdmFuY2VkIERhdGEgU2VjdXJp
dHkvb3U9Vk9MVEFJUkUvY249UmVjaXBpZW50cy9jbj1ZQVJPTkgAAwD+DwUAAAADAA00/T8AAAIB
FDQBAAAAEAAAAFSUocApfxAbpYcIACsqJRceAPo/AQAAAAwAAABZYXJvbiBIYXZpdgACAX8AAQAA
AI0AAAAwMDAwMDAwMEQ2MENEQTY4ODM4RkQxMTFCNEQyMDA4MDVGOEIxNEYzMDcwMDExNkM4MERB
NUY4RkQxMTFCNEQyMDA4MDVGOEIxNEYzMDAwMDAwMDFBQjYyMDAwMDEyNkM4MERBNUY4RkQxMTFC
NEQyMDA4MDVGOEIxNEYzMDAwMDBBQzkwQTU1MDAwMAAAAAADAAYQw5fPEgMABxDoAQAAAwAQEAAA
AAADABEQAAAAAB4ACBABAAAAZQAAAEJJTEwsUEFVTElXT1VMRExJS0VUT0NPTkNMVURFVEhBVERJ
U0NVU1NJT05XSVRIT1VUQURESU5HQU5ZQ09VTlRFUkFSR1VNRU5UU0lBR1JFRVRIQVRHT0lOR1RI
RVBBVEhPRkwAAAAAv8o=

--Boundary_(ID_FXr1pBXMduEyHZhYqhe+Zw)--

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 18 17:36:50 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA29438
	for <ipoverib-archive@odin.ietf.org>; Fri, 18 Jan 2002 17:36:50 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id RAA08090
	for ipoverib-archive@odin.ietf.org; Fri, 18 Jan 2002 17:36:52 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA07585;
	Fri, 18 Jan 2002 17:25:26 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA07553
	for <ipoverib@ns.ietf.org>; Fri, 18 Jan 2002 17:25:24 -0500 (EST)
Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA29225
	for <ipoverib@ietf.org>; Fri, 18 Jan 2002 17:25:21 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id QAA13920;
	Fri, 18 Jan 2002 16:21:00 -0600
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0IMPF043076;
	Fri, 18 Jan 2002 17:25:15 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id g0IMPFg03007;
	Fri, 18 Jan 2002 14:25:15 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0IMPE108726;
	Fri, 18 Jan 2002 14:25:14 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201182225.g0IMPE108726@eng4.beaverton.ibm.com>
Subject: Re: [ipoverib] reasons for not using LID ?
To: kashyapv@us.ibm.com (Vivek Kashyap)
Date: Fri, 18 Jan 2002 14:25:12 -0800 (PST)
Cc: yaronh@voltaire.com, Paul.Culley@COMPAQ.com ('Culley Paul'),
        ipoverib@ietf.org, bill@strahm.net ('Bill Strahm'),
        kashyapv@us.ibm.com ('Vivek Kashyap')
In-Reply-To: <200201182153.g0ILr8H06022@eng4.beaverton.ibm.com> from "Vivek Kashyap" at Jan 18, 2002 12:53:07 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Oops! personal? I ended up sending to everyone :).

The way the router implementation was discussed with the router
looking into the packet was not clean.

But the router is not involved at all. See my earlier mail in this regard.

Consider:

* ARP packet:   LRH+GRH+Payload(LID+GID)

* The router forwards using the GRH.

* The final destination is the ARP receiver - the destined endnode

* The ARP module picks up the packet payload

	- if GID-prefix local
		copy LID + GID as is
	  else if GID-prefix not local
		use GID-prefix to get the LID from the routing infrastructure


The IB router is not involved at all.



Of course for the above to be setup the IPoIB subnets must span IB
subnets first. We can do it only when they do.

Now consider the case which can be implemented. The IPoIB module
associated with the IB router responds (proxy-ARP) for a particular
node on the other IB subnet that it knows about.

If LID is not included you cannot access it directly at all. Only if
the LID is included can you point at the IB router (LID) and the
remote node (GID).

On the whole I didn't come across any reason that blocks any of the
solutions the no-LID camp wants. Additionally, if they are not
satisfied they can always ignore the LID and always go get it again
from the SA.

Vivek



> > 
> > I would like to conclude that discussion with out adding any counter
> > arguments
> > I agree that going the path of LID in the payload is not the clean approach,
> > as both Paul & Bill presented 
> > My understanding is that It could be made to work and solve other
> > multi-subnet issues that exists today
> > 
> > I'm personally willing to go for Paul's suggestion and use SLID, plus add a
> > way to indicate when to ignore it 
> > I hope that Vivek would agree to it too, and we could go to the next topic
> > on the agenda 
> > 
> > Paul, we need to see what we need to do in MSO to improve the multi-subnet
> > path discovery mechanism 
> > 
> > Yaron

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 18 18:07:58 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA00169
	for <ipoverib-archive@odin.ietf.org>; Fri, 18 Jan 2002 18:07:58 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id SAA09078
	for ipoverib-archive@odin.ietf.org; Fri, 18 Jan 2002 18:08:01 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA09054;
	Fri, 18 Jan 2002 18:07:00 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA09024
	for <ipoverib@ns.ietf.org>; Fri, 18 Jan 2002 18:06:58 -0500 (EST)
Received: from zinfandel.topspincom.com ([209.237.59.50])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA00126
	for <ipoverib@ietf.org>; Fri, 18 Jan 2002 18:06:54 -0500 (EST)
Received: from anarchy.topspincom.com (lab2.topspincom.com [10.0.0.210]) by zinfandel.topspincom.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13)
	id ZH5HKK60; Fri, 18 Jan 2002 15:10:27 -0800
Received: (from libor@localhost)
	by anarchy.topspincom.com (8.11.0/8.11.0) id g0INCAR04211
	for ipoverib@ietf.org; Fri, 18 Jan 2002 15:12:10 -0800
Date: Fri, 18 Jan 2002 15:12:10 -0800
From: Libor Michalek <libor@pobox.com>
To: ipoverib@ietf.org
Subject: Re: [ipoverib] reasons for not using LID ?
Message-ID: <20020118151210.B22989@topspincom.com>
References: <200201182153.g0ILr8H06022@eng4.beaverton.ibm.com> <200201182225.g0IMPE108726@eng4.beaverton.ibm.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <200201182225.g0IMPE108726@eng4.beaverton.ibm.com>; from kashyapv@us.ibm.com on Fri, Jan 18, 2002 at 02:25:12PM -0800
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

On Fri, Jan 18, 2002 at 02:25:12PM -0800, Vivek Kashyap wrote:
> 
> The way the router implementation was discussed with the router
> looking into the packet was not clean.
> 
> But the router is not involved at all. See my earlier mail in this regard.
> 
> Consider:
> 
> * ARP packet:   LRH+GRH+Payload(LID+GID)
> 
> * The router forwards using the GRH.
> 
> * The final destination is the ARP receiver - the destined endnode
> 
> * The ARP module picks up the packet payload
> 
> 	- if GID-prefix local
> 		copy LID + GID as is
> 	  else if GID-prefix not local
> 		use GID-prefix to get the LID from the routing infrastructure

  Are you proposing the addition of this LID+GID usage algorithm, or
something similar, to the IPoIB specification? I'm concerned about 
interoperability across multiple implementations. If it is decided
that both LID and GID are included then there needs to be a requirment
for their usage by a compliant implementation, to insure interoperability.

  The fear is that as soon as a major implementation starts using the LID 
without looking at the GID, all IPoIB subnets will be restricted to a 
single IB subnet to maintain compatability.

  If LID+GID can be used anyway an implementation wishes, then the
spec might as well come out and say that an IP subnet cannot span
multiple IB subnets. I'm not saying that this is a bad thing, only
that the intention should be made clear to avoid future confusion. 
  
-Libor

--
Libor Michalek
libor@topspin.com
Topspin Communications.


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 18 19:45:06 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA01866
	for <ipoverib-archive@odin.ietf.org>; Fri, 18 Jan 2002 19:45:06 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id TAA11401
	for ipoverib-archive@odin.ietf.org; Fri, 18 Jan 2002 19:45:08 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA11383;
	Fri, 18 Jan 2002 19:44:22 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA11352
	for <ipoverib@ns.ietf.org>; Fri, 18 Jan 2002 19:44:21 -0500 (EST)
Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA01857
	for <ipoverib@ietf.org>; Fri, 18 Jan 2002 19:44:16 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e31.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id TAA231842;
	Fri, 18 Jan 2002 19:41:16 -0500
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0J0iHD43062;
	Fri, 18 Jan 2002 17:44:17 -0700
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id g0J0iHg05614;
	Fri, 18 Jan 2002 16:44:17 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0J0iGp19434;
	Fri, 18 Jan 2002 16:44:16 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201190044.g0J0iGp19434@eng4.beaverton.ibm.com>
Subject: Re: [ipoverib] reasons for not using LID ?
To: libor@pobox.com (Libor Michalek)
Date: Fri, 18 Jan 2002 16:44:15 -0800 (PST)
Cc: ipoverib@ietf.org
In-Reply-To: <20020118151210.B22989@topspincom.com> from "Libor Michalek" at Jan 18, 2002 02:12:10 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

> 
> On Fri, Jan 18, 2002 at 02:25:12PM -0800, Vivek Kashyap wrote:
> > 
> > The way the router implementation was discussed with the router
> > looking into the packet was not clean.
> > 
> > But the router is not involved at all. See my earlier mail in this regard.
> > 
> > Consider:
> > 
> > * ARP packet:   LRH+GRH+Payload(LID+GID)
> > 
> > * The router forwards using the GRH.
> > 
> > * The final destination is the ARP receiver - the destined endnode
> > 
> > * The ARP module picks up the packet payload
> > 
> > 	- if GID-prefix local
> > 		copy LID + GID as is
> > 	  else if GID-prefix not local
> > 		use GID-prefix to get the LID from the routing infrastructure
> 
>   Are you proposing the addition of this LID+GID usage algorithm, or
> something similar, to the IPoIB specification? I'm concerned about 
> interoperability across multiple implementations. If it is decided
> that both LID and GID are included then there needs to be a requirment
> for their usage by a compliant implementation, to insure interoperability.

Things need to interoperate, I am fully behind that. Whenever the
IPoIB subnets get to span the IB subnets this algorithm would become
the requirement. 

> 
>   The fear is that as soon as a major implementation starts using the LID 
> without looking at the GID, all IPoIB subnets will be restricted to a 
> single IB subnet to maintain compatability.

        I don't think so. LID+GID are both used in the ARP packet. At
        present the implementations won't do cross-subnet packets but
        when they do the above algorithm will have to be followed.

> 
>   If LID+GID can be used anyway an implementation wishes, then the
> spec might as well come out and say that an IP subnet cannot span
> multiple IB subnets. I'm not saying that this is a bad thing, only
> that the intention should be made clear to avoid future confusion. 

        Do you mean that the spec should say that at present the IPoIB
        subnets don't span IB subnets? The architecture draft says
        that much(I have received some comments on fine-tuning the
        words but the meaning remains the same). The encapsulation
        draft can indicate that the GID-prefix needs to be checked.

        I am not in favour of solving the span IB subnet problem in
        general but the above doesn't block that.


Vivek

>   
> -Libor
> 
> --
> Libor Michalek
> libor@topspin.com
> Topspin Communications.
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 18 20:28:01 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA02371
	for <ipoverib-archive@odin.ietf.org>; Fri, 18 Jan 2002 20:28:00 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id UAA12285
	for ipoverib-archive@odin.ietf.org; Fri, 18 Jan 2002 20:28:03 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA12247;
	Fri, 18 Jan 2002 20:24:52 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA12218
	for <ipoverib@ns.ietf.org>; Fri, 18 Jan 2002 20:24:50 -0500 (EST)
Received: from calliope1.fm.intel.com (fmfdns01.fm.intel.com [132.233.247.10])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA02335
	for <ipoverib@ietf.org>; Fri, 18 Jan 2002 20:24:47 -0500 (EST)
Received: from fmsmsxvs041.fm.intel.com (fmsmsxvs041.fm.intel.com [132.233.42.126])
	by calliope1.fm.intel.com (8.9.1a+p1/8.9.1/d: relay.m4,v 1.48 2001/12/13 16:27:50 root Exp $) with SMTP id BAA22339
	for <ipoverib@ietf.org>; Sat, 19 Jan 2002 01:24:49 GMT
Received: from FMSMSX016.fm.intel.com ([132.233.42.195])
 by fmsmsxvs041.fm.intel.com (NAVGW 2.5.1.16) with SMTP id M2002011817254626858
 ; Fri, 18 Jan 2002 17:25:46 -0800
Received: by fmsmsx016.fm.intel.com with Internet Mail Service (5.5.2653.19)
	id <DFBZ7WDJ>; Fri, 18 Jan 2002 17:24:48 -0800
Message-ID: <D9223EB959A5D511A98F00508B68C20C02ACF583@ORSMSX108>
From: "Deleganes, Ellen M" <ellen.m.deleganes@intel.com>
To: "'Vivek Kashyap'" <kashyapv@us.ibm.com>, libor@pobox.com
Cc: ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?
Date: Fri, 18 Jan 2002 17:24:45 -0800
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="ISO-8859-1"
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

What I'm not clear about is why require the ARP responder to do the extra
work to figure out what LID to insert, when the correct SLID is returned in
the header? Those who want to use the LID and SL without performing a path
lookup can do so from the ARP response. All of this information is returned
in the completion status of a UD packet, as Jim Pinkerton pointed out.

It seems like from various other messages on this reflector that putting the
LID in the ARP response has potential interoperability issues if IP over IB
happens to span multiple IB subnets. Even though the infrastructure is not
yet there in IB, it seems foolish to build in something that could preclude
being able to do this in the future.

Ellen Deleganes

-----Original Message-----
From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
Sent: Friday, January 18, 2002 4:44 PM
To: libor@pobox.com
Cc: ipoverib@ietf.org
Subject: Re: [ipoverib] reasons for not using LID ?


> 
> On Fri, Jan 18, 2002 at 02:25:12PM -0800, Vivek Kashyap wrote:
> > 
> > The way the router implementation was discussed with the router
> > looking into the packet was not clean.
> > 
> > But the router is not involved at all. See my earlier mail in this
regard.
> > 
> > Consider:
> > 
> > * ARP packet:   LRH+GRH+Payload(LID+GID)
> > 
> > * The router forwards using the GRH.
> > 
> > * The final destination is the ARP receiver - the destined endnode
> > 
> > * The ARP module picks up the packet payload
> > 
> > 	- if GID-prefix local
> > 		copy LID + GID as is
> > 	  else if GID-prefix not local
> > 		use GID-prefix to get the LID from the routing
infrastructure
> 
>   Are you proposing the addition of this LID+GID usage algorithm, or
> something similar, to the IPoIB specification? I'm concerned about 
> interoperability across multiple implementations. If it is decided
> that both LID and GID are included then there needs to be a requirment
> for their usage by a compliant implementation, to insure interoperability.

Things need to interoperate, I am fully behind that. Whenever the
IPoIB subnets get to span the IB subnets this algorithm would become
the requirement. 

> 
>   The fear is that as soon as a major implementation starts using the LID 
> without looking at the GID, all IPoIB subnets will be restricted to a 
> single IB subnet to maintain compatability.

        I don't think so. LID+GID are both used in the ARP packet. At
        present the implementations won't do cross-subnet packets but
        when they do the above algorithm will have to be followed.

> 
>   If LID+GID can be used anyway an implementation wishes, then the
> spec might as well come out and say that an IP subnet cannot span
> multiple IB subnets. I'm not saying that this is a bad thing, only
> that the intention should be made clear to avoid future confusion. 

        Do you mean that the spec should say that at present the IPoIB
        subnets don't span IB subnets? The architecture draft says
        that much(I have received some comments on fine-tuning the
        words but the meaning remains the same). The encapsulation
        draft can indicate that the GID-prefix needs to be checked.

        I am not in favour of solving the span IB subnet problem in
        general but the above doesn't block that.


Vivek

>   
> -Libor
> 
> --
> Libor Michalek
> libor@topspin.com
> Topspin Communications.
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Sun Jan 20 10:48:40 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA12063
	for <ipoverib-archive@odin.ietf.org>; Sun, 20 Jan 2002 10:48:40 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id KAA19342
	for ipoverib-archive@odin.ietf.org; Sun, 20 Jan 2002 10:48:45 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id KAA19310;
	Sun, 20 Jan 2002 10:46:38 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id KAA19280
	for <ipoverib@optimus.ietf.org>; Sun, 20 Jan 2002 10:46:36 -0500 (EST)
Received: from mail2.microsoft.com (mail2.microsoft.com [131.107.3.124])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA11998
	for <ipoverib@ietf.org>; Sun, 20 Jan 2002 10:46:30 -0500 (EST)
Received: from INET-VRS-02.redmond.corp.microsoft.com ([157.54.8.110]) by mail2.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Sun, 20 Jan 2002 07:46:03 -0800
Received: from 157.54.8.109 by INET-VRS-02.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Sun, 20 Jan 2002 07:46:02 -0800
Received: from red-msg-02.redmond.corp.microsoft.com ([157.54.12.70]) by inet-hub-02.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Sun, 20 Jan 2002 07:46:02 -0800
x-mimeole: Produced By Microsoft Exchange V6.0.5762.3
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Subject: RE: [ipoverib] reasons for not using LID ?
Date: Sun, 20 Jan 2002 07:46:02 -0800
Message-ID: <24A715275661C8428C00432EFCA5CB7C055F9B03@red-msg-02.redmond.corp.microsoft.com>
Thread-Topic: [ipoverib] reasons for not using LID ?
Thread-Index: AcGfwe4W2I3YFrNARoCwBsMDx+P94wCA5GZw
From: "Jim Pinkerton" <jpink@microsoft.com>
To: <yaronh@voltaire.com>, "Arvind Murching" <ARVINDM@windows.microsoft.com>,
        <ipoverib@ietf.org>
X-OriginalArrivalTime: 20 Jan 2002 15:46:02.0752 (UTC) FILETIME=[907A5800:01C1A1C9]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id KAA19281
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit


Yaron,

The product your company has announced and demoed is much more than a
router - it is a layer 7 gateway. As such, parsing the internals of
every packet is no big deal - presumably that is what the box was
designed for. 

I believe Arvind was referring to the minimum functionality designed in
by the IBTA for an IB router - where parsing the packet body is
distinctly not required. Only parsing the LRH/GRH is required. The main
complexity is in managing the routing tables - the design intent is to
simply forward the data, while overwriting some of the LRH header fields
(such as the SLID, SL, etc). How to optimize this type of hardware
design is fairly well understood. In that world parsing each packet
payload in the fashion you describe really is a substantial amount of
additional work.



Jim




-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org] On Behalf
Of Yaron Haviv
Sent: Thursday, January 17, 2002 5:46 PM
To: Arvind Murching; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

Arvind,

The only real argument you make is implementation in Routers
Since we build such boxes, I could tell you dealing with an ARP in the
router in case of IP span IB is complex anyway 
Believe me, the simplification you are offering me creates more work
than help inside the router 
As you also stated in the passed there are also very little amount of
ARP packets, so replacing it in one or two paces doesn't change anything


Yaron

-----Original Message-----
From: Arvind Murching [mailto:arvindm@windows.microsoft.com]
Sent: Friday, January 18, 2002 3:34 AM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

Below...

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 5:31 PM
To: Arvind Murching; yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

My remarks attached

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of
Arvind Murching
Sent: Friday, January 18, 2002 2:57 AM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

The argument should be the other way around, i.e. fields need to justify
their necessity to be part of a protocol.
[YH] I think many arguments where already raised, now it's the
opposition
turn to put their arguments
[Arvind] The need for a redundant field is not obvious.

Some reasons why not:

- LID values cannot be used across IB subnet boundaries
[YH] you would any way need to store both LID (Router) and GID (node) in
this case somewhere, you wouldn't go to the SM for any new IP packet
wont
you
the routers in this case will any way need to change the LID in the ARP
packets (Proxy ARP and all that)
I believe this case is the one that benefits most from the LID, avoiding
the
long and complex multi-subnet resolution process
[Arvind] Consider the most general case of an IB router routing between
IB subnets. This router will not modify an ARP reply packet to insert
its LID into the payload. However, it will insert its SLID into the IB
header of packets that it forwards. Proxy ARP boxes can do the same,
too.

- A LID cannot be used as part of a static ARP entry
[YH] We didn't say not to have GID, you could still put GID as a static
entry

- The SLID is already present in case someone needs it
[YH] also regular ARP has it, so why not follow that path
[Arvind] This is not an argument for adding redundant fields.


My 2c
- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 4:47 PM
To: ipoverib@ietf.org
Subject: [ipoverib] reasons for not using LID ?

By now it is obvious that there are the ones that wants LID's in the ARP
packets

Could the ones that don't give their reasons for why not to
Not just things like "you don't need it", "we want to save 16-bit", etc'
But, reasons for what problems could arouse from having it in the packet
I haven't heard any real argument that would make me consider removing
it
until now
we then will review them and see if it is possible to deal with those
cases,
if not I personally would drop my support in LID's

If people want it for their reasons, the ones that don't also need to do
some explaining

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Sun Jan 20 12:18:38 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA13614
	for <ipoverib-archive@odin.ietf.org>; Sun, 20 Jan 2002 12:18:38 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id MAA21777
	for ipoverib-archive@odin.ietf.org; Sun, 20 Jan 2002 12:18:42 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA21762;
	Sun, 20 Jan 2002 12:17:53 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA21731
	for <ipoverib@optimus.ietf.org>; Sun, 20 Jan 2002 12:17:51 -0500 (EST)
Received: from mxout2.netvision.net.il (mxout2.netvision.net.il [194.90.9.21])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA13590
	for <ipoverib@ietf.org>; Sun, 20 Jan 2002 12:17:45 -0500 (EST)
Received: from vaio505he ([139.92.208.54]) by mxout2.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GQ800474YOQ8I@mxout2.netvision.net.il> for ipoverib@ietf.org;
 Sun, 20 Jan 2002 19:17:18 +0200 (IST)
Date: Sun, 20 Jan 2002 19:08:24 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: RE: [ipoverib] reasons for not using LID  - the end !
In-reply-to: 
 <24A715275661C8428C00432EFCA5CB7C055F9B03@red-msg-02.redmond.corp.microsoft.com>
To: "'Jim Pinkerton'" <jpink@microsoft.com>, yaronh@voltaire.com,
        "'Arvind Murching'" <ARVINDM@windows.microsoft.com>, ipoverib@ietf.org
Reply-to: yaronh@voltaire.com
Message-id: <005601c1a1d5$133941b0$61ac003e@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7BIT

Jim

As I mentioned in my previous email
I asked for reasons not just empty phrases, and people like Bill and Paul
presented good reasons
So I backed up from my previous position, this should be the nature of the
discussion I think
(My intention was to deal with ARP's in a different way than you describe
which is not so complicated however now it is behind us)

I accept the compromise that there will be an invalid SLID bit
This will enable in the common case to still avoid the SM as me and Vivek
wanted
And the routers will not need to look for the LID

I think Vivek by now agrees with me too, and we could move to the next topic
if there is such

Yaron


-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Jim Pinkerton
Sent: Sunday, January 20, 2002 5:46 PM
To: yaronh@voltaire.com; Arvind Murching; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?


Yaron,

The product your company has announced and demoed is much more than a
router - it is a layer 7 gateway. As such, parsing the internals of
every packet is no big deal - presumably that is what the box was
designed for.

I believe Arvind was referring to the minimum functionality designed in
by the IBTA for an IB router - where parsing the packet body is
distinctly not required. Only parsing the LRH/GRH is required. The main
complexity is in managing the routing tables - the design intent is to
simply forward the data, while overwriting some of the LRH header fields
(such as the SLID, SL, etc). How to optimize this type of hardware
design is fairly well understood. In that world parsing each packet
payload in the fashion you describe really is a substantial amount of
additional work.



Jim




-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org] On Behalf
Of Yaron Haviv
Sent: Thursday, January 17, 2002 5:46 PM
To: Arvind Murching; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

Arvind,

The only real argument you make is implementation in Routers
Since we build such boxes, I could tell you dealing with an ARP in the
router in case of IP span IB is complex anyway
Believe me, the simplification you are offering me creates more work
than help inside the router
As you also stated in the passed there are also very little amount of
ARP packets, so replacing it in one or two paces doesn't change anything


Yaron

-----Original Message-----
From: Arvind Murching [mailto:arvindm@windows.microsoft.com]
Sent: Friday, January 18, 2002 3:34 AM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

Below...

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 5:31 PM
To: Arvind Murching; yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

My remarks attached

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of
Arvind Murching
Sent: Friday, January 18, 2002 2:57 AM
To: yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?

The argument should be the other way around, i.e. fields need to justify
their necessity to be part of a protocol.
[YH] I think many arguments where already raised, now it's the
opposition
turn to put their arguments
[Arvind] The need for a redundant field is not obvious.

Some reasons why not:

- LID values cannot be used across IB subnet boundaries
[YH] you would any way need to store both LID (Router) and GID (node) in
this case somewhere, you wouldn't go to the SM for any new IP packet
wont
you
the routers in this case will any way need to change the LID in the ARP
packets (Proxy ARP and all that)
I believe this case is the one that benefits most from the LID, avoiding
the
long and complex multi-subnet resolution process
[Arvind] Consider the most general case of an IB router routing between
IB subnets. This router will not modify an ARP reply packet to insert
its LID into the payload. However, it will insert its SLID into the IB
header of packets that it forwards. Proxy ARP boxes can do the same,
too.

- A LID cannot be used as part of a static ARP entry
[YH] We didn't say not to have GID, you could still put GID as a static
entry

- The SLID is already present in case someone needs it
[YH] also regular ARP has it, so why not follow that path
[Arvind] This is not an argument for adding redundant fields.


My 2c
- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Thursday, January 17, 2002 4:47 PM
To: ipoverib@ietf.org
Subject: [ipoverib] reasons for not using LID ?

By now it is obvious that there are the ones that wants LID's in the ARP
packets

Could the ones that don't give their reasons for why not to
Not just things like "you don't need it", "we want to save 16-bit", etc'
But, reasons for what problems could arouse from having it in the packet
I haven't heard any real argument that would make me consider removing
it
until now
we then will review them and see if it is possible to deal with those
cases,
if not I personally would drop my support in LID's

If people want it for their reasons, the ones that don't also need to do
some explaining

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 21 00:58:09 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA23024
	for <ipoverib-archive@odin.ietf.org>; Mon, 21 Jan 2002 00:58:09 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id AAA11949
	for ipoverib-archive@odin.ietf.org; Mon, 21 Jan 2002 00:58:13 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id AAA11936;
	Mon, 21 Jan 2002 00:57:00 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id AAA11904
	for <ipoverib@optimus.ietf.org>; Mon, 21 Jan 2002 00:56:59 -0500 (EST)
Received: from mail3.microsoft.com (mail3.microsoft.com [131.107.3.123])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA23017
	for <ipoverib@ietf.org>; Mon, 21 Jan 2002 00:56:54 -0500 (EST)
Received: from INET-VRS-03.redmond.corp.microsoft.com ([157.54.5.27]) by mail3.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Sun, 20 Jan 2002 21:56:25 -0800
Received: from 157.54.8.109 by INET-VRS-03.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Sun, 20 Jan 2002 21:56:25 -0800
Received: from red-imc-02.redmond.corp.microsoft.com ([157.54.9.107]) by inet-hub-02.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Sun, 20 Jan 2002 21:56:25 -0800
Received: from win-imc-02.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.84]) by red-imc-02.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Sun, 20 Jan 2002 21:56:23 -0800
Received: from win-msg-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.52]) by win-imc-02.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3588.0);
	 Sun, 20 Jan 2002 21:54:12 -0800
X-MimeOLE: Produced By Microsoft Exchange V6.0.6132.0
Content-Class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Date: Sun, 20 Jan 2002 21:54:12 -0800
Message-ID: <2E33960095B58E40A4D3345AB9F65EC102450C9B@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
Thread-Topic: MCGroup Attach Failure --- IPoIB link and multicast draft - 
Thread-Index: AcGfCNOsvhHTwXEtTQe/oWYG9tphlQDLKRYQ
From: "Aditya Dube" <adube@windows.microsoft.com>
To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Cc: <ipoverib@ietf.org>
X-OriginalArrivalTime: 21 Jan 2002 05:54:12.0776 (UTC) FILETIME=[0D4BC280:01C1A240]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id AAA11905
Subject: [Ipoverib] RE: MCGroup Attach Failure --- IPoIB link and multicast draft -
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

Hello Jerry

I looked into this. An HCA can only join a limited number of MCGroups.
This limit is specified in the "Query HCA" Verb 11.2.1.2 (on pg 452). If
this number is exceeded then the "Attach QP to MCGroup" verb will fail
(11.3.1 page 496).

Whenever this number is exceeded the HCA will not be able to listen to
any MCast traffic on a new MCast GID because the sender of the data will
still continue to send data on these MCast GIDs unaware that the
recipient is unable to get the data.


Aditya Dube

-----Original Message-----
From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com] 
Sent: Wednesday, January 16, 2002 7:41 PM
To: Aditya Dube
Cc: ipoverib@ietf.org
Subject: Re: MCGroup Attach Failure --- IPoIB link and multicast draft -


Hello Aditya,

Do you have any reason to believe this case should be handled
differently by IPoIB stack than other IB multicast applications?

If the join operation is unreliable or intrinsically weak by IBA
design, we can always recommend a backup algorithm. (Any idea why
a join may fail?)

Jerry

>Hello Jerry
>
>I have a question from the Multicast draft that you posted a few days
>ago.
>
>What should a node that wishes to receive IP traffic on a MCast GID do
>if the 'Attach' (SubnAdmSet MCMemberRecord) operation on that IB MCast
>GID fails?
>
>
>Aditya Dube
>
>-----Original Message-----
>From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com] 
>Sent: Monday, January 14, 2002 12:20 PM
>To: ipoverib@ietf.org
>Subject: [Ipoverib] IPoIB link and multicast draft
>
>
>...<deleted>


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 21 12:18:57 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA17426
	for <ipoverib-archive@odin.ietf.org>; Mon, 21 Jan 2002 12:18:57 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id MAA19262
	for ipoverib-archive@odin.ietf.org; Mon, 21 Jan 2002 12:19:01 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA19233;
	Mon, 21 Jan 2002 12:18:09 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA19202
	for <ipoverib@optimus.ietf.org>; Mon, 21 Jan 2002 12:18:07 -0500 (EST)
Received: from pdxpo.dsl-only.net (pdxpo.dsl-only.net [63.105.16.3])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA17396
	for <ipoverib@ietf.org>; Mon, 21 Jan 2002 12:17:59 -0500 (EST)
Received: from bonfire (unverified [63.105.18.201]) by pdxpo.dsl-only.net
 (Rockliffe SMTPRA 4.5.4) with SMTP id <B0009293223@pdxpo.dsl-only.net> for <ipoverib@ietf.org>;
 Mon, 21 Jan 2002 09:16:57 -0800
From: "Bill Strahm" <bill@strahm.net>
To: <ipoverib@ietf.org>
Date: Mon, 21 Jan 2002 09:26:36 -0800
Message-ID: <HHEDIJGOEFLPCPNILMPACEDOCFAA.bill@strahm.net>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0)
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
Importance: Normal
Content-Transfer-Encoding: 7bit
Subject: [Ipoverib] Scheduling for Minnesota
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

It is time to schedule for Minnesota.

If you would like to make a presentation please contact me either through
the list or by e-mail and tell me what the subject is, when you will have
your draft posted on the subject, and the ammount of time that you will
need.

I am hoping that we can close on many of the issues before us at the
meeting, so please if you have an opinion present it...

Bill


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Mon Jan 21 18:00:21 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA27530
	for <ipoverib-archive@odin.ietf.org>; Mon, 21 Jan 2002 18:00:21 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id SAA00733
	for ipoverib-archive@odin.ietf.org; Mon, 21 Jan 2002 18:00:24 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA00503;
	Mon, 21 Jan 2002 17:59:49 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA00474
	for <ipoverib@ns.ietf.org>; Mon, 21 Jan 2002 17:59:48 -0500 (EST)
Received: from zinfandel.topspincom.com ([209.237.59.50])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA27501
	for <ipoverib@ietf.org>; Mon, 21 Jan 2002 17:59:44 -0500 (EST)
Received: from anarchy.topspincom.com (lab2.topspincom.com [10.0.0.210]) by zinfandel.topspincom.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13)
	id ZH5HKM19; Mon, 21 Jan 2002 15:03:22 -0800
Received: (from libor@localhost)
	by anarchy.topspincom.com (8.11.0/8.11.0) id g0LN4cv12207;
	Mon, 21 Jan 2002 15:04:38 -0800
Date: Mon, 21 Jan 2002 15:04:38 -0800
From: Libor Michalek <libor@pobox.com>
To: Vivek Kashyap <kashyapv@us.ibm.com>
Cc: ipoverib@ietf.org
Subject: Re: [ipoverib] reasons for not using LID ?
Message-ID: <20020121150438.D22989@topspincom.com>
References: <20020118151210.B22989@topspincom.com> <200201190044.g0J0iGp19434@eng4.beaverton.ibm.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <200201190044.g0J0iGp19434@eng4.beaverton.ibm.com>; from kashyapv@us.ibm.com on Fri, Jan 18, 2002 at 04:44:15PM -0800
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

On Fri, Jan 18, 2002 at 04:44:15PM -0800, Vivek Kashyap wrote:
> > On Fri, Jan 18, 2002 at 02:25:12PM -0800, Vivek Kashyap wrote:
> > > 
> > > Consider:
> > > 
> > > * ARP packet:   LRH+GRH+Payload(LID+GID)
> > > 
> > > * The router forwards using the GRH.
> > > 
> > > * The final destination is the ARP receiver - the destined endnode
> > > 
> > > * The ARP module picks up the packet payload
> > > 
> > > 	- if GID-prefix local
> > > 		copy LID + GID as is
> > > 	  else if GID-prefix not local
> > > 		use GID-prefix to get the LID from the routing infrastructure
> > 
> >   Are you proposing the addition of this LID+GID usage algorithm, or
> > something similar, to the IPoIB specification? I'm concerned about 
> > interoperability across multiple implementations. If it is decided
> > that both LID and GID are included then there needs to be a requirment
> > for their usage by a compliant implementation, to insure interoperability.
> 
> Things need to interoperate, I am fully behind that. Whenever the
> IPoIB subnets get to span the IB subnets this algorithm would become
> the requirement. 
> 
> > 
> >   The fear is that as soon as a major implementation starts using the LID 
> > without looking at the GID, all IPoIB subnets will be restricted to a 
> > single IB subnet to maintain compatability.
> 
>         I don't think so. LID+GID are both used in the ARP packet. At
>         present the implementations won't do cross-subnet packets but
>         when they do the above algorithm will have to be followed.

  If the algorithm is something that will have to be followed, then
shouldn't it be a part of the specification? If it is not part of
the specification, then why assume that all implementations would
work this way.

  In fact I would assume that if something is not stated it will be
implemented in two, potentially incompatable, ways. So if the spec
can result in two incompatable implementations, at least across IB
subnets, it should be stated that the scope of the spec is only
within a single IB subnet.
 
> > 
> >   If LID+GID can be used anyway an implementation wishes, then the
> > spec might as well come out and say that an IP subnet cannot span
> > multiple IB subnets. I'm not saying that this is a bad thing, only
> > that the intention should be made clear to avoid future confusion. 
> 
>         Do you mean that the spec should say that at present the IPoIB
>         subnets don't span IB subnets? The architecture draft says
>         that much(I have received some comments on fine-tuning the
>         words but the meaning remains the same). The encapsulation
>         draft can indicate that the GID-prefix needs to be checked.

  The architecture draft states a number of IB problems that prevent
cross subnet packets. I'm concerned about problems with how the spec
uses IB which would introduce incompatability once/if the IB problems
are resolved.

>         I am not in favour of solving the span IB subnet problem in
>         general but the above doesn't block that.

  There are problems with IB and multiple subnets, and I have no wish
to tackle those problems. However, this is more of a problem with how
IB is used. Even if all the IB specific problems are resolved, an IP
subnet will not work across two IB subnets if the GID is not used
correctly, the correct usage should be in the spec. If the correct
usage is not in the spec, it needs to be stated that two compliant
implementations can only be guaranteed to interoperate in a single
IB subnet. 


-Libor

--
Libor Michalek
libor@topspin.com
Topspin Communications.

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Mon Jan 21 21:24:06 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA00069
	for <ipoverib-archive@odin.ietf.org>; Mon, 21 Jan 2002 21:24:01 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id VAA07095
	for ipoverib-archive@odin.ietf.org; Mon, 21 Jan 2002 21:24:04 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA07068;
	Mon, 21 Jan 2002 21:22:35 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA07035
	for <ipoverib@ns.ietf.org>; Mon, 21 Jan 2002 21:22:29 -0500 (EST)
Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA00056
	for <ipoverib@ietf.org>; Mon, 21 Jan 2002 21:22:25 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e31.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id VAA72258;
	Mon, 21 Jan 2002 21:19:24 -0500
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0M2MR7100962;
	Mon, 21 Jan 2002 19:22:27 -0700
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0M2MOV15446;
	Mon, 21 Jan 2002 18:22:24 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0M2MME15887;
	Mon, 21 Jan 2002 18:22:22 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201220222.g0M2MME15887@eng4.beaverton.ibm.com>
Subject: Re: [ipoverib] reasons for not using LID ?
To: libor@pobox.com (Libor Michalek)
Date: Mon, 21 Jan 2002 18:22:21 -0800 (PST)
Cc: kashyapv@us.ibm.com (Vivek Kashyap), ipoverib@ietf.org
In-Reply-To: <20020121150438.D22989@topspincom.com> from "Libor Michalek" at Jan 21, 2002 02:04:38 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit


>   If the algorithm is something that will have to be followed, then
Libor,
        Maybe I wasn't as clear as you wanted it but I was agreeing
        with you that we need to make it part of the spec. No argument
        there. 


> shouldn't it be a part of the specification? If it is not part of
> the specification, then why assume that all implementations would
> work this way.
> 
>   In fact I would assume that if something is not stated it will be
> implemented in two, potentially incompatable, ways. So if the spec
> can result in two incompatable implementations, at least across IB
> subnets, it should be stated that the scope of the spec is only
> within a single IB subnet.

Yes, it needs to be stated in the spec.

We also need to, independent of the above, answer the question - why
do we need to span IP subnet across IB subnets ? It has been stated
often enough that IP subnets will span IB subnets without any
justification. The maximum I have heard is that it is possible to do
when IBTA solves the IB level issues. That is neither here nor there.

Why do we need IP subnets spanning IB subnets ?

Vivek



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Mon Jan 21 21:34:21 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA00653
	for <ipoverib-archive@odin.ietf.org>; Mon, 21 Jan 2002 21:34:21 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id VAA07930
	for ipoverib-archive@odin.ietf.org; Mon, 21 Jan 2002 21:34:25 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA07729;
	Mon, 21 Jan 2002 21:32:36 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA07656
	for <ipoverib@ns.ietf.org>; Mon, 21 Jan 2002 21:32:32 -0500 (EST)
Received: from palrel11.hp.com (palrel11.hp.com [156.153.255.246])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA00164
	for <ipoverib@ietf.org>; Mon, 21 Jan 2002 21:32:24 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel11.hp.com (Postfix) with ESMTP id E5BD1E0016C
	for <ipoverib@ietf.org>; Mon, 21 Jan 2002 18:31:57 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id SAA08004
	for <ipoverib@ietf.org>; Mon, 21 Jan 2002 18:31:57 -0800 (PST)
Message-Id: <5.1.0.14.2.20020121102805.00aef950@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Mon, 21 Jan 2002 10:40:16 -0800
To: ipoverib@ietf.org
From: Michael Krause <krause@cup.hp.com>
Subject: Re: [Ipoverib] RE: MCGroup Attach Failure --- IPoIB link and
  multicast draft -
In-Reply-To: <2E33960095B58E40A4D3345AB9F65EC102450C9B@win-msg-01.wingro
 up.windeploy.ntdev.microsoft.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

At 09:54 PM 1/20/2002 -0800, Aditya Dube wrote:
>Hello Jerry
>
>I looked into this. An HCA can only join a limited number of MCGroups.
>This limit is specified in the "Query HCA" Verb 11.2.1.2 (on pg 452). If
>this number is exceeded then the "Attach QP to MCGroup" verb will fail
>(11.3.1 page 496).
>
>Whenever this number is exceeded the HCA will not be able to listen to
>any MCast traffic on a new MCast GID because the sender of the data will
>still continue to send data on these MCast GIDs unaware that the
>recipient is unable to get the data.

There is no promiscuous mode within IB that would allow all joins to always 
succeed. This stems in part from allowing SMs to layout the any logical 
routing topology that they desire.  In addition, HCAs are restricted to how 
many MGIDs they can multiplex on a given port which is an 
implementation-specific issue.  For IP over IB, I do not view any of these 
issues as a problem or requiring additional work from the IBTA as these are 
not architectural limitations.  The IP over IB effort should simply require 
that for each partition that the endnode wants to provide IP service, there 
should be at a minimum a corresponding set of multicast resources 
available.  HCA vendors will then rely upon market input to guide them in 
updating their requirements.

In May 2001, HP made available a set of IB HCA, TCA, Switch, and Router 
requirements and motivations to help vendors decide what features and 
associated resources to implement.  This material is available to all and 
within one will find the multicast requirements.  The minimum acceptable 
for a HCA was 512 per port; optimal was 2048.  Additional values were 
provided for TCA, Switch, and Routers.  In discussions with many IHVs who 
have received this material, it is fair to say that the majority will 
implement generally more than the minimums and quite often near the 
optimals with some exceeding even these.  As such, the market will be able 
to support IP over IB requirements for the vast majority of the customer 
requirements thereby eliminating the need for IP over IB to do anything 
special to recover from such a group join problem.

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Mon Jan 21 21:34:25 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA00698
	for <ipoverib-archive@odin.ietf.org>; Mon, 21 Jan 2002 21:34:25 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id VAA07944
	for ipoverib-archive@odin.ietf.org; Mon, 21 Jan 2002 21:34:28 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA07902;
	Mon, 21 Jan 2002 21:32:45 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA07679
	for <ipoverib@ns.ietf.org>; Mon, 21 Jan 2002 21:32:33 -0500 (EST)
Received: from palrel11.hp.com (palrel11.hp.com [156.153.255.246])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA00166
	for <ipoverib@ietf.org>; Mon, 21 Jan 2002 21:32:25 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel11.hp.com (Postfix) with ESMTP id 92F85E00CE0
	for <ipoverib@ietf.org>; Mon, 21 Jan 2002 18:31:58 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id SAA08016
	for <ipoverib@ietf.org>; Mon, 21 Jan 2002 18:31:58 -0800 (PST)
Message-Id: <5.1.0.14.2.20020121123302.02a24e88@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Mon, 21 Jan 2002 12:40:34 -0800
To: ipoverib@ietf.org
From: Michael Krause <krause@cup.hp.com>
Subject: Re: [ipoverib] reasons for not using LID ?
In-Reply-To: <200201181822.g0IIMIf19680@eng4.beaverton.ibm.com>
References: <HHEDIJGOEFLPCPNILMPAOECLCFAA.bill@strahm.net>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

At 10:22 AM 1/18/2002 -0800, Vivek Kashyap wrote:
>Bill, Yaron,
>
>         There are two situations
>                 - proxy-ARP for a node on another subnet
>                 - span IP subnets across IB subnets
>
>1.      proxy-ARP for a node on another subnet works very easily.
>         You return the LID of the local IB router and the GID of the
>         remote IB node. Note this is possible with the current
>         specification. If only the GID is included excluding the LID
>         this is not possible since you can't lookup the remote node's
>         GID with the SA. You need to include both. [ SA to SA
>         coordination within an IB subnet isn't specified what of
>         across the subnet. ]
>
>         Note that the ARP packet is fielded by the IPoIB device
>         logically/physically associated with the IB router. The ARP
>         packet itself does not travel beyond the IB subnet.

Within the requester's IB subnet, there will exist at least one endnode 
that will respond on behalf of any request that targets an endnode that 
exists on another IB or IP subnet.   The responder will work with the SA to 
determine the IB router; an IP router would respond on behalf of its 
responsibility "cloud" and it too would work with the SA to determine the 
IB LRH values to use to establish appropriate communication between the 
requester and the IP router.  In both cases, the responder is responsible 
for determining the appropriate LRH / GRH (where applicable) header 
parameters to establish the communication channel.

>2.      For IP subnets to span IB subnets there is too much
>         speculation and no justification. First of all we don't know
>         why we need it. We don't know when it will be possible. We
>         don't know if it will be possible. It is a complicated issue -
>         too many things have to be updated in IB architecutre.

The justification is that vendors and customers believe it is the right 
thing to do and there is both technical and economic feasibility to achieve 
this objective.  Unless there is strong consensus to not support this type 
of functionality, the workgroup must take steps to enable it or at least 
not preclude this from being implemented at some future date once the IBTA 
has provided the appropriate specification updates.

>         Even for argument's sake lets take it that it was possible for
>         IP subnets to span IB subnets. The solution is very simple
>         when LID is included. And it is compatible with the solution
>         in 1. above.
>
>         The receiving node's ARP module looks at the GID-prefix. If
>         the prefix is of another subnet then the routing
>         infrastructure is looked up and the router's LID is recorded
>         otherwise the LID in the packet is recorded.
>
>
>I find it odd that we are discussing 'IP subnets spanning IB subnets'
>but not proxy-ARP that can work, happen and maybe is all that will
>ever be needed or possible(if IBTA doesn't solve routing/cross subnet
>issues).

I believe Paul provided a good starting position on a compromise a few days 
back.  Why don't people debate that position instead of repeating the same 
arguments on either side.

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Mon Jan 21 21:34:27 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA00727
	for <ipoverib-archive@odin.ietf.org>; Mon, 21 Jan 2002 21:34:26 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id VAA07958
	for ipoverib-archive@odin.ietf.org; Mon, 21 Jan 2002 21:34:30 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA07796;
	Mon, 21 Jan 2002 21:32:39 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA07661
	for <ipoverib@ns.ietf.org>; Mon, 21 Jan 2002 21:32:33 -0500 (EST)
Received: from palrel10.hp.com (palrel10.hp.com [156.153.255.245])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA00170
	for <ipoverib@ietf.org>; Mon, 21 Jan 2002 21:32:28 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel10.hp.com (Postfix) with ESMTP id CD6D2400676
	for <ipoverib@ietf.org>; Mon, 21 Jan 2002 18:31:58 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id SAA08019
	for <ipoverib@ietf.org>; Mon, 21 Jan 2002 18:31:58 -0800 (PST)
Message-Id: <5.1.0.14.2.20020121124553.028e2350@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Mon, 21 Jan 2002 12:48:19 -0800
To: ipoverib@ietf.org
From: Michael Krause <krause@cup.hp.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
In-Reply-To: <00ed01c19f93$7e74b070$970f020a@Fishhead>
References: <200201170927.g0H9RBg12021@eng4.beaverton.ibm.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

At 02:13 PM 1/17/2002 -0600, Roger Ronald wrote:
>I think it's important to note that changes in LID should be infrequent.
>Very infrequent.
>
>VIEO is one of a small number of companies that has a SM at this time.
>We plan to maintain LID assignments whenever possible. That means
>across reboots, across power fails and across subnet merges (when possible).
>
>There are corner cases where LID may need to change, just like IP
>addresses occasionaly need to change. But these are VERY odd
>events...like the first time merge of fabrics. They won't happen
>hourly, daily, monthly..... They may happen once or twice in a
>fabric. Ever (unless the admin wants to make changes for the heck of it).

While the LID range for a given port may not change often, the LID / SL 
appropriate for a given path between two endnodes may change for a variety 
of reasons based on workload objectives / QoS requirements within the 
fabric for a given service.  The IB APM mechanisms may be used to update 
these path selections without informing IP of the change (an implementation 
option).  Thus, the rate of LID change may not be as low as you contend but 
it will also not be as high as others contend making the performance issues 
raised perhaps moot.

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Mon Jan 21 21:34:38 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA00840
	for <ipoverib-archive@odin.ietf.org>; Mon, 21 Jan 2002 21:34:38 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id VAA07974
	for ipoverib-archive@odin.ietf.org; Mon, 21 Jan 2002 21:34:42 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA07818;
	Mon, 21 Jan 2002 21:32:40 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA07668
	for <ipoverib@ns.ietf.org>; Mon, 21 Jan 2002 21:32:33 -0500 (EST)
Received: from palrel10.hp.com (palrel10.hp.com [156.153.255.245])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA00169
	for <ipoverib@ietf.org>; Mon, 21 Jan 2002 21:32:28 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel10.hp.com (Postfix) with ESMTP id 4AA774003C7
	for <ipoverib@ietf.org>; Mon, 21 Jan 2002 18:31:58 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id SAA08010
	for <ipoverib@ietf.org>; Mon, 21 Jan 2002 18:31:58 -0800 (PST)
Message-Id: <5.1.0.14.2.20020121112853.00aefbd0@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Mon, 21 Jan 2002 11:46:01 -0800
To: ipoverib@ietf.org
From: Michael Krause <krause@cup.hp.com>
Subject: RE: [ipoverib] reasons for not using LID ?
In-Reply-To: <001501c1a01f$161156f0$61ac003e@vaio505he>
References: <HHEDIJGOEFLPCPNILMPAEEBACFAA.bill@strahm.net>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

At 02:53 PM 1/18/2002 +0200, Yaron Haviv wrote:
>Bill,
>
>First when forwarding packets from one subnet to another, the LRH header
>needs to be reconstructed

 From a logical perspective, the LRH is stripped off by the ingress IB 
router port and added by the egress router port.  LRH construction and GRH 
updates are described in 19.2.4 volume 1, version 1.0.a.  The primary 
fields impacted by crossing IB subnet boundaries are: DLID, SLID, SL, VL, 
and Hop Count.  There is no statement concerning the GRH flow label but 
this is also subject to modification.

>You assume that the IB routers works like a layer 2 Ethernet switch
>Unfortunately this is not the case, since as mike noted I couldn't reveal
>the content of the LWG discussions I cannot go into it

An IB router from an IP perspective layer 2 logical perspective is no 
different than an Ethernet switch. This was noted at the first IETF BoF 
last Spring when this issue was raised and one person stated that 
individual IB subnets can be treated as point-to-point links from an IP 
perspective for much of the issues being raised.  There is nothing within 
the current specification or from an IB architecture perspective that would 
preclude someone from treating them as such for IP purposes.  IB routers do 
not know anything about what they carry within their payload and thus are 
not aware whether the traffic is IP, ARP, Application, etc.  They only know 
parse the GRH to determine the longest prefix match and combine this with 
other fields within the GRH to determine the egress port, update the GRH, 
and build a new LRH.

>Lets just say as one of the people active in the router sub-group that it is
>totally different and a lot of things still needs to be determined
>I would choose at least initially to implement something like Proxy ARP to
>deal with the Issues
>One difference I could point out is that the Ethernet spanning tree is a
>self-learning mechanism where as IB routing is not
>The Proxy ARP mechanism compensates for that fact

An IB router is not an IP router (though an implementation may support 
both) and as such the proxy ARP mechanism is only applicable to an IP 
router - it has no bearing on the IB router protocol or the IB router 
responsibilities.

>When digging into it and going over all the potential problems, I would
>personally like to avoid the SM and especially when going to a different
>subnet
>And not just because of the few extra milliseconds delay, and SM overhead,
>but because I know it could work in all the scenarios

The IB router must interact with each subnet SM to understand its LID, LMC, 
VL, link width, etc.  The IB router must interact with each subnet SA to 
understand all of the destination endnodes that it is allowed to reach.  It 
must also interact with the SA to understand if there are any other IB 
routers attached to this subnet so that the router protocol can establish 
the appropriate router tables, etc. between these two ports.  All of this 
has been described in the various presentations made by the IBTA as well as 
the current specification.  I do not see how the router can avoid 
communication with the SM for fabric-specific values and with the SA for 
path record determination.  I expect the router to learn most of this 
information during its initialization steps and then update as required to 
deal with fabric event changes.  I do not see the need for the router to 
impose any serious performance requirements on the SM or the SA.

>One thing worth digging into is how to enable avoiding the SM using the SLID
>and other extra information that will enable detecting when that LID is
>invalid, plus maybe the Valid LID, this way if/when IB routing will be
>different it might answer your desires too

Again, I don't see how one avoids the SM or the SA if each local subnet's 
switching / QoS policies are to be complied with to ensure the fabric 
remains operational.

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Mon Jan 21 21:34:50 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA00941
	for <ipoverib-archive@odin.ietf.org>; Mon, 21 Jan 2002 21:34:50 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id VAA07999
	for ipoverib-archive@odin.ietf.org; Mon, 21 Jan 2002 21:34:53 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA07861;
	Mon, 21 Jan 2002 21:32:42 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA07685
	for <ipoverib@ns.ietf.org>; Mon, 21 Jan 2002 21:32:33 -0500 (EST)
Received: from palrel11.hp.com (palrel11.hp.com [156.153.255.246])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA00165
	for <ipoverib@ietf.org>; Mon, 21 Jan 2002 21:32:25 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel11.hp.com (Postfix) with ESMTP id 6DCCAE00C2C
	for <ipoverib@ietf.org>; Mon, 21 Jan 2002 18:31:58 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id SAA08013
	for <ipoverib@ietf.org>; Mon, 21 Jan 2002 18:31:58 -0800 (PST)
Message-Id: <5.1.0.14.2.20020121114657.028dbc10@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Mon, 21 Jan 2002 12:31:53 -0800
To: ipoverib@ietf.org
From: Michael Krause <krause@cup.hp.com>
Subject: Re: [ipoverib] reasons for not using LID ?
In-Reply-To: <200201181802.g0II2iI17782@eng4.beaverton.ibm.com>
References: <4D027986353D1341ADA4F2A4F8A170760E3175@cceexc18.americas.cpqcorp.net>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

At 10:02 AM 1/18/2002 -0800, Vivek Kashyap wrote:
> >
> > 1) a LID in the body of the of the ARP is insufficient for IB routing,
> > you need the LID/SL pair in order to actually get a packet anywhere.  IB
> > provides NO WAY to select an SL, based only on the LID, unless the ARP
> > requestor already has that LID and an SL cached or is willing to look it
> > up with the SA.  Since one of the reasons for including the LID at all
> > is to avoid this lookup, the solution is not complete without SL.
> > [Please don't respond that some implementation specific mechanism can be
> > chosen to select SL; there is no way for an endnode implementation to
> > determine the SA policies for SL usage at this time, so it is not
> > possible for the endnode to reliably chose an SL without asking the SA
> > (or caching a LID/SL derived from the SA).
>
>         SL can be determined based on workload characteristics. It is
>         not tied to LID. An implementation is free to go to the SA or
>         determine it on such a mapping. The IB spec leaves it upto the
>         imlementation to make the choice - and some of us have.

The SL is an opaque identifier derived from the path record. The meaning of 
this identifier is local to each IB subnet, i.e. it is not a global or 
well-defined value.  Therefore there is no workload characteristics one may 
use to determine what is the appropriate value for the SL.. Further, given 
the SL is used to map VLs within a switch and thus may be used to break 
deadlock loops within an IB subnet, there can be an implicit relationship 
between the SL and the SLID in terms of the appropriate LID to use to 
return a response.  The only way I can see to take the SL and SLID and 
re-use them for a unicast ARP response would be to have a homogeneous 
implementation, i.e. single vendor driven - I see nothing to preclude 
someone from developing such a solution and nothing within the current IP 
over IB architecture per se.

> > 2) LID in the body of the ARP will be invalid if the ARP crosses IB
> > subnets.  This has been discussed a lot already, but the following
> > observations seem pertinent:
> >  a) IB routers will happen someday, it would be nice to not have to
> >   go back to this process to fix ARP when it does happen.
>
>         As noted in replies to the above 'reason' there is nothing to
>         fix. Still nothing has been specified when this will happen
>         and why will it even be needed.
>
>         ARP across IB subnet:
>
>         On receiving an ARP request the receiving node will check the
>         GID-prefix. If it is of the remote node it asks the routing
>         infrastructure for the LID. If the GID is local it just uses
>         it. It is very simple.

The GID within the ARP request payload is used to determine whether an 
endnode is on a local or remote IB subnet.  The responder will need to 
consult either it local cache of address vectors for this endnode or 
consult the SA to acquire a path record to determine the appropriate path 
(IB LRH parameters) to return a response.  If the response does not cross 
an IB subnet, the responder does not need to include a GRH within the 
response thus the GID is not necessarily used / returned on the wire itself 
(though it will be used to update the responder's ARP cache).

> >  b) The IB Management group currently intends to support cross
> >   subnet operation for IB via use of the same SA PathRecord Lookup
> >   mechanism as is used for local operations.  This is primarily so
> >   that early endnode implementations can operate without changes.
> >   It is possible that other mechanisms may be defined as
> >   optimizations for routers, but that is at best an early concept. If
> >   we support a GID lookup for the ARP packets (and either drop body
> >   LIDs, use IB Header SLIDs, or indicate when to ignore them) ARP and
> >   IP can work across IB subnet boundaries.
>
>         Lot of speculation. Out of scope. We need to work with what is
>         there and is likely to be present for a long time. A short
>         incomplete list of IBTA to do:
>
>                 First routers need to be fully defined.

IB routers are defined in terms of the SM / SA interactions.  This is 
documented in the current IB specification.  While there are on-going 
errata to update the current specification, there is nothing that directly 
impacts IP operation over IB routers as they are currently defined.

>                 Then multicast needs to fleshed out.

IB multicast currently only lacks the multicast group management 
protocol.  Within the industry, there are many technologies that have 
allowed multiple group management protocols to be implemented on top of a 
single fabric type.  There is currently nothing to preclude IB management 
vendors from providing multicast using existing technologies and perhaps 
the IBTA should take notice of how it might benefit from a similar strategy.

>                 Partitioning across subnets has to be figured out.
>                 SA/SM coordination has to be specified.
>                 Routing protocol has to be specified.
>                 Node to Routing infrastructure interaction has to be 
> specified
>                 SA to Router interaction
>
>
>         Then the usefulness of such a setup has to be justified.

The fact that some of these items have yet to be worked out does not 
translate into the requirement to support IP across multiple IB subnets 
cannot be justified.  Requirements are business and technology driven and 
many have concluded that IP over multiple IB subnets is a MUST requirement 
for this technology and have people working within the IETF and the IBTA to 
ensure this requirement can be met as products become ready.  Understanding 
the issues up front provides both organizations with the needed guidance to 
create something viable and interoperable.


> > 3) As it happens, it is perfectly legal for an IB fabric to use
> > different LID addresses to talk to the same device from different other
> > devices, that is one of the reasons that we allow up to 2^LMC addresses
> > for each device.  (That's right, there are potentially up to 128 LID
> > addresses for each port!)  And not all LID/SL addresses need be
> > reachable from all other devices.  This means that taking a LID from the
> > ARP Request as a way to communicate, may not work.  And it is not
> > currently possible for an endnode to determine in a defined way that the
> > SA uses this type of policy, so endnodes have NO architected WAY to tell
> > when to ignore the LID and go for GID on their own.
>
>There is no architected way to choose which of the LMC bits to pick.
>If multiple paths are registered at the SA a GID lookup will return a
>set of them. It is then upto the responder to pick the path it
>chooses. which one will it choose ? how? It is a lot better for the
>server, which setup the paths for a reason, specifies the one it wants
>used. It is *better* to use LIDs in this case.

Hence why one should use the address vector that corresponds to a given 
service.  The address vector includes a number of parameters beyond just 
the GID and LID.   The endnode should examine the service being established 
and consult whatever policies service management that exists (either local 
or remote) to determine the appropriate vector and from that derive the LRH 
/ GRH fields to use.

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Mon Jan 21 21:35:01 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA01040
	for <ipoverib-archive@odin.ietf.org>; Mon, 21 Jan 2002 21:35:01 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id VAA08014
	for ipoverib-archive@odin.ietf.org; Mon, 21 Jan 2002 21:35:04 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA07886;
	Mon, 21 Jan 2002 21:32:44 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA07695
	for <ipoverib@ns.ietf.org>; Mon, 21 Jan 2002 21:32:34 -0500 (EST)
Received: from palrel11.hp.com (palrel11.hp.com [156.153.255.246])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA00167
	for <ipoverib@ietf.org>; Mon, 21 Jan 2002 21:32:25 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel11.hp.com (Postfix) with ESMTP id 0F93EE00EC0
	for <ipoverib@ietf.org>; Mon, 21 Jan 2002 18:31:59 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id SAA08022
	for <ipoverib@ietf.org>; Mon, 21 Jan 2002 18:31:58 -0800 (PST)
Message-Id: <5.1.0.14.2.20020121124841.02a35830@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Mon, 21 Jan 2002 12:51:46 -0800
To: <ipoverib@ietf.org>
From: Michael Krause <krause@cup.hp.com>
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
In-Reply-To: <2E33960095B58E40A4D3345AB9F65EC102261870@win-msg-01.wingro
 up.windeploy.ntdev.microsoft.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

At 12:16 PM 1/17/2002 -0800, Arvind Murching wrote:
>I'm not an IB expert, but I would think that -if- we decided to use
>gratuitious ARP to notify remote nodes of changes, then those remote
>nodes would consult with the SA and re-create their address handles
>using the indicated GID.

It is an implementation choice.  However, one should note that for 
Ethernet, this notification method is often used and for IB to use another 
mechanism would require a change in the endnode's implementation.  Given 
many have a strong desire to not change the upper layer / services 
implementations, we would most likely use such a mechanism.  Depending upon 
the amount of cached objects maintained for each service, the SA may not 
need to be consulted for these subsequent updates though it would be for 
the initial set-up.  It really is implementation specific.

>"Optimized/simplified" implementations would pick up new SLIDs from the
>headers of these gratuitous ARP packets.

As noted by Paul, one cannot trust this to always work.  One would have to 
set up all of the routes between all endnodes using IP to allow the LID / 
SL pair to always work.

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Tue Jan 22 17:53:25 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA13789
	for <ipoverib-archive@odin.ietf.org>; Tue, 22 Jan 2002 17:53:16 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id RAA16664
	for ipoverib-archive@odin.ietf.org; Tue, 22 Jan 2002 17:53:19 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA16589;
	Tue, 22 Jan 2002 17:52:19 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA16563
	for <ipoverib@ns.ietf.org>; Tue, 22 Jan 2002 17:52:17 -0500 (EST)
Received: from mail6.microsoft.com (mail6.microsoft.com [131.107.3.126])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA13751
	for <ipoverib@ietf.org>; Tue, 22 Jan 2002 17:52:14 -0500 (EST)
Received: from inet-vrs-06.redmond.corp.microsoft.com ([157.54.6.201]) by mail6.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Tue, 22 Jan 2002 14:51:46 -0800
Received: from 157.54.8.155 by inet-vrs-06.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Tue, 22 Jan 2002 14:51:44 -0800
Received: from red-imc-04.redmond.corp.microsoft.com ([157.54.2.168]) by inet-hub-04.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Tue, 22 Jan 2002 14:51:46 -0800
Received: from win-imc-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.39]) by red-imc-04.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Tue, 22 Jan 2002 14:51:45 -0800
Received: from win-msg-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.52]) by win-imc-01.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3588.0);
	 Tue, 22 Jan 2002 14:49:34 -0800
X-MimeOLE: Produced By Microsoft Exchange V6.0.6132.0
Content-Class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Subject: RE: [Ipoverib] RE: MCGroup Attach Failure --- IPoIB link and multicast draft -
Date: Tue, 22 Jan 2002 14:49:33 -0800
Message-ID: <2E33960095B58E40A4D3345AB9F65EC10285F652@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
Thread-Topic: [Ipoverib] RE: MCGroup Attach Failure --- IPoIB link and multicast draft -
Thread-Index: AcGi7ME850PBPdnsR3OReC4gJuQWygAeYU8w
From: "Aditya Dube" <adube@windows.microsoft.com>
To: "Michael Krause" <krause@cup.hp.com>, <ipoverib@ietf.org>
X-OriginalArrivalTime: 22 Jan 2002 22:49:34.0201 (UTC) FILETIME=[0FB53E90:01C1A397]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id RAA16564
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit


I agree that this is an edge case, but it is one that should be
addressed by our IP over Infiniband multicast proposal.

As a result of the HCA constraint and the IP over Infiniband - MCast GID
mapping scheme, we have limited the number of IP MCast addresses an
Interface can listen on. Ethernet has no such limitation.

On another note, the current multicast draft should say that that the
current scheme has this limitation. 

Aditya Dube
-----Original Message-----
From: Michael Krause [mailto:krause@cup.hp.com] 
Sent: Monday, January 21, 2002 10:40 AM
To: ipoverib@ietf.org
Subject: Re: [Ipoverib] RE: MCGroup Attach Failure --- IPoIB link and
multicast draft -

At 09:54 PM 1/20/2002 -0800, Aditya Dube wrote:
>Hello Jerry
>
>I looked into this. An HCA can only join a limited number of MCGroups.
>This limit is specified in the "Query HCA" Verb 11.2.1.2 (on pg 452).
If
>this number is exceeded then the "Attach QP to MCGroup" verb will fail
>(11.3.1 page 496).
>
>Whenever this number is exceeded the HCA will not be able to listen to
>any MCast traffic on a new MCast GID because the sender of the data
will
>still continue to send data on these MCast GIDs unaware that the
>recipient is unable to get the data.

There is no promiscuous mode within IB that would allow all joins to
always 
succeed. This stems in part from allowing SMs to layout the any logical 
routing topology that they desire.  In addition, HCAs are restricted to
how 
many MGIDs they can multiplex on a given port which is an 
implementation-specific issue.  For IP over IB, I do not view any of
these 
issues as a problem or requiring additional work from the IBTA as these
are 
not architectural limitations.  The IP over IB effort should simply
require 
that for each partition that the endnode wants to provide IP service,
there 
should be at a minimum a corresponding set of multicast resources 
available.  HCA vendors will then rely upon market input to guide them
in 
updating their requirements.

In May 2001, HP made available a set of IB HCA, TCA, Switch, and Router 
requirements and motivations to help vendors decide what features and 
associated resources to implement.  This material is available to all
and 
within one will find the multicast requirements.  The minimum acceptable

for a HCA was 512 per port; optimal was 2048.  Additional values were 
provided for TCA, Switch, and Routers.  In discussions with many IHVs
who 
have received this material, it is fair to say that the majority will 
implement generally more than the minimums and quite often near the 
optimals with some exceeding even these.  As such, the market will be
able 
to support IP over IB requirements for the vast majority of the customer

requirements thereby eliminating the need for IP over IB to do anything 
special to recover from such a group join problem.

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Tue Jan 22 18:15:59 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA14168
	for <ipoverib-archive@odin.ietf.org>; Tue, 22 Jan 2002 18:15:55 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id SAA17489
	for ipoverib-archive@odin.ietf.org; Tue, 22 Jan 2002 18:15:58 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA17465;
	Tue, 22 Jan 2002 18:15:29 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA17440
	for <ipoverib@ns.ietf.org>; Tue, 22 Jan 2002 18:15:28 -0500 (EST)
Received: from e33.co.us.ibm.com (e33.co.us.ibm.com [32.97.110.131])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA14162
	for <ipoverib@ietf.org>; Tue, 22 Jan 2002 18:15:24 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e33.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id SAA16260;
	Tue, 22 Jan 2002 18:12:24 -0500
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0MNFRs22102;
	Tue, 22 Jan 2002 16:15:27 -0700
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id g0MNFQg07643;
	Tue, 22 Jan 2002 15:15:26 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0MNFPV01413;
	Tue, 22 Jan 2002 15:15:25 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201222315.g0MNFPV01413@eng4.beaverton.ibm.com>
Subject: Re: [ipoverib] reasons for not using LID ?
To: krause@cup.hp.com (Michael Krause)
Date: Tue, 22 Jan 2002 15:15:24 -0800 (PST)
Cc: ipoverib@ietf.org
In-Reply-To: <5.1.0.14.2.20020121123302.02a24e88@esalpha2.cup.hp.com> from "Michael Krause" at Jan 21, 2002 11:40:34 AM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit


>>Bill, Yaron,
>>
>>         There are two situations
>>                 - proxy-ARP for a node on another subnet
>>                 - span IP subnets across IB subnets
>>
>>1.      proxy-ARP for a node on another subnet works very easily.
>>         You return the LID of the local IB router and the GID of the
>>         remote IB node. Note this is possible with the current
>>         specification. If only the GID is included excluding the LID
>>         this is not possible since you can't lookup the remote node's
>>         GID with the SA. You need to include both. [ SA to SA
>>         coordination within an IB subnet isn't specified what of
>>         across the subnet. ]
>>
>>         Note that the ARP packet is fielded by the IPoIB device
>>         logically/physically associated with the IB router. The ARP
>>         packet itself does not travel beyond the IB subnet.
>
>Within the requester's IB subnet, there will exist at least one endnode
>that will respond on behalf of any request that targets an endnode that
>exists on another IB or IP subnet.   The responder will work with the SA to
>
>determine the IB router; an IP router would respond on behalf of its
>responsibility "cloud" and it too would work with the SA to determine the
>IB LRH values to use to establish appropriate communication between the
>requester and the IP router.  In both cases, the responder is responsible
>for determining the appropriate LRH / GRH (where applicable) header
>parameters to establish the communication channel.
>

Is the solution then that the remote subnet's GIDs not only have to be
known to the IPoIB device but also recorded in the SA/SM or at least
the remote GID-prefix is recorded with the SA/SM ?

My note above describes a simpler solution that merely takes the LID and
the GID from the ARP reply and the SA/SM need not be involved. If the
implementation prefers to get the SL/path rate from the SA (for the IB
router) it gets them based on the LID received (which is that of the
IB router). The solution is much simpler. Any IPoIB system on the
subnet can answer for the remote node and the packet will reach
through the router with no setup required.

The suggestion that the LID from the LRH be used works efficiently but
only if the responder is the IB router itself. If another device
(IPoIB device) answers for it then it will, as has been suggested,
have to indicate that the LID in the LRH is not the right one and the
solution will default to the more complex 'record non-local GIDs in
the SA'.

Is there a query/set of queries that returns the routers in the subnet and
that tells which routers to choose ? Also, if a non-local GID prefix
is used can one determine the router to use ?

If so then the solution 'use LID in LRH but only if ARP/ND didn't mark
it invalid' can work. If such a 'non-local GID' solution will not work
now because the routing infrastructure/SA records that are needed
don't exist and one has to await IB routing then the invalid flag is
not needed either. Or, if in general such an across the subnet
proxy-ARP solution can be 'ignored' then too we don't need the invalid
flag. Using the LID from LRH is sufficient.

As far as I know 'non-local GID/associated router(s)' records don't
exist and have to be invented. I would rather that we have a solution
that works now and so support LID+GID. However, if we 'ignore' this
one case of proxy-ARP across IB-subnet then LID from LRH will do.

Question 1: Can the kind of proxy-ARP described here be implemented if
the LID is not included in the ARP/ND ? i.e. do the above queries and
records exist in the SA database? If not, we either need to say that
such a solution is not possible or define a solution.

Question 2: Is the LRH:LID always available for all UD communication ? Is
it equally applicable to receiving UD packets on xCAs, IB routers and IB
switches (port 0 probably) ? If not, then when is it not applicable. (I do
expect it to be available in all cases but want to be sure).



>>2.      For IP subnets to span IB subnets there is too much
>>         speculation and no justification. First of all we don't know
>>         why we need it. We don't know when it will be possible. We
>>         don't know if it will be possible. It is a complicated issue -
>>         too many things have to be updated in IB architecutre.
>
>The justification is that vendors and customers believe it is the right
>thing to do and there is both technical and economic feasibility to achieve
>
>this objective.  Unless there is strong consensus to not support this type
>of functionality, the workgroup must take steps to enable it or at least
>not preclude this from being implemented at some future date once the IBTA
>has provided the appropriate specification updates.
>

The reason I am asking for such justificaiton because it is being used
to argue for various solutions. It is better to know what is the
reason for wanting it and then we can decide if we still want to go
ahead or not. Till now no real reason has been specified. We create IB
subnets for isolation, management etc. then why join them back again
in IPoIB subnets. Wouldn't IP subnetting serve the purpose anyway ?

Vivek

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Tue Jan 22 19:03:40 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA14893
	for <ipoverib-archive@odin.ietf.org>; Tue, 22 Jan 2002 19:03:40 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id TAA19100
	for ipoverib-archive@odin.ietf.org; Tue, 22 Jan 2002 19:03:42 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA19083;
	Tue, 22 Jan 2002 19:02:14 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA19053
	for <ipoverib@ns.ietf.org>; Tue, 22 Jan 2002 19:02:12 -0500 (EST)
Received: from mxout1.netvision.net.il (mxout1.netvision.net.il [194.90.9.20])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA14884
	for <ipoverib@ietf.org>; Tue, 22 Jan 2002 19:02:08 -0500 (EST)
Received: from vaio505he ([139.92.208.145]) by mxout1.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GQD00GFF6QO7Z@mxout1.netvision.net.il> for ipoverib@ietf.org;
 Wed, 23 Jan 2002 02:01:39 +0200 (IST)
Date: Wed, 23 Jan 2002 02:02:18 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: RE: [ipoverib] reasons for not using LID ?
In-reply-to: <200201222315.g0MNFPV01413@eng4.beaverton.ibm.com>
To: "'Vivek Kashyap'" <kashyapv@us.ibm.com>,
        "'Michael Krause'" <krause@cup.hp.com>
Cc: ipoverib@ietf.org
Reply-to: yaronh@voltaire.com
Message-id: <001101c1a3a1$3a034700$91d05c8b@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: text/plain; charset=US-ASCII
Content-transfer-encoding: 7BIT
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7BIT

Vivek,

As one of the guys suggesting the Proxy ARP approach, I could answer your
first question
Yes, the LID in the LRH will do, no need for LID in the ARP
I just need the Valid bit mainly for efficiency in multi subnet
configurations

I also assume regarding your second question, LID is always there in UD

I believe that we could remove the LID from the ARP, and not harm any
possible configuration
And as one of the supporters for that approach in the pass, I double checked
it before making the decision

So, lets agree on this subject and move forward with the next topic

Yaron

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Vivek Kashyap
Sent: Wednesday, January 23, 2002 1:15 AM
To: Michael Krause
Cc: ipoverib@ietf.org
Subject: Re: [ipoverib] reasons for not using LID ?


>>Bill, Yaron,
>>
>>         There are two situations
>>                 - proxy-ARP for a node on another subnet
>>                 - span IP subnets across IB subnets
>>
>>1.      proxy-ARP for a node on another subnet works very easily.
>>         You return the LID of the local IB router and the GID of the
>>         remote IB node. Note this is possible with the current
>>         specification. If only the GID is included excluding the LID
>>         this is not possible since you can't lookup the remote node's
>>         GID with the SA. You need to include both. [ SA to SA
>>         coordination within an IB subnet isn't specified what of
>>         across the subnet. ]
>>
>>         Note that the ARP packet is fielded by the IPoIB device
>>         logically/physically associated with the IB router. The ARP
>>         packet itself does not travel beyond the IB subnet.
>
>Within the requester's IB subnet, there will exist at least one endnode
>that will respond on behalf of any request that targets an endnode that
>exists on another IB or IP subnet.   The responder will work with the SA to
>
>determine the IB router; an IP router would respond on behalf of its
>responsibility "cloud" and it too would work with the SA to determine the
>IB LRH values to use to establish appropriate communication between the
>requester and the IP router.  In both cases, the responder is responsible
>for determining the appropriate LRH / GRH (where applicable) header
>parameters to establish the communication channel.
>

Is the solution then that the remote subnet's GIDs not only have to be
known to the IPoIB device but also recorded in the SA/SM or at least
the remote GID-prefix is recorded with the SA/SM ?

My note above describes a simpler solution that merely takes the LID and
the GID from the ARP reply and the SA/SM need not be involved. If the
implementation prefers to get the SL/path rate from the SA (for the IB
router) it gets them based on the LID received (which is that of the
IB router). The solution is much simpler. Any IPoIB system on the
subnet can answer for the remote node and the packet will reach
through the router with no setup required.

The suggestion that the LID from the LRH be used works efficiently but
only if the responder is the IB router itself. If another device
(IPoIB device) answers for it then it will, as has been suggested,
have to indicate that the LID in the LRH is not the right one and the
solution will default to the more complex 'record non-local GIDs in
the SA'.

Is there a query/set of queries that returns the routers in the subnet and
that tells which routers to choose ? Also, if a non-local GID prefix
is used can one determine the router to use ?

If so then the solution 'use LID in LRH but only if ARP/ND didn't mark
it invalid' can work. If such a 'non-local GID' solution will not work
now because the routing infrastructure/SA records that are needed
don't exist and one has to await IB routing then the invalid flag is
not needed either. Or, if in general such an across the subnet
proxy-ARP solution can be 'ignored' then too we don't need the invalid
flag. Using the LID from LRH is sufficient.

As far as I know 'non-local GID/associated router(s)' records don't
exist and have to be invented. I would rather that we have a solution
that works now and so support LID+GID. However, if we 'ignore' this
one case of proxy-ARP across IB-subnet then LID from LRH will do.

Question 1: Can the kind of proxy-ARP described here be implemented if
the LID is not included in the ARP/ND ? i.e. do the above queries and
records exist in the SA database? If not, we either need to say that
such a solution is not possible or define a solution.

Question 2: Is the LRH:LID always available for all UD communication ? Is
it equally applicable to receiving UD packets on xCAs, IB routers and IB
switches (port 0 probably) ? If not, then when is it not applicable. (I do
expect it to be available in all cases but want to be sure).



>>2.      For IP subnets to span IB subnets there is too much
>>         speculation and no justification. First of all we don't know
>>         why we need it. We don't know when it will be possible. We
>>         don't know if it will be possible. It is a complicated issue -
>>         too many things have to be updated in IB architecutre.
>
>The justification is that vendors and customers believe it is the right
>thing to do and there is both technical and economic feasibility to achieve
>
>this objective.  Unless there is strong consensus to not support this type
>of functionality, the workgroup must take steps to enable it or at least
>not preclude this from being implemented at some future date once the IBTA
>has provided the appropriate specification updates.
>

The reason I am asking for such justificaiton because it is being used
to argue for various solutions. It is better to know what is the
reason for wanting it and then we can decide if we still want to go
ahead or not. Till now no real reason has been specified. We create IB
subnets for isolation, management etc. then why join them back again
in IPoIB subnets. Wouldn't IP subnetting serve the purpose anyway ?

Vivek

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Tue Jan 22 21:33:48 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA17651
	for <ipoverib-archive@odin.ietf.org>; Tue, 22 Jan 2002 21:33:47 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id VAA24151
	for ipoverib-archive@odin.ietf.org; Tue, 22 Jan 2002 21:33:50 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA24103;
	Tue, 22 Jan 2002 21:31:38 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA24070
	for <ipoverib@ns.ietf.org>; Tue, 22 Jan 2002 21:31:36 -0500 (EST)
Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA17595
	for <ipoverib@ietf.org>; Tue, 22 Jan 2002 21:31:32 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id UAA24142;
	Tue, 22 Jan 2002 20:27:12 -0600
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0N2VXF209656;
	Tue, 22 Jan 2002 21:31:34 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0N2VVV24558;
	Tue, 22 Jan 2002 18:31:31 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0N2VUn21118;
	Tue, 22 Jan 2002 18:31:30 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201230231.g0N2VUn21118@eng4.beaverton.ibm.com>
Subject: Re: [ipoverib] reasons for not using LID ?
To: yaronh@voltaire.com
Date: Tue, 22 Jan 2002 18:31:29 -0800 (PST)
Cc: kashyapv@us.ibm.com ('Vivek Kashyap'),
        krause@cup.hp.com ('Michael Krause'), ipoverib@ietf.org
In-Reply-To: <001101c1a3a1$3a034700$91d05c8b@vaio505he> from "Yaron Haviv" at Jan 23, 2002 01:02:18 AM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit


> 
> Vivek,
> 
> As one of the guys suggesting the Proxy ARP approach, I could answer your
> first question
> Yes, the LID in the LRH will do, no need for LID in the ARP

Yes, it will work. No issues with it generally.

-----------

But proxy-ARP across the IB subnet has been on the desired list of
some people - you have mentioned it too. What I am pointing out is
that such proxy-ARP won't work. This is because there is no way to
determine the IB router from the GID received. Since that is the case
anyway, the invalid bit does not help.

So I am saying is either we say that such setups will have to wait
till IB routing comes up or if it is important suggest a solution.

[The inclusion of LID in the ARP/ND provided the solution but we are
dropping that option].

Vivek

> I just need the Valid bit mainly for efficiency in multi subnet
> configurations
> 
> I also assume regarding your second question, LID is always there in UD
> 
> I believe that we could remove the LID from the ARP, and not harm any
> possible configuration
> And as one of the supporters for that approach in the pass, I double checked
> it before making the decision
> 
> So, lets agree on this subject and move forward with the next topic
> 
> Yaron
> 
> -----Original Message-----
> From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
> Vivek Kashyap
> Sent: Wednesday, January 23, 2002 1:15 AM
> To: Michael Krause
> Cc: ipoverib@ietf.org
> Subject: Re: [ipoverib] reasons for not using LID ?
> 
> 
> >>Bill, Yaron,
> >>
> >>         There are two situations
> >>                 - proxy-ARP for a node on another subnet
> >>                 - span IP subnets across IB subnets
> >>
> >>1.      proxy-ARP for a node on another subnet works very easily.
> >>         You return the LID of the local IB router and the GID of the
> >>         remote IB node. Note this is possible with the current
> >>         specification. If only the GID is included excluding the LID
> >>         this is not possible since you can't lookup the remote node's
> >>         GID with the SA. You need to include both. [ SA to SA
> >>         coordination within an IB subnet isn't specified what of
> >>         across the subnet. ]
> >>
> >>         Note that the ARP packet is fielded by the IPoIB device
> >>         logically/physically associated with the IB router. The ARP
> >>         packet itself does not travel beyond the IB subnet.
> >
> >Within the requester's IB subnet, there will exist at least one endnode
> >that will respond on behalf of any request that targets an endnode that
> >exists on another IB or IP subnet.   The responder will work with the SA to
> >
> >determine the IB router; an IP router would respond on behalf of its
> >responsibility "cloud" and it too would work with the SA to determine the
> >IB LRH values to use to establish appropriate communication between the
> >requester and the IP router.  In both cases, the responder is responsible
> >for determining the appropriate LRH / GRH (where applicable) header
> >parameters to establish the communication channel.
> >
> 
> Is the solution then that the remote subnet's GIDs not only have to be
> known to the IPoIB device but also recorded in the SA/SM or at least
> the remote GID-prefix is recorded with the SA/SM ?
> 
> My note above describes a simpler solution that merely takes the LID and
> the GID from the ARP reply and the SA/SM need not be involved. If the
> implementation prefers to get the SL/path rate from the SA (for the IB
> router) it gets them based on the LID received (which is that of the
> IB router). The solution is much simpler. Any IPoIB system on the
> subnet can answer for the remote node and the packet will reach
> through the router with no setup required.
> 
> The suggestion that the LID from the LRH be used works efficiently but
> only if the responder is the IB router itself. If another device
> (IPoIB device) answers for it then it will, as has been suggested,
> have to indicate that the LID in the LRH is not the right one and the
> solution will default to the more complex 'record non-local GIDs in
> the SA'.
> 
> Is there a query/set of queries that returns the routers in the subnet and
> that tells which routers to choose ? Also, if a non-local GID prefix
> is used can one determine the router to use ?
> 
> If so then the solution 'use LID in LRH but only if ARP/ND didn't mark
> it invalid' can work. If such a 'non-local GID' solution will not work
> now because the routing infrastructure/SA records that are needed
> don't exist and one has to await IB routing then the invalid flag is
> not needed either. Or, if in general such an across the subnet
> proxy-ARP solution can be 'ignored' then too we don't need the invalid
> flag. Using the LID from LRH is sufficient.
> 
> As far as I know 'non-local GID/associated router(s)' records don't
> exist and have to be invented. I would rather that we have a solution
> that works now and so support LID+GID. However, if we 'ignore' this
> one case of proxy-ARP across IB-subnet then LID from LRH will do.
> 
> Question 1: Can the kind of proxy-ARP described here be implemented if
> the LID is not included in the ARP/ND ? i.e. do the above queries and
> records exist in the SA database? If not, we either need to say that
> such a solution is not possible or define a solution.
> 
> Question 2: Is the LRH:LID always available for all UD communication ? Is
> it equally applicable to receiving UD packets on xCAs, IB routers and IB
> switches (port 0 probably) ? If not, then when is it not applicable. (I do
> expect it to be available in all cases but want to be sure).
> 
> 
> 
> >>2.      For IP subnets to span IB subnets there is too much
> >>         speculation and no justification. First of all we don't know
> >>         why we need it. We don't know when it will be possible. We
> >>         don't know if it will be possible. It is a complicated issue -
> >>         too many things have to be updated in IB architecutre.
> >
> >The justification is that vendors and customers believe it is the right
> >thing to do and there is both technical and economic feasibility to achieve
> >
> >this objective.  Unless there is strong consensus to not support this type
> >of functionality, the workgroup must take steps to enable it or at least
> >not preclude this from being implemented at some future date once the IBTA
> >has provided the appropriate specification updates.
> >
> 
> The reason I am asking for such justificaiton because it is being used
> to argue for various solutions. It is better to know what is the
> reason for wanting it and then we can decide if we still want to go
> ahead or not. Till now no real reason has been specified. We create IB
> subnets for isolation, management etc. then why join them back again
> in IPoIB subnets. Wouldn't IP subnetting serve the purpose anyway ?
> 
> Vivek
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)
-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Tue Jan 22 21:53:57 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA18940
	for <ipoverib-archive@odin.ietf.org>; Tue, 22 Jan 2002 21:53:57 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id VAA25276
	for ipoverib-archive@odin.ietf.org; Tue, 22 Jan 2002 21:54:01 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA25260;
	Tue, 22 Jan 2002 21:53:28 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA25230
	for <ipoverib@ns.ietf.org>; Tue, 22 Jan 2002 21:53:27 -0500 (EST)
Received: from fgwmail6.fujitsu.co.jp (fgwmail6.fujitsu.co.jp [192.51.44.36])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA18915
	for <ipoverib@ietf.org>; Tue, 22 Jan 2002 21:53:22 -0500 (EST)
Received: from m7.gw.fujitsu.co.jp by fgwmail6.fujitsu.co.jp (8.9.3/3.7W-MX0201-Fujitsu Gateway)
	id LAA05269; Wed, 23 Jan 2002 11:52:53 +0900 (JST)
	(envelope-from imai@giulia.ds.oaks.cs.fujitsu.co.jp)
Received: from dm2.oaks.cs.fujitsu.co.jp by m7.gw.fujitsu.co.jp (8.9.3/3.7W-0112-Fujitsu Domain Master)
	id LAA19267; Wed, 23 Jan 2002 11:52:52 +0900
	(envelope-from imai@giulia.ds.oaks.cs.fujitsu.co.jp)
Received: from lovewars.ds.oaks.cs.fujitsu.co.jp by dm2.oaks.cs.fujitsu.co.jp (8.7.4+2.6Wbeta6/3.4W4-MXM960524)
	id KAA27138; Wed, 23 Jan 2002 10:44:56 +0900 (JST)
Received: from giulia.ds.oaks.cs.fujitsu.co.jp by lovewars.ds.oaks.cs.fujitsu.co.jp (5.0/6.4J.6)
	id AA17335; Wed, 23 Jan 2002 12:13:24 --900
Received: from giulia (imai@localhost)
	by giulia.ds.oaks.cs.fujitsu.co.jp (8.10.2+Sun/8.10.2) with ESMTP id g0N2qlc26384;
	Wed, 23 Jan 2002 11:52:47 +0900 (JST)
Message-Id: <200201230252.g0N2qlc26384@giulia.ds.oaks.cs.fujitsu.co.jp>
To: Vivek Kashyap <kashyapv@us.ibm.com>
Cc: ipoverib@ietf.org
From: akihiko_imai@cs.fujitsu.co.jp
Subject: Re: [Ipoverib] encapsulation issues 
In-Reply-To: Your message of "Thu, 17 Jan 2002 20:45:42 JST."
             <200201180445.g0I4jhq23173@eng4.beaverton.ibm.com> 
Date: Wed, 23 Jan 2002 11:52:47 +0900
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Vivek,

Hello.
We have been catching up almost every ipoib mail-thread and thinking "gaps"
 between our ipoib driver(incl.its related modules) implementations and
 latest ipoib mail-thread arguments.

I hope you read my comments about ethertype in payload.

>From: Vivek Kashyap <kashyapv@us.ibm.com>
>Message-Id: <200201180445.g0I4jhq23173@eng4.beaverton.ibm.com>
>To: ipoverib@ietf.org
>Date: Thu, 17 Jan 2002 20:45:42 -0800 (PST)
>> > 2. 	I believe based on the recent set of mails that the consensus no
>> >         longer favours inclusion of the 'ethertype'. Is there a
>> >         disagreement ?
>> 
>> There were many that said that an Ethertype was desirable and useful in
>> their implementations.  Your counter argument was that it was not needed
>> because it could be inferred through other means so why add an additional
>> mechanism.  If you would allow us to use the same logic used in #1 above
>> ("the architecture not forcing a limited solution on any set of
>> aspirations"), then I would like an Ethertype defined as part of the
>> payload.  As noted earlier, the inclusion of an Ethertype does not handicap
>> or restrict any solution.
>> 
>> Roy
>
>        I know that many found it desirable in the beginning but it
>        appeared to me that people had changed their mind.
>
>        The reason ethertype is unnecessary is that it doesn't buy us
>        the solution it is supposed to due to the requirement that
>        every other IB protocol follows the same rules. IPoIB has no
>        control over that.
>
>Vivek

We also need EtherType field in IB packet payload.

Let me explain the reasons
1. Our ipoib datalink driver binds multiple protocols (IPv4,ARP,IPv6)
 to a single QP (per ipoib datalink interface),
 and No upper layer (which dispatch packets according to its protocol kind
 for received packets) exists above us(ipoib datalink driver).
 i.e. If IPv4 stack would receive ARP packets, IPv4 just discard them
    and vice versa.
2. In our some environments, we provide another upper datalink modules
 (which exist between IPv4/ARP/IPv6 stacks and ipoib datalink driver).
 They want to use ANOTHER ethertypes other than IPv4/ARP/IPv6. They are
 upper layers using ipoib but not-IPv4/ARP/IPv6.
 It should be possible that other protocols packets than IPv4/ARP/IPv6 can be
  carried over ipoib.

Thanks,
------------------------------------------------------------------------------
Akihiko Imai/Development Dept.5, Infrastructure Systems Division, Fujitsu Ltd.
  e-mail: akihiko_imai@cs.fujitsu.co.jp  TEL: +81-044-754-3456 (ext)7113-8706
  address: 4-4-1 Kamiodanaka Nakahara-ku, Kawasaki 211-8588, Japan


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 23 08:49:56 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA08156
	for <ipoverib-archive@odin.ietf.org>; Wed, 23 Jan 2002 08:49:52 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id IAA26183
	for ipoverib-archive@odin.ietf.org; Wed, 23 Jan 2002 08:49:54 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id IAA26111;
	Wed, 23 Jan 2002 08:48:22 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id IAA26079
	for <ipoverib@optimus.ietf.org>; Wed, 23 Jan 2002 08:48:18 -0500 (EST)
Received: from vieo.com (vieo.com [216.30.79.131])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA08063
	for <ipoverib@ietf.org>; Wed, 23 Jan 2002 08:48:14 -0500 (EST)
Received: (from root@localhost)
	by vieo.com (8.11.2/8.11.2) id g0NDljF81035
	for ipoverib@ietf.org; Wed, 23 Jan 2002 07:47:45 -0600 (CST)
	(envelope-from rronald@VIEO.com)
Received: from Fishhead (dhcp-10-2-15-164.vieo.com [10.2.15.164])
	by vieo.com (8.11.2/8.11.2) with SMTP id g0NDlic80995;
	Wed, 23 Jan 2002 07:47:44 -0600 (CST)
	(envelope-from rronald@VIEO.com)
Message-ID: <004401c1a414$806071f0$a40f020a@Fishhead>
From: "Roger Ronald" <rronald@VIEO.com>
To: <ipoverib@ietf.org>, "Michael Krause" <krause@cup.hp.com>
References: <200201170927.g0H9RBg12021@eng4.beaverton.ibm.com> <5.1.0.14.2.20020121124553.028e2350@esalpha2.cup.hp.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
Date: Wed, 23 Jan 2002 07:47:29 -0600
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 5.50.4133.2400
X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400
X-scanner: scanned by Inflex 0.1.5c+ on vieo.com
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

I concur with your analysis. While LID assignments will change
infrequently, the proper LID/SL may change more often due
to APM.

RR
----- Original Message ----- 
From: "Michael Krause" <krause@cup.hp.com>
To: <ipoverib@ietf.org>
Sent: Monday, January 21, 2002 2:48 PM
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?


> At 02:13 PM 1/17/2002 -0600, Roger Ronald wrote:
> >I think it's important to note that changes in LID should be infrequent.
> >Very infrequent.
> >
> >VIEO is one of a small number of companies that has a SM at this time.
> >We plan to maintain LID assignments whenever possible. That means
> >across reboots, across power fails and across subnet merges (when possible).
> >
> >There are corner cases where LID may need to change, just like IP
> >addresses occasionaly need to change. But these are VERY odd
> >events...like the first time merge of fabrics. They won't happen
> >hourly, daily, monthly..... They may happen once or twice in a
> >fabric. Ever (unless the admin wants to make changes for the heck of it).
> 
> While the LID range for a given port may not change often, the LID / SL 
> appropriate for a given path between two endnodes may change for a variety 
> of reasons based on workload objectives / QoS requirements within the 
> fabric for a given service.  The IB APM mechanisms may be used to update 
> these path selections without informing IP of the change (an implementation 
> option).  Thus, the rate of LID change may not be as low as you contend but 
> it will also not be as high as others contend making the performance issues 
> raised perhaps moot.
> 
> Mike
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 
> 


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 23 09:37:10 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA10456
	for <ipoverib-archive@odin.ietf.org>; Wed, 23 Jan 2002 09:37:10 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id JAA28476
	for ipoverib-archive@odin.ietf.org; Wed, 23 Jan 2002 09:37:11 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id JAA28420;
	Wed, 23 Jan 2002 09:36:33 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id JAA28364
	for <ipoverib@optimus.ietf.org>; Wed, 23 Jan 2002 09:36:30 -0500 (EST)
Received: from ztxmail04.ztx.compaq.com (ztxmail04.ztx.compaq.com [161.114.1.208])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA10410
	for <ipoverib@ietf.org>; Wed, 23 Jan 2002 09:36:28 -0500 (EST)
Received: by ztxmail04.ztx.compaq.com (Postfix, from userid 12345)
	id 96CA125C9; Wed, 23 Jan 2002 08:35:44 -0600 (CST)
Received: from cceexg11.americas.cpqcorp.net (cceexg11.americas.cpqcorp.net [16.110.250.125])
	by ztxmail04.ztx.compaq.com (Postfix) with ESMTP
	id 6FE8E2782; Wed, 23 Jan 2002 08:35:44 -0600 (CST)
Received: from cceexc18.americas.cpqcorp.net ([16.110.250.64]) by cceexg11.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966);
	 Wed, 23 Jan 2002 08:35:43 -0600
X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="US-ASCII"
Subject: RE: [ipoverib] reasons for not using LID ?
Date: Wed, 23 Jan 2002 08:35:43 -0600
Message-ID: <4D027986353D1341ADA4F2A4F8A170760E317C@cceexc18.americas.cpqcorp.net>
Thread-Topic: [ipoverib] reasons for not using LID ?
Thread-Index: AcGjmrdgdIZ9DKB7QqWqUCOwoDvLTwAe0ckw
From: "Culley, Paul" <Paul.Culley@compaq.com>
To: "Vivek Kashyap" <kashyapv@us.ibm.com>,
        "Michael Krause" <krause@cup.hp.com>
Cc: <ipoverib@ietf.org>
X-OriginalArrivalTime: 23 Jan 2002 14:35:43.0873 (UTC) FILETIME=[3D11BF10:01C1A41B]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id JAA28365
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

embedded...

Paul R. Culley
Compaq Sr. Fellow
281-514-5543
P.S. are we having fun yet?

-----Original Message-----
From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
Sent: Tuesday, January 22, 2002 5:15 PM
To: Michael Krause
Cc: ipoverib@ietf.org
Subject: Re: [ipoverib] reasons for not using LID ?



>>Bill, Yaron,
>>
>>         There are two situations
>>                 - proxy-ARP for a node on another subnet
>>                 - span IP subnets across IB subnets
>>
>>1.      proxy-ARP for a node on another subnet works very easily.
>>         You return the LID of the local IB router and the GID of the
>>         remote IB node. Note this is possible with the current
>>         specification. If only the GID is included excluding the LID
>>         this is not possible since you can't lookup the remote node's
>>         GID with the SA. You need to include both. [ SA to SA
>>         coordination within an IB subnet isn't specified what of
>>         across the subnet. ]
>>
>>         Note that the ARP packet is fielded by the IPoIB device
>>         logically/physically associated with the IB router. The ARP
>>         packet itself does not travel beyond the IB subnet.
>
>Within the requester's IB subnet, there will exist at least one endnode
>that will respond on behalf of any request that targets an endnode that
>exists on another IB or IP subnet.   The responder will work with the
SA to
>
>determine the IB router; an IP router would respond on behalf of its
>responsibility "cloud" and it too would work with the SA to determine
the
>IB LRH values to use to establish appropriate communication between the
>requester and the IP router.  In both cases, the responder is
responsible
>for determining the appropriate LRH / GRH (where applicable) header
>parameters to establish the communication channel.
>

Is the solution then that the remote subnet's GIDs not only have to be
known to the IPoIB device but also recorded in the SA/SM or at least
the remote GID-prefix is recorded with the SA/SM ?
<prc>
 Yes, one way or another the SA/SM must know at least which router port
the GID prefix needs to route to.  We may make it a requirement (to
simplify SA's) that the route cloud be able to deal with anything
arriving at any router port on the subnet; then a simple SA need only
route to a single router port.  More complex SA's should still be able
to chose an optimum port. 
<endprc>

My note above describes a simpler solution that merely takes the LID and
the GID from the ARP reply and the SA/SM need not be involved. If the
implementation prefers to get the SL/path rate from the SA (for the IB
router) it gets them based on the LID received (which is that of the
IB router). The solution is much simpler. Any IPoIB system on the
subnet can answer for the remote node and the packet will reach
through the router with no setup required.
<prc>
By only looking up the path to the router, the SA can only report on
that path.  Since we expect to support SA lookups for the end to end
path, including the full router cloud, it will be possible to get good
information on the full path.  This is needed for backwards
compatibility with devices that don't understand anything but IB 1.x
interfaces.
<endprc>

The suggestion that the LID from the LRH be used works efficiently but
only if the responder is the IB router itself. If another device
(IPoIB device) answers for it then it will, as has been suggested,
have to indicate that the LID in the LRH is not the right one and the
solution will default to the more complex 'record non-local GIDs in
the SA'.

Is there a query/set of queries that returns the routers in the subnet
and
that tells which routers to choose ? Also, if a non-local GID prefix
is used can one determine the router to use ?
<prc>
At this time, there is no method defined for the SA to interact with the
routers, or for endnodes to get router information.  As stated before,
the current plan is to use the PathRecord query to report on the full
path, as is necessary for 1.x compatibility.  The returned path
information would include LID/SL for the appropriate router port.  When
the IB specs are updated to support routers, I expect the SA/router
interface will be included, requiring some additional support from SAs.
If you wish to submit a proposal for additional interfaces that support
the more traditional IP like methods (where endnodes must chose the
router ports for non-local work), you are free to do so.
<endprc>

If so then the solution 'use LID in LRH but only if ARP/ND didn't mark
it invalid' can work. If such a 'non-local GID' solution will not work
now because the routing infrastructure/SA records that are needed
don't exist and one has to await IB routing then the invalid flag is
not needed either. Or, if in general such an across the subnet
proxy-ARP solution can be 'ignored' then too we don't need the invalid
flag. Using the LID from LRH is sufficient.

As far as I know 'non-local GID/associated router(s)' records don't
exist and have to be invented. I would rather that we have a solution
that works now and so support LID+GID. However, if we 'ignore' this
one case of proxy-ARP across IB-subnet then LID from LRH will do.

Question 1: Can the kind of proxy-ARP described here be implemented if
the LID is not included in the ARP/ND ? i.e. do the above queries and
records exist in the SA database? If not, we either need to say that
such a solution is not possible or define a solution.
<prc> As long as we fully support the 1.x IB spec in the routed fabric,
the PathRecord query will work, so we have a solution.
<endprc>

Question 2: Is the LRH:LID always available for all UD communication ?
Is
it equally applicable to receiving UD packets on xCAs, IB routers and IB
switches (port 0 probably) ? If not, then when is it not applicable. (I
do
expect it to be available in all cases but want to be sure).
<prc> The values are in the headers for all UD packets; HCAs have an
interface for getting this information, switches should not ever need
this, routers are yet to be fully defined, and TCAs can provide the
header info if their designer's care about cross subnet support...
<endprc>


>>2.      For IP subnets to span IB subnets there is too much
>>         speculation and no justification. First of all we don't know
>>         why we need it. We don't know when it will be possible. We
>>         don't know if it will be possible. It is a complicated issue
-
>>         too many things have to be updated in IB architecutre.
>
>The justification is that vendors and customers believe it is the right
>thing to do and there is both technical and economic feasibility to
achieve
>
>this objective.  Unless there is strong consensus to not support this
type
>of functionality, the workgroup must take steps to enable it or at
least
>not preclude this from being implemented at some future date once the
IBTA
>has provided the appropriate specification updates.
>

The reason I am asking for such justificaiton because it is being used
to argue for various solutions. It is better to know what is the
reason for wanting it and then we can decide if we still want to go
ahead or not. Till now no real reason has been specified. We create IB
subnets for isolation, management etc. then why join them back again
in IPoIB subnets. Wouldn't IP subnetting serve the purpose anyway ?
<prc>
It has been stated before, but here goes again:
1) the customers want it
2) We expect that IB routers are coming, probably about the time IPoIB
really gets going...
3) We think we have a workable, if not optimum, method of supporting the
routers in mind right now (1.x PathRecord support).  So you don't really
have to wait for anything new to complete the ARP RFC.
4) The alternative for customers (to IP across IB subnets) is to install
extra IP routing functionality across any IB router; it seems that this
would slow the adoption rate of either IB routers, or IPoIB...  as well
as cost more for the customers.
5) I personally (and some others as well) would like to avoid the NEED
to return to the ietf to fix up the IPoIB RFCs when routers do become
available.  That creates too tight a tie on the RFC process to releasing
useful IB routers.  And creates chicken/egg problems if every endnode
needs to be fixed as well. That does not preclude providing some
optimizations through new RFCs as IB fabrics grow and the need for those
optimizations becomes apparent.
<endprc>

Vivek

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 23 10:52:02 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA13356
	for <ipoverib-archive@odin.ietf.org>; Wed, 23 Jan 2002 10:52:02 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id KAA01327
	for ipoverib-archive@odin.ietf.org; Wed, 23 Jan 2002 10:52:04 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id KAA01306;
	Wed, 23 Jan 2002 10:51:19 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id KAA01274
	for <ipoverib@optimus.ietf.org>; Wed, 23 Jan 2002 10:51:17 -0500 (EST)
Received: from palrel11.hp.com (palrel11.hp.com [156.153.255.246])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA13312
	for <ipoverib@ietf.org>; Wed, 23 Jan 2002 10:51:14 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel11.hp.com (Postfix) with ESMTP id 099C7E00448
	for <ipoverib@ietf.org>; Wed, 23 Jan 2002 07:50:46 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id HAA24645
	for <ipoverib@ietf.org>; Wed, 23 Jan 2002 07:50:45 -0800 (PST)
Message-Id: <5.1.0.14.2.20020123072200.02a22c68@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Wed, 23 Jan 2002 07:34:33 -0800
To: <ipoverib@ietf.org>
From: Michael Krause <krause@cup.hp.com>
Subject: RE: [Ipoverib] RE: MCGroup Attach Failure --- IPoIB link and
  multicast draft -
In-Reply-To: <2E33960095B58E40A4D3345AB9F65EC10285F652@win-msg-01.wingro
 up.windeploy.ntdev.microsoft.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

At 02:49 PM 1/22/2002 -0800, Aditya Dube wrote:

>I agree that this is an edge case, but it is one that should be
>addressed by our IP over Infiniband multicast proposal.
>
>As a result of the HCA constraint and the IP over Infiniband - MCast GID
>mapping scheme, we have limited the number of IP MCast addresses an
>Interface can listen on. Ethernet has no such limitation.

To clarify:

A MGID is a logical handle of an IB multicast group.  MGIDs are validated 
within the CA channel interface which may be implemented either in hardware 
or software (MGIDs are also processed by IB routers who forward to the 
appropriate next hop based on the multicast routing protocol).  For each 
MGID, there is associated a QP (for IP, one would most likely have one QP 
per IP multicast group).

A MLID is a local subnet IB address handle that defines the multicast group 
within the subnet.  MLIDs are typically implemented in hardware - e.g. the 
switch chapter defines how the multicast routing table is updated.  The 
number of MLIDs may be limited within a given hardware implementation - the 
maximum is ~16K as defined within the IB specification.

It is allowed to map multiple MGIDs to a single MLID thereby allowing an 
implementation to support many more IB multicast groups than the physical 
hardware may define.  As such, there really isn't any major limitation that 
IP over IB must focus upon in terms of specification language.  There might 
be some benefit derived from having some informative text to explain some 
of this material but one can just use the IB specification as a reference 
on these issues.


Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 23 14:53:37 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA22626
	for <ipoverib-archive@odin.ietf.org>; Wed, 23 Jan 2002 14:53:33 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id OAA14102
	for ipoverib-archive@odin.ietf.org; Wed, 23 Jan 2002 14:53:35 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA14058;
	Wed, 23 Jan 2002 14:51:49 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA14026
	for <ipoverib@optimus.ietf.org>; Wed, 23 Jan 2002 14:51:42 -0500 (EST)
Received: from omegaband.com (faust.seagullsemi.com [206.196.68.34])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA22545
	for <ipoverib@ietf.org>; Wed, 23 Jan 2002 14:51:39 -0500 (EST)
Received: from seahorsepc [207.8.52.226] by omegaband.com with ESMTP
  (SMTPD32-7.04) id A2ABB58D0142; Wed, 23 Jan 2002 13:44:43 -0600
From: "Ashwin Kamath" <kamath@omegaband.com>
To: <ipoverib@ietf.org>
Subject: RE: [ipoverib] reasons for not using LID ?
Date: Wed, 23 Jan 2002 13:49:46 -0600
Message-ID: <IFEKJBDLPFOEGGPIIGMFEEGGCAAA.kamath@omegaband.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0)
Importance: Normal
In-Reply-To: <4D027986353D1341ADA4F2A4F8A170760E317C@cceexc18.americas.cpqcorp.net>
X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit


 It is possible that you will not find a valid path to a LID in the LRH of
an ARP using any given SL. You can ( and will, if any kind of traffic
management is done ) have different SL/LID (including LMC) combinations in
the paths to any given end node, such that any given LID will not have paths
for all possible SLs.

 The response path can use a different LID ( even within an IB subnet ) so
that the response reaches the same end node. When the ARP request is
generated, the requestor will not know about the existing paths to the end
node for Unicast packets. So it is bound to use a local LID which should be
used to send packets to the multicast group. The responding end node can use
the following lids in its response
 1. The LID from the LRH
    The problem with this is that a path to this LID may not exist for any
given SL (and possibly for all SLs, depending on how traffic management is
done). A path to a different LID on the same end-node/port might have to be
used for things to work.
 2. The LID from the ARP request
    The problem with this is that the requestor cannot know this value in
advance. It will use a lid but what value of LMC will it choose to guarantee
the existence of the path to the ARP responder.

 In the interop case where different implementations are working together,
the compliances on the responder must be stricter. The responder has to know
what fields to use and what fields to ignore, to guarantee proper operation.

 As is the case with ARPs today, even if an end node has a persistent ARP
cache entry, it is still bound to respond to ARP requests and it does not
make any assumptions like - the ARP requestor has a persistent ARP cache
entry/static tables.

 Question is, as a responder what should I be doing so that I can
interoperate with all implementations, because as a responder I would not
know how the requestor is implemented? The draft does not lay down any
compliance for this. Is the response gauranteed to work if I use the fields
in the ARP response, or is it gauranteed to work if I use the SA for
obtaining path information, or is it gauranteed to work in both cases.

-Ashwin.







-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Culley, Paul
Sent: Wednesday, January 23, 2002 8:36 AM
To: Vivek Kashyap; Michael Krause
Cc: ipoverib@ietf.org
Subject: RE: [ipoverib] reasons for not using LID ?


embedded...

Paul R. Culley
Compaq Sr. Fellow
281-514-5543
P.S. are we having fun yet?

-----Original Message-----
From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
Sent: Tuesday, January 22, 2002 5:15 PM
To: Michael Krause
Cc: ipoverib@ietf.org
Subject: Re: [ipoverib] reasons for not using LID ?



>>Bill, Yaron,
>>
>>         There are two situations
>>                 - proxy-ARP for a node on another subnet
>>                 - span IP subnets across IB subnets
>>
>>1.      proxy-ARP for a node on another subnet works very easily.
>>         You return the LID of the local IB router and the GID of the
>>         remote IB node. Note this is possible with the current
>>         specification. If only the GID is included excluding the LID
>>         this is not possible since you can't lookup the remote node's
>>         GID with the SA. You need to include both. [ SA to SA
>>         coordination within an IB subnet isn't specified what of
>>         across the subnet. ]
>>
>>         Note that the ARP packet is fielded by the IPoIB device
>>         logically/physically associated with the IB router. The ARP
>>         packet itself does not travel beyond the IB subnet.
>
>Within the requester's IB subnet, there will exist at least one endnode
>that will respond on behalf of any request that targets an endnode that
>exists on another IB or IP subnet.   The responder will work with the
SA to
>
>determine the IB router; an IP router would respond on behalf of its
>responsibility "cloud" and it too would work with the SA to determine
the
>IB LRH values to use to establish appropriate communication between the
>requester and the IP router.  In both cases, the responder is
responsible
>for determining the appropriate LRH / GRH (where applicable) header
>parameters to establish the communication channel.
>

Is the solution then that the remote subnet's GIDs not only have to be
known to the IPoIB device but also recorded in the SA/SM or at least
the remote GID-prefix is recorded with the SA/SM ?
<prc>
 Yes, one way or another the SA/SM must know at least which router port
the GID prefix needs to route to.  We may make it a requirement (to
simplify SA's) that the route cloud be able to deal with anything
arriving at any router port on the subnet; then a simple SA need only
route to a single router port.  More complex SA's should still be able
to chose an optimum port.
<endprc>

My note above describes a simpler solution that merely takes the LID and
the GID from the ARP reply and the SA/SM need not be involved. If the
implementation prefers to get the SL/path rate from the SA (for the IB
router) it gets them based on the LID received (which is that of the
IB router). The solution is much simpler. Any IPoIB system on the
subnet can answer for the remote node and the packet will reach
through the router with no setup required.
<prc>
By only looking up the path to the router, the SA can only report on
that path.  Since we expect to support SA lookups for the end to end
path, including the full router cloud, it will be possible to get good
information on the full path.  This is needed for backwards
compatibility with devices that don't understand anything but IB 1.x
interfaces.
<endprc>

The suggestion that the LID from the LRH be used works efficiently but
only if the responder is the IB router itself. If another device
(IPoIB device) answers for it then it will, as has been suggested,
have to indicate that the LID in the LRH is not the right one and the
solution will default to the more complex 'record non-local GIDs in
the SA'.

Is there a query/set of queries that returns the routers in the subnet
and
that tells which routers to choose ? Also, if a non-local GID prefix
is used can one determine the router to use ?
<prc>
At this time, there is no method defined for the SA to interact with the
routers, or for endnodes to get router information.  As stated before,
the current plan is to use the PathRecord query to report on the full
path, as is necessary for 1.x compatibility.  The returned path
information would include LID/SL for the appropriate router port.  When
the IB specs are updated to support routers, I expect the SA/router
interface will be included, requiring some additional support from SAs.
If you wish to submit a proposal for additional interfaces that support
the more traditional IP like methods (where endnodes must chose the
router ports for non-local work), you are free to do so.
<endprc>

If so then the solution 'use LID in LRH but only if ARP/ND didn't mark
it invalid' can work. If such a 'non-local GID' solution will not work
now because the routing infrastructure/SA records that are needed
don't exist and one has to await IB routing then the invalid flag is
not needed either. Or, if in general such an across the subnet
proxy-ARP solution can be 'ignored' then too we don't need the invalid
flag. Using the LID from LRH is sufficient.

As far as I know 'non-local GID/associated router(s)' records don't
exist and have to be invented. I would rather that we have a solution
that works now and so support LID+GID. However, if we 'ignore' this
one case of proxy-ARP across IB-subnet then LID from LRH will do.

Question 1: Can the kind of proxy-ARP described here be implemented if
the LID is not included in the ARP/ND ? i.e. do the above queries and
records exist in the SA database? If not, we either need to say that
such a solution is not possible or define a solution.
<prc> As long as we fully support the 1.x IB spec in the routed fabric,
the PathRecord query will work, so we have a solution.
<endprc>

Question 2: Is the LRH:LID always available for all UD communication ?
Is
it equally applicable to receiving UD packets on xCAs, IB routers and IB
switches (port 0 probably) ? If not, then when is it not applicable. (I
do
expect it to be available in all cases but want to be sure).
<prc> The values are in the headers for all UD packets; HCAs have an
interface for getting this information, switches should not ever need
this, routers are yet to be fully defined, and TCAs can provide the
header info if their designer's care about cross subnet support...
<endprc>


>>2.      For IP subnets to span IB subnets there is too much
>>         speculation and no justification. First of all we don't know
>>         why we need it. We don't know when it will be possible. We
>>         don't know if it will be possible. It is a complicated issue
-
>>         too many things have to be updated in IB architecutre.
>
>The justification is that vendors and customers believe it is the right
>thing to do and there is both technical and economic feasibility to
achieve
>
>this objective.  Unless there is strong consensus to not support this
type
>of functionality, the workgroup must take steps to enable it or at
least
>not preclude this from being implemented at some future date once the
IBTA
>has provided the appropriate specification updates.
>

The reason I am asking for such justificaiton because it is being used
to argue for various solutions. It is better to know what is the
reason for wanting it and then we can decide if we still want to go
ahead or not. Till now no real reason has been specified. We create IB
subnets for isolation, management etc. then why join them back again
in IPoIB subnets. Wouldn't IP subnetting serve the purpose anyway ?
<prc>
It has been stated before, but here goes again:
1) the customers want it
2) We expect that IB routers are coming, probably about the time IPoIB
really gets going...
3) We think we have a workable, if not optimum, method of supporting the
routers in mind right now (1.x PathRecord support).  So you don't really
have to wait for anything new to complete the ARP RFC.
4) The alternative for customers (to IP across IB subnets) is to install
extra IP routing functionality across any IB router; it seems that this
would slow the adoption rate of either IB routers, or IPoIB...  as well
as cost more for the customers.
5) I personally (and some others as well) would like to avoid the NEED
to return to the ietf to fix up the IPoIB RFCs when routers do become
available.  That creates too tight a tie on the RFC process to releasing
useful IB routers.  And creates chicken/egg problems if every endnode
needs to be fixed as well. That does not preclude providing some
optimizations through new RFCs as IB fabrics grow and the need for those
optimizations becomes apparent.
<endprc>

Vivek

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 23 17:25:42 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA28012
	for <ipoverib-archive@odin.ietf.org>; Wed, 23 Jan 2002 17:25:42 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id RAA20845
	for ipoverib-archive@odin.ietf.org; Wed, 23 Jan 2002 17:25:44 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA20792;
	Wed, 23 Jan 2002 17:23:48 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA20760
	for <ipoverib@optimus.ietf.org>; Wed, 23 Jan 2002 17:23:46 -0500 (EST)
Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA27999
	for <ipoverib@ietf.org>; Wed, 23 Jan 2002 17:23:44 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e31.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id RAA73878;
	Wed, 23 Jan 2002 17:18:01 -0500
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0NML2R83486;
	Wed, 23 Jan 2002 15:21:02 -0700
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0NMKxV27559;
	Wed, 23 Jan 2002 14:20:59 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0NMKle26198;
	Wed, 23 Jan 2002 14:20:47 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201232220.g0NMKle26198@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
To: rronald@VIEO.com (Roger Ronald)
Date: Wed, 23 Jan 2002 14:20:46 -0800 (PST)
Cc: ipoverib@ietf.org, krause@cup.hp.com (Michael Krause)
In-Reply-To: <004401c1a414$806071f0$a40f020a@Fishhead> from "Roger Ronald" at Jan 23, 2002 06:47:29 AM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Is this a valid conclusion in IPoIB-UD context since APM is not
supported for Unreliable Datagram mode of IB?????


Vivek


> 
> I concur with your analysis. While LID assignments will change
> infrequently, the proper LID/SL may change more often due
> to APM.
> 
> RR
> ----- Original Message ----- 
> From: "Michael Krause" <krause@cup.hp.com>
> To: <ipoverib@ietf.org>
> Sent: Monday, January 21, 2002 2:48 PM
> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> 
> 
> > At 02:13 PM 1/17/2002 -0600, Roger Ronald wrote:
> > >I think it's important to note that changes in LID should be infrequent.
> > >Very infrequent.
> > >
> > >VIEO is one of a small number of companies that has a SM at this time.
> > >We plan to maintain LID assignments whenever possible. That means
> > >across reboots, across power fails and across subnet merges (when possible).
> > >
> > >There are corner cases where LID may need to change, just like IP
> > >addresses occasionaly need to change. But these are VERY odd
> > >events...like the first time merge of fabrics. They won't happen
> > >hourly, daily, monthly..... They may happen once or twice in a
> > >fabric. Ever (unless the admin wants to make changes for the heck of it).
> > 
> > While the LID range for a given port may not change often, the LID / SL 
> > appropriate for a given path between two endnodes may change for a variety 
> > of reasons based on workload objectives / QoS requirements within the 
> > fabric for a given service.  The IB APM mechanisms may be used to update 
> > these path selections without informing IP of the change (an implementation 
> > option).  Thus, the rate of LID change may not be as low as you contend but 
> > it will also not be as high as others contend making the performance issues 
> > raised perhaps moot.
> > 
> > Mike
> > 
> > 
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > https://www1.ietf.org/mailman/listinfo/ipoverib
> > 
> > 
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 23 17:42:23 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA28411
	for <ipoverib-archive@odin.ietf.org>; Wed, 23 Jan 2002 17:42:23 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id RAA21590
	for ipoverib-archive@odin.ietf.org; Wed, 23 Jan 2002 17:42:25 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA21579;
	Wed, 23 Jan 2002 17:41:44 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA21540
	for <ipoverib@optimus.ietf.org>; Wed, 23 Jan 2002 17:41:42 -0500 (EST)
Received: from vieo.com (vieo.com [216.30.79.131])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA28392
	for <ipoverib@ietf.org>; Wed, 23 Jan 2002 17:41:39 -0500 (EST)
Received: (from root@localhost)
	by vieo.com (8.11.2/8.11.2) id g0NMfBT84764
	for ipoverib@ietf.org; Wed, 23 Jan 2002 16:41:11 -0600 (CST)
	(envelope-from rronald@VIEO.com)
Received: from Fishhead (dhcp-10-2-15-164.vieo.com [10.2.15.164])
	by vieo.com (8.11.2/8.11.2) with SMTP id g0NMfAc84682;
	Wed, 23 Jan 2002 16:41:10 -0600 (CST)
	(envelope-from rronald@VIEO.com)
Message-ID: <032e01c1a45f$057a15e0$a40f020a@Fishhead>
From: "Roger Ronald" <rronald@VIEO.com>
To: "Vivek Kashyap" <kashyapv@us.ibm.com>
Cc: <ipoverib@ietf.org>, "Michael Krause" <krause@cup.hp.com>
References: <200201232220.g0NMKle26198@eng4.beaverton.ibm.com>
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
Date: Wed, 23 Jan 2002 16:40:55 -0600
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 5.50.4133.2400
X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400
X-scanner: scanned by Inflex 0.1.5c+ on vieo.com
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Doh! That's what I get for trying to be agreeable. The statements below
are true for the general case, but you are correct that APM is
not relevant for UD. 

So, I'll revert to the claim that LID changes
should be truely unusual events, at least with the SM/SA 
behavior that my company expects to provide.

RR


----- Original Message ----- 
From: "Vivek Kashyap" <kashyapv@us.ibm.com>
To: "Roger Ronald" <rronald@vieo.com>
Cc: <ipoverib@ietf.org>; "Michael Krause" <krause@cup.hp.com>
Sent: Wednesday, January 23, 2002 4:20 PM
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?


> Is this a valid conclusion in IPoIB-UD context since APM is not
> supported for Unreliable Datagram mode of IB?????
> 
> 
> Vivek
> 
> 
> > 
> > I concur with your analysis. While LID assignments will change
> > infrequently, the proper LID/SL may change more often due
> > to APM.
> > 
> > RR
> > ----- Original Message ----- 
> > From: "Michael Krause" <krause@cup.hp.com>
> > To: <ipoverib@ietf.org>
> > Sent: Monday, January 21, 2002 2:48 PM
> > Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> > 
> > 
> > > At 02:13 PM 1/17/2002 -0600, Roger Ronald wrote:
> > > >I think it's important to note that changes in LID should be infrequent.
> > > >Very infrequent.
> > > >
> > > >VIEO is one of a small number of companies that has a SM at this time.
> > > >We plan to maintain LID assignments whenever possible. That means
> > > >across reboots, across power fails and across subnet merges (when possible).
> > > >
> > > >There are corner cases where LID may need to change, just like IP
> > > >addresses occasionaly need to change. But these are VERY odd
> > > >events...like the first time merge of fabrics. They won't happen
> > > >hourly, daily, monthly..... They may happen once or twice in a
> > > >fabric. Ever (unless the admin wants to make changes for the heck of it).
> > > 
> > > While the LID range for a given port may not change often, the LID / SL 
> > > appropriate for a given path between two endnodes may change for a variety 
> > > of reasons based on workload objectives / QoS requirements within the 
> > > fabric for a given service.  The IB APM mechanisms may be used to update 
> > > these path selections without informing IP of the change (an implementation 
> > > option).  Thus, the rate of LID change may not be as low as you contend but 
> > > it will also not be as high as others contend making the performance issues 
> > > raised perhaps moot.
> > > 
> > > Mike
> > > 
> > > 
> > > _______________________________________________
> > > IPoverIB mailing list
> > > IPoverIB@ietf.org
> > > https://www1.ietf.org/mailman/listinfo/ipoverib
> > > 
> > > 
> > 
> > 
> > _______________________________________________
> > IPoverIB mailing list
> > IPoverIB@ietf.org
> > https://www1.ietf.org/mailman/listinfo/ipoverib
> > 
> 
> 
> -- 
> Vivek Kashyap
> Linux Technology Center, IBM
> kashyapv@us.ibm.com 
> vivk@us.ibm.com 
> 503 578 3422 (o)
> 
> 


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 23 17:45:34 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA28456
	for <ipoverib-archive@odin.ietf.org>; Wed, 23 Jan 2002 17:45:33 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id RAA21689
	for ipoverib-archive@odin.ietf.org; Wed, 23 Jan 2002 17:45:35 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA21668;
	Wed, 23 Jan 2002 17:44:48 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA21641
	for <ipoverib@optimus.ietf.org>; Wed, 23 Jan 2002 17:44:47 -0500 (EST)
Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA28438
	for <ipoverib@ietf.org>; Wed, 23 Jan 2002 17:44:44 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e31.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id RAA33820;
	Wed, 23 Jan 2002 17:41:37 -0500
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0NMiWR24686;
	Wed, 23 Jan 2002 15:44:32 -0700
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id g0NMiUg10852;
	Wed, 23 Jan 2002 14:44:30 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0NMiTB28223;
	Wed, 23 Jan 2002 14:44:29 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201232244.g0NMiTB28223@eng4.beaverton.ibm.com>
Subject: Re: [ipoverib] reasons for not using LID ?
To: Paul.Culley@compaq.com (Culley, Paul)
Date: Wed, 23 Jan 2002 14:44:28 -0800 (PST)
Cc: kashyapv@us.ibm.com (Vivek Kashyap), krause@cup.hp.com (Michael Krause),
        ipoverib@ietf.org
In-Reply-To: <4D027986353D1341ADA4F2A4F8A170760E317C@cceexc18.americas.cpqcorp.net> from "Culley, Paul" at Jan 23, 2002 07:35:43 AM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Paul,
        I was asking about the present status. Based on your answers
        it seems that proxy-ARP for a node on anotehr subnet won't
        work at present. THus such setups will have to wait for IB
        routing to be developed.



> Is there a query/set of queries that returns the routers in the subnet
> and
> that tells which routers to choose ? Also, if a non-local GID prefix
> is used can one determine the router to use ?
> <prc>
> At this time, there is no method defined for the SA to interact with the
> routers, or for endnodes to get router information.  As stated before,
> the current plan is to use the PathRecord query to report on the full
> path, as is necessary for 1.x compatibility.  The returned path
> information would include LID/SL for the appropriate router port.  When
> the IB specs are updated to support routers, I expect the SA/router
> interface will be included, requiring some additional support from SAs.
> If you wish to submit a proposal for additional interfaces that support
> the more traditional IP like methods (where endnodes must chose the
> router ports for non-local work), you are free to do so.
> <endprc>
> 


> Question 1: Can the kind of proxy-ARP described here be implemented if
> the LID is not included in the ARP/ND ? i.e. do the above queries and
> records exist in the SA database? If not, we either need to say that
> such a solution is not possible or define a solution.

> <prc> As long as we fully support the 1.x IB spec in the routed fabric,
> the PathRecord query will work, so we have a solution.
> <endprc>

I did not understand - unless you meant in the future you expect the
pathrecord query to work with the GID whether with local/remote
prefix.  Am I right if I say there is no solution at present?

> 
> Question 2: Is the LRH:LID always available for all UD communication ?
> Is
> it equally applicable to receiving UD packets on xCAs, IB routers and IB
> switches (port 0 probably) ? If not, then when is it not applicable. (I
> do
> expect it to be available in all cases but want to be sure).

> <prc> The values are in the headers for all UD packets; HCAs have an
> interface for getting this information, switches should not ever need
> this, routers are yet to be fully defined, and TCAs can provide the
> header info if their designer's care about cross subnet support...
> <endprc>

So, this is not a universal interface. 

You say routers are not defined yet - will they include this information in the future.

Switches receive MADs which are sent over the UD mode - aren't they
going to receive the information? 

I didn't realise that TCA's are not bound by these rules. What about
multicasting then - are TCAs bound by the same rules as the HCAs?

This also springs a doubt - is it mandatory for the HCAs to provide
this interface?


> 
> The reason I am asking for such justificaiton because it is being used
> to argue for various solutions. It is better to know what is the
> reason for wanting it and then we can decide if we still want to go
> ahead or not. Till now no real reason has been specified. We create IB
> subnets for isolation, management etc. then why join them back again
> in IPoIB subnets. Wouldn't IP subnetting serve the purpose anyway ?
> <prc>
> It has been stated before, but here goes again:
> 1) the customers want it

	To do what  ?

> 2) We expect that IB routers are coming, probably about the time IPoIB
> really gets going...
> 3) We think we have a workable, if not optimum, method of supporting the
> routers in mind right now (1.x PathRecord support).  So you don't really
> have to wait for anything new to complete the ARP RFC.
> 4) The alternative for customers (to IP across IB subnets) is to install
> extra IP routing functionality across any IB router; it seems that this
> would slow the adoption rate of either IB routers, or IPoIB...  as well
> as cost more for the customers.
> 5) I personally (and some others as well) would like to avoid the NEED
> to return to the ietf to fix up the IPoIB RFCs when routers do become
> available.  That creates too tight a tie on the RFC process to releasing
> useful IB routers.  And creates chicken/egg problems if every endnode
> needs to be fixed as well. That does not preclude providing some
> optimizations through new RFCs as IB fabrics grow and the need for those
> optimizations becomes apparent.
> <endprc>

I am sorry if you had to repeat these but this is the first time I
have seen a list of reasons.

Agree with 5, but wanted to do the extra dotting of i's only if
needed. Still not sure if IPoIB multi-IB subnets are needed -
basically need to understand what use could they be put to. However,
am dropping this line of discussion - wouldn't mind any 'customer' to
list their reasons for asking this function though.

Vivek

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 23 23:12:28 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA05065
	for <ipoverib-archive@odin.ietf.org>; Wed, 23 Jan 2002 23:12:27 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id XAA00527
	for ipoverib-archive@odin.ietf.org; Wed, 23 Jan 2002 23:12:31 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id XAA00503;
	Wed, 23 Jan 2002 23:11:42 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id XAA00476
	for <ipoverib@optimus.ietf.org>; Wed, 23 Jan 2002 23:11:39 -0500 (EST)
Received: from dcmtechdom.dcmtech.co.in ([203.190.136.130])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA05038
	for <ipoverib@ietf.org>; Wed, 23 Jan 2002 23:11:33 -0500 (EST)
Received: by DCMTECHDOM with Internet Mail Service (5.5.2653.19)
	id <DQ4APZ31>; Thu, 24 Jan 2002 09:45:14 +0530
Message-ID: <7FADCB99FC82D41199F9000629A85D1A03004643@DCMTECHDOM>
From: Rakesh Khurana <Khurana.R@dcmtech.co.in>
To: ipoverib@ietf.org
Date: Thu, 24 Jan 2002 09:45:13 +0530
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="iso-8859-1"
Subject: [Ipoverib] (no subject)
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Thu Jan 24 16:25:07 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA06375
	for <ipoverib-archive@odin.ietf.org>; Thu, 24 Jan 2002 16:25:06 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id QAA24449
	for ipoverib-archive@odin.ietf.org; Thu, 24 Jan 2002 16:25:10 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA24378;
	Thu, 24 Jan 2002 16:24:24 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA24346
	for <ipoverib@ns.ietf.org>; Thu, 24 Jan 2002 16:24:23 -0500 (EST)
Received: from mxout1.netvision.net.il (mxout1.netvision.net.il [194.90.9.20])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA06363
	for <ipoverib@ietf.org>; Thu, 24 Jan 2002 16:24:17 -0500 (EST)
Received: from vaio505he ([139.92.208.144]) by mxout1.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GQG0060RORNCO@mxout1.netvision.net.il> for ipoverib@ietf.org;
 Thu, 24 Jan 2002 23:23:49 +0200 (IST)
Date: Thu, 24 Jan 2002 23:24:22 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: [ipoverib] moving forward with the open Issues
To: ipoverib@ietf.org
Reply-to: yaronh@voltaire.com
Message-id: <000401c1a51d$7ea212b0$90d05c8b@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: text/plain; charset=windows-1255
Content-transfer-encoding: 7BIT
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7BIT

Hi all

By now it looks to me like the LID issues is going to be resolved
By just dropping the LID from the ARP, and using the one in the LRH
And there is a consensus around it also from Vivek

For the capability bits, I suggested reserving the bits for future use (in
this RFC)
After we finish with this RFC we could start discussing the bits to be
defined in a separate RFC
This will enable every one to agree at least for the UD RFC on every thing

Are there any other open issues that are not in consensus right now ?
Does any one want to bring a new topic ?

If these are the last points it would really help close the standard quickly

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Thu Jan 24 18:10:45 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA09240
	for <ipoverib-archive@odin.ietf.org>; Thu, 24 Jan 2002 18:10:45 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id SAA01071
	for ipoverib-archive@odin.ietf.org; Thu, 24 Jan 2002 18:10:47 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA00989;
	Thu, 24 Jan 2002 18:09:55 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA00957
	for <ipoverib@ns.ietf.org>; Thu, 24 Jan 2002 18:09:53 -0500 (EST)
Received: from mail5.microsoft.com (mail5.microsoft.com [131.107.3.121])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA09208
	for <ipoverib@ietf.org>; Thu, 24 Jan 2002 18:09:50 -0500 (EST)
Received: from inet-vrs-05.redmond.corp.microsoft.com ([157.54.6.148]) by mail5.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Thu, 24 Jan 2002 15:09:23 -0800
Received: from 157.54.5.25 by inet-vrs-05.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Thu, 24 Jan 2002 15:09:22 -0800
Received: from red-imc-01.redmond.corp.microsoft.com ([157.54.9.102]) by inet-hub-03.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Thu, 24 Jan 2002 15:09:22 -0800
Received: from win-imc-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.39]) by red-imc-01.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Thu, 24 Jan 2002 15:09:21 -0800
Received: from win-msg-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.52]) by win-imc-01.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3588.0);
	 Thu, 24 Jan 2002 15:07:07 -0800
X-MimeOLE: Produced By Microsoft Exchange V6.0.6132.0
Content-Class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Subject: RE: [ipoverib] moving forward with the open Issues
Date: Thu, 24 Jan 2002 15:07:01 -0800
Message-ID: <2E33960095B58E40A4D3345AB9F65EC104F90228@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
Thread-Topic: [ipoverib] moving forward with the open Issues
Thread-Index: AcGlHTKivTSpVva3TvCSI3WPKcIIRgADjMCQ
From: "Arvind Murching" <arvindm@windows.microsoft.com>
To: <yaronh@voltaire.com>, <ipoverib@ietf.org>
X-OriginalArrivalTime: 24 Jan 2002 23:07:07.0715 (UTC) FILETIME=[D87A3930:01C1A52B]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id SAA00958
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

We should settle on whether or not an EtherType field is needed.

- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com] 
Sent: Thursday, January 24, 2002 1:24 PM
To: ipoverib@ietf.org
Subject: [ipoverib] moving forward with the open Issues

Hi all

By now it looks to me like the LID issues is going to be resolved
By just dropping the LID from the ARP, and using the one in the LRH
And there is a consensus around it also from Vivek

For the capability bits, I suggested reserving the bits for future use
(in
this RFC)
After we finish with this RFC we could start discussing the bits to be
defined in a separate RFC
This will enable every one to agree at least for the UD RFC on every
thing

Are there any other open issues that are not in consensus right now ?
Does any one want to bring a new topic ?

If these are the last points it would really help close the standard
quickly

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Thu Jan 24 18:36:04 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA09911
	for <ipoverib-archive@odin.ietf.org>; Thu, 24 Jan 2002 18:36:03 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id SAA02297
	for ipoverib-archive@odin.ietf.org; Thu, 24 Jan 2002 18:36:06 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA02097;
	Thu, 24 Jan 2002 18:31:22 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA02066
	for <ipoverib@ns.ietf.org>; Thu, 24 Jan 2002 18:31:21 -0500 (EST)
Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA09754
	for <ipoverib@ietf.org>; Thu, 24 Jan 2002 18:31:16 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e31.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id SAA117260;
	Thu, 24 Jan 2002 18:28:15 -0500
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0ONVHv23568;
	Thu, 24 Jan 2002 16:31:17 -0700
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0ONVEV09659;
	Thu, 24 Jan 2002 15:31:14 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0ONVCN04098;
	Thu, 24 Jan 2002 15:31:12 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201242331.g0ONVCN04098@eng4.beaverton.ibm.com>
Subject: Re: [ipoverib] reasons for not using LID ?
To: kamath@omegaband.com (Ashwin Kamath)
Date: Thu, 24 Jan 2002 15:31:11 -0800 (PST)
Cc: ipoverib@ietf.org
In-Reply-To: <IFEKJBDLPFOEGGPIIGMFEEGGCAAA.kamath@omegaband.com> from "Ashwin Kamath" at Jan 23, 2002 12:49:46 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Ashwin,
        That is a good point but connectivity is not guaranteed across
        the IB fabric even if you exchange the GIDs. You may enquire
        on a GID, received in a multicast (ARP/ND), but that does not
        mean there is *any* path back to the sender.

        This therefore translates into a requirement for IPoIB - the
        administrator must ensure connectivity within the partition.

        An IP interface essentially can be tied to any particular
        value - the GID (at certain level - port/HCA etc.), the port
        (GID or LID), the LID (no LMC).

        The leftmost in the list provides for most flexibility - can
        choose from many paths, even ports. The rightmost provides for
        a single path but certainty that all nodes (senders/receivers)
        come to the same conclusion.

        The choice of the link-descriptor, in general, need not be
        directly mapped to the IP interface's associations since the
        internal storage of an implementation can always be different.
        We have chosen to return the GID and the QPN :

	QPN+ Flags+ GID

        A GID resolution from the SA can return a multiple DLIDs, 1
        DLID or nothing. 
	
	If only 1 path : things are simple and fine.

        If nothing - we have the requirement for connectivity and a
        non-working connection for the administrator to debug.

        If multiple paths the choice is more difficult. The SA doesn't
        provide many hints - it gives the MTU, SL etc. For things to
        be effective we need the SA to provide or reflect a policy so
        that the sender and a multiple set of receivers reach the same
        conclusion.  

        At present there is no general way for the sender/receiver to
        agree on the paths to be followed back to the sender.
        Load-balancing will not work if all responders just choose the
        same DLID for example. It might work in a limited setup where
        different receivers in the subnet all can reach the sender
        only on a single path - this has to be ensured by the
        administrator. If there are multiple paths there is no way of
        ensuring that the receivers follow a particular path. That can
        be accomplished by letting the receiver know the exact LID to
        use either directly or setting the SA with only one path.

        Given the above I guess a way out - though not guaranteeing
        connectivity - is to say that the IB subnet will decide based
        on IB methods. These methods include - getting the LIDs from
        the SA, getting the LID from the LRH, getting the parameters
        from the SA, getting the parameters in some other way etc. The
        choice of the specific path based on these parameters is upto
        the implementations. They will leverage any IB solution
        available whenever it is.

        The other end is to state that the subnet has to be more
        tightly controlled - the administrator sets up the
        connectivity and every LID that may be used in the IPoIB
        subnet must have connectivity to every other node.

Vivek	


> 
> 
>  It is possible that you will not find a valid path to a LID in the LRH of
> an ARP using any given SL. You can ( and will, if any kind of traffic
> management is done ) have different SL/LID (including LMC) combinations in
> the paths to any given end node, such that any given LID will not have paths
> for all possible SLs.
> 
>  The response path can use a different LID ( even within an IB subnet ) so
> that the response reaches the same end node. When the ARP request is
> generated, the requestor will not know about the existing paths to the end
> node for Unicast packets. So it is bound to use a local LID which should be
> used to send packets to the multicast group. The responding end node can use
> the following lids in its response
>  1. The LID from the LRH
>     The problem with this is that a path to this LID may not exist for any
> given SL (and possibly for all SLs, depending on how traffic management is
> done). A path to a different LID on the same end-node/port might have to be
> used for things to work.
>  2. The LID from the ARP request
>     The problem with this is that the requestor cannot know this value in
> advance. It will use a lid but what value of LMC will it choose to guarantee
> the existence of the path to the ARP responder.
> 
>  In the interop case where different implementations are working together,
> the compliances on the responder must be stricter. The responder has to know
> what fields to use and what fields to ignore, to guarantee proper operation.
> 
>  As is the case with ARPs today, even if an end node has a persistent ARP
> cache entry, it is still bound to respond to ARP requests and it does not
> make any assumptions like - the ARP requestor has a persistent ARP cache
> entry/static tables.
> 
>  Question is, as a responder what should I be doing so that I can
> interoperate with all implementations, because as a responder I would not
> know how the requestor is implemented? The draft does not lay down any
> compliance for this. Is the response gauranteed to work if I use the fields
> in the ARP response, or is it gauranteed to work if I use the SA for
> obtaining path information, or is it gauranteed to work in both cases.
> 
> -Ashwin.
> 
> 
> 
> 
> 
> 
> 
> -----Original Message-----
> From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
> Of Culley, Paul
> Sent: Wednesday, January 23, 2002 8:36 AM
> To: Vivek Kashyap; Michael Krause
> Cc: ipoverib@ietf.org
> Subject: RE: [ipoverib] reasons for not using LID ?
> 
> 
> embedded...
> 
> Paul R. Culley
> Compaq Sr. Fellow
> 281-514-5543
> P.S. are we having fun yet?
> 
> -----Original Message-----
> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> Sent: Tuesday, January 22, 2002 5:15 PM
> To: Michael Krause
> Cc: ipoverib@ietf.org
> Subject: Re: [ipoverib] reasons for not using LID ?
> 
> 
> 
> >>Bill, Yaron,
> >>
> >>         There are two situations
> >>                 - proxy-ARP for a node on another subnet
> >>                 - span IP subnets across IB subnets
> >>
> >>1.      proxy-ARP for a node on another subnet works very easily.
> >>         You return the LID of the local IB router and the GID of the
> >>         remote IB node. Note this is possible with the current
> >>         specification. If only the GID is included excluding the LID
> >>         this is not possible since you can't lookup the remote node's
> >>         GID with the SA. You need to include both. [ SA to SA
> >>         coordination within an IB subnet isn't specified what of
> >>         across the subnet. ]
> >>
> >>         Note that the ARP packet is fielded by the IPoIB device
> >>         logically/physically associated with the IB router. The ARP
> >>         packet itself does not travel beyond the IB subnet.
> >
> >Within the requester's IB subnet, there will exist at least one endnode
> >that will respond on behalf of any request that targets an endnode that
> >exists on another IB or IP subnet.   The responder will work with the
> SA to
> >
> >determine the IB router; an IP router would respond on behalf of its
> >responsibility "cloud" and it too would work with the SA to determine
> the
> >IB LRH values to use to establish appropriate communication between the
> >requester and the IP router.  In both cases, the responder is
> responsible
> >for determining the appropriate LRH / GRH (where applicable) header
> >parameters to establish the communication channel.
> >
> 
> Is the solution then that the remote subnet's GIDs not only have to be
> known to the IPoIB device but also recorded in the SA/SM or at least
> the remote GID-prefix is recorded with the SA/SM ?
> <prc>
>  Yes, one way or another the SA/SM must know at least which router port
> the GID prefix needs to route to.  We may make it a requirement (to
> simplify SA's) that the route cloud be able to deal with anything
> arriving at any router port on the subnet; then a simple SA need only
> route to a single router port.  More complex SA's should still be able
> to chose an optimum port.
> <endprc>
> 
> My note above describes a simpler solution that merely takes the LID and
> the GID from the ARP reply and the SA/SM need not be involved. If the
> implementation prefers to get the SL/path rate from the SA (for the IB
> router) it gets them based on the LID received (which is that of the
> IB router). The solution is much simpler. Any IPoIB system on the
> subnet can answer for the remote node and the packet will reach
> through the router with no setup required.
> <prc>
> By only looking up the path to the router, the SA can only report on
> that path.  Since we expect to support SA lookups for the end to end
> path, including the full router cloud, it will be possible to get good
> information on the full path.  This is needed for backwards
> compatibility with devices that don't understand anything but IB 1.x
> interfaces.
> <endprc>
> 
> The suggestion that the LID from the LRH be used works efficiently but
> only if the responder is the IB router itself. If another device
> (IPoIB device) answers for it then it will, as has been suggested,
> have to indicate that the LID in the LRH is not the right one and the
> solution will default to the more complex 'record non-local GIDs in
> the SA'.
> 
> Is there a query/set of queries that returns the routers in the subnet
> and
> that tells which routers to choose ? Also, if a non-local GID prefix
> is used can one determine the router to use ?
> <prc>
> At this time, there is no method defined for the SA to interact with the
> routers, or for endnodes to get router information.  As stated before,
> the current plan is to use the PathRecord query to report on the full
> path, as is necessary for 1.x compatibility.  The returned path
> information would include LID/SL for the appropriate router port.  When
> the IB specs are updated to support routers, I expect the SA/router
> interface will be included, requiring some additional support from SAs.
> If you wish to submit a proposal for additional interfaces that support
> the more traditional IP like methods (where endnodes must chose the
> router ports for non-local work), you are free to do so.
> <endprc>
> 
> If so then the solution 'use LID in LRH but only if ARP/ND didn't mark
> it invalid' can work. If such a 'non-local GID' solution will not work
> now because the routing infrastructure/SA records that are needed
> don't exist and one has to await IB routing then the invalid flag is
> not needed either. Or, if in general such an across the subnet
> proxy-ARP solution can be 'ignored' then too we don't need the invalid
> flag. Using the LID from LRH is sufficient.
> 
> As far as I know 'non-local GID/associated router(s)' records don't
> exist and have to be invented. I would rather that we have a solution
> that works now and so support LID+GID. However, if we 'ignore' this
> one case of proxy-ARP across IB-subnet then LID from LRH will do.
> 
> Question 1: Can the kind of proxy-ARP described here be implemented if
> the LID is not included in the ARP/ND ? i.e. do the above queries and
> records exist in the SA database? If not, we either need to say that
> such a solution is not possible or define a solution.
> <prc> As long as we fully support the 1.x IB spec in the routed fabric,
> the PathRecord query will work, so we have a solution.
> <endprc>
> 
> Question 2: Is the LRH:LID always available for all UD communication ?
> Is
> it equally applicable to receiving UD packets on xCAs, IB routers and IB
> switches (port 0 probably) ? If not, then when is it not applicable. (I
> do
> expect it to be available in all cases but want to be sure).
> <prc> The values are in the headers for all UD packets; HCAs have an
> interface for getting this information, switches should not ever need
> this, routers are yet to be fully defined, and TCAs can provide the
> header info if their designer's care about cross subnet support...
> <endprc>
> 
> 
> >>2.      For IP subnets to span IB subnets there is too much
> >>         speculation and no justification. First of all we don't know
> >>         why we need it. We don't know when it will be possible. We
> >>         don't know if it will be possible. It is a complicated issue
> -
> >>         too many things have to be updated in IB architecutre.
> >
> >The justification is that vendors and customers believe it is the right
> >thing to do and there is both technical and economic feasibility to
> achieve
> >
> >this objective.  Unless there is strong consensus to not support this
> type
> >of functionality, the workgroup must take steps to enable it or at
> least
> >not preclude this from being implemented at some future date once the
> IBTA
> >has provided the appropriate specification updates.
> >
> 
> The reason I am asking for such justificaiton because it is being used
> to argue for various solutions. It is better to know what is the
> reason for wanting it and then we can decide if we still want to go
> ahead or not. Till now no real reason has been specified. We create IB
> subnets for isolation, management etc. then why join them back again
> in IPoIB subnets. Wouldn't IP subnetting serve the purpose anyway ?
> <prc>
> It has been stated before, but here goes again:
> 1) the customers want it
> 2) We expect that IB routers are coming, probably about the time IPoIB
> really gets going...
> 3) We think we have a workable, if not optimum, method of supporting the
> routers in mind right now (1.x PathRecord support).  So you don't really
> have to wait for anything new to complete the ARP RFC.
> 4) The alternative for customers (to IP across IB subnets) is to install
> extra IP routing functionality across any IB router; it seems that this
> would slow the adoption rate of either IB routers, or IPoIB...  as well
> as cost more for the customers.
> 5) I personally (and some others as well) would like to avoid the NEED
> to return to the ietf to fix up the IPoIB RFCs when routers do become
> available.  That creates too tight a tie on the RFC process to releasing
> useful IB routers.  And creates chicken/egg problems if every endnode
> needs to be fixed as well. That does not preclude providing some
> optimizations through new RFCs as IB fabrics grow and the need for those
> optimizations becomes apparent.
> <endprc>
> 
> Vivek
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Thu Jan 24 21:53:39 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA13575
	for <ipoverib-archive@odin.ietf.org>; Thu, 24 Jan 2002 21:53:34 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id VAA09658
	for ipoverib-archive@odin.ietf.org; Thu, 24 Jan 2002 21:53:37 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA09590;
	Thu, 24 Jan 2002 21:53:01 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA09561
	for <ipoverib@ns.ietf.org>; Thu, 24 Jan 2002 21:52:59 -0500 (EST)
Received: from patan.sun.com (patan.Sun.COM [192.18.98.43])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA13554
	for <ipoverib@ietf.org>; Thu, 24 Jan 2002 21:52:55 -0500 (EST)
Received: from sunmail1.Sun.COM ([129.145.1.2])
	by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id TAA22662;
	Thu, 24 Jan 2002 19:51:36 -0700 (MST)
Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.83.130])
	by sunmail1.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1-Sun.COM.mod.2) with ESMTP id SAA13915;
	Thu, 24 Jan 2002 18:52:58 -0800 (PST)
Received: from taipei (taipei.Eng.Sun.COM [129.146.85.178])
	by jurassic.eng.sun.com (8.12.2+Sun/8.12.2) with SMTP id g0P2pZ3g562715;
	Thu, 24 Jan 2002 18:51:35 -0800 (PST)
Message-Id: <200201250251.g0P2pZ3g562715@jurassic.eng.sun.com>
Date: Thu, 24 Jan 2002 18:48:51 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Subject: RE: [Ipoverib] RE: MCGroup Attach Failure --- IPoIB link and  multicast draft -
To: krause@cup.hp.com, adube@windows.microsoft.com
Cc: ipoverib@ietf.org
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: LoL83K30dqqcFiMhPxwtig==
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.6_06 SunOS 5.8 sun4u sparc 
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

So it looks like the contraint on the number of MGIDs is much less
stringent than MLIDs. (I suspect many implementations will simply
implement MGIDs in software so the real contraint for it is how
much memory.) Since IB allows MLIDs sharing among different MGIDs,
I'd say the chance of running out of MGIDs in the real world is very
small. If it does happen, so be it. Note that this could happen on
IPoverEther as well.

If the failure occurs when an end node tries to attach to an IPoIB
link, e.g. the attach to the all-node or all-router mutlicast group
fails, it seems reasonable to expect the node can't join the link
until it frees up some MC resources.

Jerry

>X-Sender: krause@esalpha2.cup.hp.com
>Date: Wed, 23 Jan 2002 07:34:33 -0800
>To: <ipoverib@ietf.org>
>From: Michael Krause <krause@cup.hp.com>
>Subject: RE: [Ipoverib] RE: MCGroup Attach Failure --- IPoIB link and multicast 
draft -
>Mime-Version: 1.0
>X-Mailman-Version: 1.0
>List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
>X-BeenThere: ipoverib@ietf.org
>
>At 02:49 PM 1/22/2002 -0800, Aditya Dube wrote:
>
>>I agree that this is an edge case, but it is one that should be
>>addressed by our IP over Infiniband multicast proposal.
>>
>>As a result of the HCA constraint and the IP over Infiniband - MCast GID
>>mapping scheme, we have limited the number of IP MCast addresses an
>>Interface can listen on. Ethernet has no such limitation.
>
>To clarify:
>
>A MGID is a logical handle of an IB multicast group.  MGIDs are validated 
>within the CA channel interface which may be implemented either in hardware 
>or software (MGIDs are also processed by IB routers who forward to the 
>appropriate next hop based on the multicast routing protocol).  For each 
>MGID, there is associated a QP (for IP, one would most likely have one QP 
>per IP multicast group).
>
>A MLID is a local subnet IB address handle that defines the multicast group 
>within the subnet.  MLIDs are typically implemented in hardware - e.g. the 
>switch chapter defines how the multicast routing table is updated.  The 
>number of MLIDs may be limited within a given hardware implementation - the 
>maximum is ~16K as defined within the IB specification.
>
>It is allowed to map multiple MGIDs to a single MLID thereby allowing an 
>implementation to support many more IB multicast groups than the physical 
>hardware may define.  As such, there really isn't any major limitation that 
>IP over IB must focus upon in terms of specification language.  There might 
>be some benefit derived from having some informative text to explain some 
>of this material but one can just use the IB specification as a reference 
>on these issues.
>
>
>Mike
>
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Fri Jan 25 02:20:09 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA26349
	for <ipoverib-archive@odin.ietf.org>; Fri, 25 Jan 2002 02:20:08 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id CAA29609
	for ipoverib-archive@odin.ietf.org; Fri, 25 Jan 2002 02:20:09 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id CAA29573;
	Fri, 25 Jan 2002 02:18:53 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id CAA29541
	for <ipoverib@optimus.ietf.org>; Fri, 25 Jan 2002 02:18:51 -0500 (EST)
Received: from pdxpo.dsl-only.net (pdxpo.dsl-only.net [63.105.16.3])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA26323
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 02:18:46 -0500 (EST)
Received: from bonfire (unverified [63.105.18.201]) by pdxpo.dsl-only.net
 (Rockliffe SMTPRA 4.5.6) with SMTP id <B0009377713@pdxpo.dsl-only.net> for <ipoverib@ietf.org>;
 Thu, 24 Jan 2002 23:17:34 -0800
From: "Bill Strahm" <bill@strahm.net>
To: <ipoverib@ietf.org>
Date: Thu, 24 Jan 2002 23:27:36 -0800
Message-ID: <HHEDIJGOEFLPCPNILMPAIEHACFAA.bill@strahm.net>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="windows-1255"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0)
Importance: Normal
In-Reply-To: <000401c1a51d$7ea212b0$90d05c8b@vaio505he>
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
Content-Transfer-Encoding: 7bit
Subject: [Ipoverib] Attempt for a Consensus call for LIDs in the ARP response
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit


[Working Group Chair Hat firmly in place]
I would like to make a concensus call on not having a LID in the ARP
response.  I believe the two main proponents for having the LID have found
other ways of getting the LID that they need and do not need it in the ARP
response.

Who has remaining technical reasons for maintaining the LID in the ARP
response
[End WG Chair message]

Bill


-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Yaron Haviv
Sent: Thursday, January 24, 2002 1:24 PM
To: ipoverib@ietf.org
Subject: [ipoverib] moving forward with the open Issues


Hi all

By now it looks to me like the LID issues is going to be resolved
By just dropping the LID from the ARP, and using the one in the LRH
And there is a consensus around it also from Vivek

For the capability bits, I suggested reserving the bits for future use (in
this RFC)
After we finish with this RFC we could start discussing the bits to be
defined in a separate RFC
This will enable every one to agree at least for the UD RFC on every thing

Are there any other open issues that are not in consensus right now ?
Does any one want to bring a new topic ?

If these are the last points it would really help close the standard quickly

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Fri Jan 25 12:50:11 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA09095
	for <ipoverib-archive@odin.ietf.org>; Fri, 25 Jan 2002 12:50:11 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id MAA29352
	for ipoverib-archive@odin.ietf.org; Fri, 25 Jan 2002 12:50:12 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA29330;
	Fri, 25 Jan 2002 12:49:05 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA29296
	for <ipoverib@optimus.ietf.org>; Fri, 25 Jan 2002 12:49:03 -0500 (EST)
Received: from mxout1.netvision.net.il (mxout1.netvision.net.il [194.90.9.20])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA09021
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 12:48:59 -0500 (EST)
Received: from vaio505he ([139.92.208.238]) by mxout1.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GQI00BI79GR1Y@mxout1.netvision.net.il> for ipoverib@ietf.org;
 Fri, 25 Jan 2002 19:48:29 +0200 (IST)
Date: Fri, 25 Jan 2002 19:46:21 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: [ipoverib] resolving the Ethertype issue
To: ipoverib@ietf.org
Reply-to: yaronh@voltaire.com
Message-id: <001101c1a5c8$344a6ea0$a3d05c8b@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: text/plain; charset=windows-1255
Content-transfer-encoding: 8BIT
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
Content-Transfer-Encoding: 8BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8BIT


As Arvind mentioned the Ethertype field issue is one of the last items to
agree up on

Currently there are 2 proposals:
1.	(Vivek) no need for ethertype, we could see its an ARP from the first
fields in the Packet (IP Ver)
2.	put the Ethertype field in the first few bytes of each message

I personally don't see it as a critical decision choosing 1 or 2, the only
thing that Vivek proposal might not enable is doing Ethernet over IB which
is not in the scope of this RFC anyway
I would tend to agree with Vivek for the simplicity reasons (not the
performance ones)

If any one has a firm reason why to put the Ethertype field in the payload
please make your arguments
And it would help in settling this matter too
Please try to give real reasons in this case too, and not things like “it is
better”, etc'

Yaron



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 25 14:21:21 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA11626
	for <ipoverib-archive@odin.ietf.org>; Fri, 25 Jan 2002 14:21:16 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id OAA04318
	for ipoverib-archive@odin.ietf.org; Fri, 25 Jan 2002 14:21:19 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA04226;
	Fri, 25 Jan 2002 14:19:48 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA04194
	for <ipoverib@ns.ietf.org>; Fri, 25 Jan 2002 14:19:45 -0500 (EST)
Received: from mail6.microsoft.com (mail6.microsoft.com [131.107.3.126])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA11600
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 14:19:42 -0500 (EST)
Received: from inet-vrs-06.redmond.corp.microsoft.com ([157.54.6.201]) by mail6.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Fri, 25 Jan 2002 11:19:12 -0800
Received: from 157.54.6.197 by inet-vrs-06.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Fri, 25 Jan 2002 11:19:13 -0800
Received: from red-imc-04.redmond.corp.microsoft.com ([157.54.2.168]) by inet-hub-06.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Fri, 25 Jan 2002 11:19:12 -0800
Received: from win-imc-02.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.84]) by red-imc-04.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Fri, 25 Jan 2002 11:19:12 -0800
Received: from win-msg-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.52]) by win-imc-02.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3588.0);
	 Fri, 25 Jan 2002 11:16:58 -0800
X-MimeOLE: Produced By Microsoft Exchange V6.0.6132.0
Content-Class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Subject: RE: [ipoverib] moving forward with the open Issues - end node configuration
Date: Fri, 25 Jan 2002 11:16:57 -0800
Message-ID: <2E33960095B58E40A4D3345AB9F65EC10285F690@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
Thread-Topic: [ipoverib] moving forward with the open Issues - end node configuration
Thread-Index: AcGlHTKivTSpVva3TvCSI3WPKcIIRgADjMCQACpV1EA=
From: "Aditya Dube" <adube@windows.microsoft.com>
To: <ipoverib@ietf.org>
X-OriginalArrivalTime: 25 Jan 2002 19:16:58.0210 (UTC) FILETIME=[DBC8BC20:01C1A5D4]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id OAA04195
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit


									
The current IPOverIB spec requires that the P_Key and MTU be configured
on each node. 

The IPOverIB spec should provide a default method for an end-node to
come online with no IPoIB-specific configuration.

During initialization, a node could query an IPOverIB specific
ServiceRecord (Table 164, pg 685). This query will return a P_Key and
ServiceSpecificFlags. The ServiceSpecificFlags can specify whether 2048
or 4096 is the link MTU. 

The ServiceRecord's ServiceID should have a default value specific to
IPOverIB. This number could be 0xFFFF:4000 in accordance with vol3
section on "ServiceID Namespace" (Table 128, pg 215).

The ServiceRecord needs to be created just as the all-node or broadcast
MCGroup is created, i.e. by the first node on the network. 

If the Network Admin wishes to put the end-node on a different P_Key,
then the Admin may override the values returned by the ServiceRecord.
That is outside the scope of my proposal.


Aditya Dube



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 25 14:50:07 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA12515
	for <ipoverib-archive@odin.ietf.org>; Fri, 25 Jan 2002 14:50:06 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id OAA06066
	for ipoverib-archive@odin.ietf.org; Fri, 25 Jan 2002 14:50:09 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA05999;
	Fri, 25 Jan 2002 14:48:20 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA05971
	for <ipoverib@ns.ietf.org>; Fri, 25 Jan 2002 14:48:19 -0500 (EST)
Received: from pdxpo.dsl-only.net (pdxpo.dsl-only.net [63.105.16.3])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA12462
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 14:48:02 -0500 (EST)
Received: from bonfire (unverified [63.105.18.201]) by pdxpo.dsl-only.net
 (Rockliffe SMTPRA 4.5.6) with SMTP id <B0009387921@pdxpo.dsl-only.net>;
 Fri, 25 Jan 2002 11:46:49 -0800
From: "Bill Strahm" <bill@strahm.net>
To: "Aditya Dube" <adube@windows.microsoft.com>, <ipoverib@ietf.org>
Subject: RE: [ipoverib] moving forward with the open Issues - end node configuration
Date: Fri, 25 Jan 2002 11:56:49 -0800
Message-ID: <HHEDIJGOEFLPCPNILMPAAEHHCFAA.bill@strahm.net>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0)
Importance: Normal
In-Reply-To: <2E33960095B58E40A4D3345AB9F65EC10285F690@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Quick comment,

I thought Vol 3 was never an approved IBTA specification and was dropped...

I definately agree with you that a zero-config type environment would be
nice for IPoIB traffic, however there might be some other alternatives.

I can see DHCP extensions
I can see Service Records (can the IETF define these Mike Krause ?) like you
suggested
I can see manual configuration - technically not zero config at this point
I can see defaults in the protocol - Though SHOULD use 2K MTU and a P_Key of
mikimitdigoat or I'll even let you use a ascii representation of your
initials if you are vain <G> (Just don't tell anyone)

Do you think you could write a proposal describing how any/all of these
methods that you think will work and any others you might come across...

Thank you
Bill

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Aditya Dube
Sent: Friday, January 25, 2002 11:17 AM
To: ipoverib@ietf.org
Subject: RE: [ipoverib] moving forward with the open Issues - end node
configuration




The current IPOverIB spec requires that the P_Key and MTU be configured
on each node.

The IPOverIB spec should provide a default method for an end-node to
come online with no IPoIB-specific configuration.

During initialization, a node could query an IPOverIB specific
ServiceRecord (Table 164, pg 685). This query will return a P_Key and
ServiceSpecificFlags. The ServiceSpecificFlags can specify whether 2048
or 4096 is the link MTU.

The ServiceRecord's ServiceID should have a default value specific to
IPOverIB. This number could be 0xFFFF:4000 in accordance with vol3
section on "ServiceID Namespace" (Table 128, pg 215).

The ServiceRecord needs to be created just as the all-node or broadcast
MCGroup is created, i.e. by the first node on the network.

If the Network Admin wishes to put the end-node on a different P_Key,
then the Admin may override the values returned by the ServiceRecord.
That is outside the scope of my proposal.


Aditya Dube



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 25 15:06:36 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA12855
	for <ipoverib-archive@odin.ietf.org>; Fri, 25 Jan 2002 15:06:36 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id PAA06982
	for ipoverib-archive@odin.ietf.org; Fri, 25 Jan 2002 15:06:39 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA06959;
	Fri, 25 Jan 2002 15:05:53 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA06916
	for <ipoverib@ns.ietf.org>; Fri, 25 Jan 2002 15:05:48 -0500 (EST)
Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA12838
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 15:05:44 -0500 (EST)
Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23])
	by e31.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id PAA51900
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 15:02:41 -0500
Received: from d03nm036.boulder.ibm.com (avpilot.boulder.ibm.com [9.17.188.135])
	by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0PK5iO18986
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 13:05:44 -0700
Subject: RE: [ipoverib] moving forward with the open Issues - end node configuration
To: ipoverib@ietf.org
X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000
Message-ID: <OFBDD1AEE8.F7BE200D-ON88256B4C.006AF5B3@boulder.ibm.com>
From: "Vivek Kashyap" <vivk@us.ibm.com>
Date: Fri, 25 Jan 2002 12:04:20 -0800
X-MIMETrack: Serialize by Router on D03NM036/03/M/IBM(Release 5.0.9 |November 16, 2001) at
 01/25/2002 01:05:44 PM
MIME-Version: 1.0
Content-type: text/plain; charset=us-ascii
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org


<VK> I am not sure what you gain by that. The P_Key is programmed into the
port by the SM on startup. You already have it in the P_Key tables.  How
you determine the P_Key to use is upto your administrator.

If there are multiple IPoIB subnets over the fabric - how do you know which
serviceID to choose ? These will have to be specified by the administrator.
So the administrator might as well specify the P_Key per IP interface. You
may automate the process by ensuring that your interface tries all the
P_Keys in the relevant port.

The P_Key then allows you to determine the all-nodes multicast/broadcast
address (as defined in the multicast draft). This address that has to be
setup by the administrator (first node/other method). Every interface is
required to join this address. The MTU returned on the join request is the
MTU of the interface. So the MTU is automatically configured. The join
request is less than 256 bytes and hence will be received by the SM
whatever be the link MTU/path MTU.

Zero-conf can be done in the following way though:

      The P_Keys are in the P_Key tables. Let, by default, there exist an
MGID formed by zeroing out the link-id (P_Key) field in the MGID. So for
IPv4 it will be FF12::255.255.255.255. This group is joined by the
interfaces. If they are in the partition of this MGID they will receive the
right P_Key to use for the subnet. They will also get the subnet MTU. There
can be only one such subnet in the IB subnet. If multiple IPoIB links are
defined then this scheme won't be universal but it allows complete
automation in the default case.

For multiple IPoIB links:

      If the IPoIB ports can support at least two P_Keys (since we are
looking at multiple IPoIB links) we can always define a default subnet (as
above). A DHCP server can be put on this IPoIB link. A node then enquires
this DHCP server for the P_Key. I can suggest the DHCP extensions/process
required if needed.

Comments ?

Vivek
<VK>


The current IPOverIB spec requires that the P_Key and MTU be configured
on each node.

The IPOverIB spec should provide a default method for an end-node to
come online with no IPoIB-specific configuration.

During initialization, a node could query an IPOverIB specific
ServiceRecord (Table 164, pg 685). This query will return a P_Key and
ServiceSpecificFlags. The ServiceSpecificFlags can specify whether 2048
or 4096 is the link MTU.

The ServiceRecord's ServiceID should have a default value specific to
IPOverIB. This number could be 0xFFFF:4000 in accordance with vol3
section on "ServiceID Namespace" (Table 128, pg 215).

The ServiceRecord needs to be created just as the all-node or broadcast
MCGroup is created, i.e. by the first node on the network.

If the Network Admin wishes to put the end-node on a different P_Key,
then the Admin may override the values returned by the ServiceRecord.
That is outside the scope of my proposal.


Aditya Dube



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib





_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 25 15:37:29 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA13589
	for <ipoverib-archive@odin.ietf.org>; Fri, 25 Jan 2002 15:37:29 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id PAA08709
	for ipoverib-archive@odin.ietf.org; Fri, 25 Jan 2002 15:37:32 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA08629;
	Fri, 25 Jan 2002 15:36:32 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA08598
	for <ipoverib@ns.ietf.org>; Fri, 25 Jan 2002 15:36:30 -0500 (EST)
Received: from calliope1.fm.intel.com (fmfdns01.fm.intel.com [132.233.247.10])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA13571
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 15:36:26 -0500 (EST)
Received: from fmsmsxvs041.fm.intel.com (fmsmsxv041-1.fm.intel.com [132.233.48.109])
	by calliope1.fm.intel.com (8.9.1a+p1/8.9.1/d: relay.m4,v 1.49 2002/01/25 02:16:58 root Exp $) with SMTP id UAA27081
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 20:36:27 GMT
Received: from fmsmsx26.fm.intel.com ([132.233.42.26])
 by fmsmsxvs041.fm.intel.com (NAVGW 2.5.1.16) with SMTP id M2002012512372114137
 ; Fri, 25 Jan 2002 12:37:21 -0800
Received: by fmsmsx26.fm.intel.com with Internet Mail Service (5.5.2653.19)
	id <DNDVX7N2>; Fri, 25 Jan 2002 12:36:26 -0800
Message-ID: <D9223EB959A5D511A98F00508B68C20C21A882@ORSMSX108>
From: "Larsen, Roy K" <roy.k.larsen@intel.com>
To: "'yaronh@voltaire.com'" <yaronh@voltaire.com>, ipoverib@ietf.org
Subject: RE: [ipoverib] resolving the Ethertype issue
Date: Fri, 25 Jan 2002 12:36:25 -0800
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="windows-1255"
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id PAA08599
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

Although my feelings aren't strong enough to drag the reflector into a never
ending rat hole, I believe the IPoIB draft should specify a
protoco/"Ethertype" as the first field of the payload which is treated as an
IPoIB protocol specific field.  The reasons are as follows:

1.  Interesting OSs require the link driver to identify the destination
protocol of received frames.

2.  QPNs do not provide a clean mechanism to do this because, at a minimum,
both ARP responses and IPv4 frames must share the same QPN.

3.  Requiring the IPoIB driver to peek into an upper-level protocol frame is
architecturally displeasing. (Read that as an euphemism for KLUDGE.)

4.  Arguments that some day the IB header may contain such a field are
irrelevant in defining a protocol today.  (Same reasoning we've seen for IB
subnet routing issues.)

Roy

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Friday, January 25, 2002 9:46 AM
To: ipoverib@ietf.org
Subject: [ipoverib] resolving the Ethertype issue



As Arvind mentioned the Ethertype field issue is one of the last items to
agree up on

Currently there are 2 proposals:
1.	(Vivek) no need for ethertype, we could see its an ARP from the
first
fields in the Packet (IP Ver)
2.	put the Ethertype field in the first few bytes of each message

I personally don't see it as a critical decision choosing 1 or 2, the only
thing that Vivek proposal might not enable is doing Ethernet over IB which
is not in the scope of this RFC anyway
I would tend to agree with Vivek for the simplicity reasons (not the
performance ones)

If any one has a firm reason why to put the Ethertype field in the payload
please make your arguments
And it would help in settling this matter too
Please try to give real reasons in this case too, and not things like “it is
better”, etc'

Yaron



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 25 16:09:35 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA14427
	for <ipoverib-archive@odin.ietf.org>; Fri, 25 Jan 2002 16:09:35 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id QAA10142
	for ipoverib-archive@odin.ietf.org; Fri, 25 Jan 2002 16:09:39 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA10037;
	Fri, 25 Jan 2002 16:05:05 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA10005
	for <ipoverib@ns.ietf.org>; Fri, 25 Jan 2002 16:05:03 -0500 (EST)
Received: from pdxpo.dsl-only.net (pdxpo.dsl-only.net [63.105.16.3])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA14311
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 16:04:58 -0500 (EST)
Received: from bonfire (unverified [63.105.18.201]) by pdxpo.dsl-only.net
 (Rockliffe SMTPRA 4.5.6) with SMTP id <B0009389472@pdxpo.dsl-only.net> for <ipoverib@ietf.org>;
 Fri, 25 Jan 2002 13:03:43 -0800
From: "Bill Strahm" <bill@strahm.net>
To: <ipoverib@ietf.org>
Subject: RE: [Ipoverib] Attempt for a Consensus call for LIDs in the ARP response
Date: Fri, 25 Jan 2002 13:13:42 -0800
Message-ID: <HHEDIJGOEFLPCPNILMPAGEHJCFAA.bill@strahm.net>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="windows-1255"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0)
Importance: Normal
In-Reply-To: <HHEDIJGOEFLPCPNILMPAIEHACFAA.bill@strahm.net>
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

[WG Hat still on]
Just for the record,
Can I get people who agree that the LID should not be apart of the Link
Address to affirm this decision ?

Bill
[/Hat]

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Bill Strahm
Sent: Thursday, January 24, 2002 11:28 PM
To: ipoverib@ietf.org
Subject: [Ipoverib] Attempt for a Consensus call for LIDs in the ARP
response



[Working Group Chair Hat firmly in place]
I would like to make a concensus call on not having a LID in the ARP
response.  I believe the two main proponents for having the LID have found
other ways of getting the LID that they need and do not need it in the ARP
response.

Who has remaining technical reasons for maintaining the LID in the ARP
response
[End WG Chair message]

Bill


-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Yaron Haviv
Sent: Thursday, January 24, 2002 1:24 PM
To: ipoverib@ietf.org
Subject: [ipoverib] moving forward with the open Issues


Hi all

By now it looks to me like the LID issues is going to be resolved
By just dropping the LID from the ARP, and using the one in the LRH
And there is a consensus around it also from Vivek

For the capability bits, I suggested reserving the bits for future use (in
this RFC)
After we finish with this RFC we could start discussing the bits to be
defined in a separate RFC
This will enable every one to agree at least for the UD RFC on every thing

Are there any other open issues that are not in consensus right now ?
Does any one want to bring a new topic ?

If these are the last points it would really help close the standard quickly

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 25 16:37:58 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA14972
	for <ipoverib-archive@odin.ietf.org>; Fri, 25 Jan 2002 16:37:57 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id QAA12073
	for ipoverib-archive@odin.ietf.org; Fri, 25 Jan 2002 16:38:01 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA12011;
	Fri, 25 Jan 2002 16:37:13 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA11981
	for <ipoverib@ns.ietf.org>; Fri, 25 Jan 2002 16:37:11 -0500 (EST)
Received: from mail2.microsoft.com (mail2.microsoft.com [131.107.3.124])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA14963
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 16:37:06 -0500 (EST)
Received: from INET-VRS-02.redmond.corp.microsoft.com ([157.54.8.110]) by mail2.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Fri, 25 Jan 2002 13:36:38 -0800
Received: from 157.54.8.23 by INET-VRS-02.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Fri, 25 Jan 2002 13:36:38 -0800
Received: from red-imc-02.redmond.corp.microsoft.com ([157.54.9.107]) by inet-hub-01.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Fri, 25 Jan 2002 13:36:37 -0800
Received: from win-imc-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.39]) by red-imc-02.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Fri, 25 Jan 2002 13:36:36 -0800
Received: from win-msg-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.52]) by win-imc-01.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3588.0);
	 Fri, 25 Jan 2002 13:34:21 -0800
X-MimeOLE: Produced By Microsoft Exchange V6.0.6132.0
Content-Class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Subject: RE: [Ipoverib] Attempt for a Consensus call for LIDs in the ARP response
Date: Fri, 25 Jan 2002 13:34:20 -0800
Message-ID: <2E33960095B58E40A4D3345AB9F65EC10353E39B@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
Thread-Topic: [Ipoverib] Attempt for a Consensus call for LIDs in the ARP response
Thread-Index: AcGl46fnYHTSWxraQGmNLCDJlTa7BgABKYSw
From: "Arvind Murching" <arvindm@windows.microsoft.com>
To: <ipoverib@ietf.org>
X-OriginalArrivalTime: 25 Jan 2002 21:34:21.0377 (UTC) FILETIME=[0D183310:01C1A5E8]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id QAA11982
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

I agree that the LID should not be part of the link layer address.

Thanks
Arvind

-----Original Message-----
From: Bill Strahm [mailto:bill@strahm.net] 
Sent: Friday, January 25, 2002 1:14 PM
To: ipoverib@ietf.org
Subject: RE: [Ipoverib] Attempt for a Consensus call for LIDs in the ARP
response

[WG Hat still on]
Just for the record,
Can I get people who agree that the LID should not be apart of the Link
Address to affirm this decision ?

Bill
[/Hat]

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of
Bill Strahm
Sent: Thursday, January 24, 2002 11:28 PM
To: ipoverib@ietf.org
Subject: [Ipoverib] Attempt for a Consensus call for LIDs in the ARP
response



[Working Group Chair Hat firmly in place]
I would like to make a concensus call on not having a LID in the ARP
response.  I believe the two main proponents for having the LID have
found
other ways of getting the LID that they need and do not need it in the
ARP
response.

Who has remaining technical reasons for maintaining the LID in the ARP
response
[End WG Chair message]

Bill


-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of
Yaron Haviv
Sent: Thursday, January 24, 2002 1:24 PM
To: ipoverib@ietf.org
Subject: [ipoverib] moving forward with the open Issues


Hi all

By now it looks to me like the LID issues is going to be resolved
By just dropping the LID from the ARP, and using the one in the LRH
And there is a consensus around it also from Vivek

For the capability bits, I suggested reserving the bits for future use
(in
this RFC)
After we finish with this RFC we could start discussing the bits to be
defined in a separate RFC
This will enable every one to agree at least for the UD RFC on every
thing

Are there any other open issues that are not in consensus right now ?
Does any one want to bring a new topic ?

If these are the last points it would really help close the standard
quickly

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 25 16:39:00 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA15013
	for <ipoverib-archive@odin.ietf.org>; Fri, 25 Jan 2002 16:39:00 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id QAA12174
	for ipoverib-archive@odin.ietf.org; Fri, 25 Jan 2002 16:39:04 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA12131;
	Fri, 25 Jan 2002 16:38:16 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA12089
	for <ipoverib@ns.ietf.org>; Fri, 25 Jan 2002 16:38:14 -0500 (EST)
Received: from calliope1.fm.intel.com (fmfdns01.fm.intel.com [132.233.247.10])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA14988
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 16:38:09 -0500 (EST)
Received: from fmsmsxvs043.fm.intel.com (fmsmsxv043-1.fm.intel.com [132.233.48.128])
	by calliope1.fm.intel.com (8.9.1a+p1/8.9.1/d: relay.m4,v 1.49 2002/01/25 02:16:58 root Exp $) with SMTP id VAA16652
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 21:38:11 GMT
Received: from FMSMSX017.fm.intel.com ([132.233.42.196])
 by fmsmsxvs043.fm.intel.com (NAVGW 2.5.1.16) with SMTP id M2002012513372032509
 ; Fri, 25 Jan 2002 13:37:20 -0800
Received: by fmsmsx017.fm.intel.com with Internet Mail Service (5.5.2653.19)
	id <DVGXSLV9>; Fri, 25 Jan 2002 13:38:10 -0800
Message-ID: <D9223EB959A5D511A98F00508B68C20C21A883@ORSMSX108>
From: "Larsen, Roy K" <roy.k.larsen@intel.com>
To: "'Bill Strahm'" <bill@strahm.net>, ipoverib@ietf.org
Subject: RE: [Ipoverib] Attempt for a Consensus call for LIDs in the ARP r
	esponse
Date: Fri, 25 Jan 2002 13:38:00 -0800
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="windows-1255"
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Affirmed - The LID is not a required attribute of the link address.

Roy

-----Original Message-----
From: Bill Strahm [mailto:bill@strahm.net]
Sent: Friday, January 25, 2002 1:14 PM
To: ipoverib@ietf.org
Subject: RE: [Ipoverib] Attempt for a Consensus call for LIDs in the ARP
response


[WG Hat still on]
Just for the record,
Can I get people who agree that the LID should not be apart of the Link
Address to affirm this decision ?

Bill
[/Hat]

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Bill Strahm
Sent: Thursday, January 24, 2002 11:28 PM
To: ipoverib@ietf.org
Subject: [Ipoverib] Attempt for a Consensus call for LIDs in the ARP
response



[Working Group Chair Hat firmly in place]
I would like to make a concensus call on not having a LID in the ARP
response.  I believe the two main proponents for having the LID have found
other ways of getting the LID that they need and do not need it in the ARP
response.

Who has remaining technical reasons for maintaining the LID in the ARP
response
[End WG Chair message]

Bill


-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Yaron Haviv
Sent: Thursday, January 24, 2002 1:24 PM
To: ipoverib@ietf.org
Subject: [ipoverib] moving forward with the open Issues


Hi all

By now it looks to me like the LID issues is going to be resolved
By just dropping the LID from the ARP, and using the one in the LRH
And there is a consensus around it also from Vivek

For the capability bits, I suggested reserving the bits for future use (in
this RFC)
After we finish with this RFC we could start discussing the bits to be
defined in a separate RFC
This will enable every one to agree at least for the UD RFC on every thing

Are there any other open issues that are not in consensus right now ?
Does any one want to bring a new topic ?

If these are the last points it would really help close the standard quickly

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 25 16:45:45 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA15162
	for <ipoverib-archive@odin.ietf.org>; Fri, 25 Jan 2002 16:45:45 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id QAA12489
	for ipoverib-archive@odin.ietf.org; Fri, 25 Jan 2002 16:45:49 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA12458;
	Fri, 25 Jan 2002 16:44:53 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA12422
	for <ipoverib@ns.ietf.org>; Fri, 25 Jan 2002 16:44:51 -0500 (EST)
Received: from thalia.fm.intel.com (fmfdns02.fm.intel.com [132.233.247.11])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA15156
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 16:44:46 -0500 (EST)
Received: from fmsmsxvs041.fm.intel.com (fmsmsxv041-1.fm.intel.com [132.233.48.109])
	by thalia.fm.intel.com (8.9.1a+p1/8.9.1/d: relay.m4,v 1.49 2002/01/25 02:16:58 root Exp $) with SMTP id VAA19506
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 21:44:47 GMT
Received: from FMSMSX016.fm.intel.com ([132.233.42.195])
 by fmsmsxvs041.fm.intel.com (NAVGW 2.5.1.16) with SMTP id M2002012513454024937
 ; Fri, 25 Jan 2002 13:45:40 -0800
Received: by fmsmsx016.fm.intel.com with Internet Mail Service (5.5.2653.19)
	id <DVGXBDMH>; Fri, 25 Jan 2002 13:44:44 -0800
Message-ID: <D9223EB959A5D511A98F00508B68C20C02ACF5CC@ORSMSX108>
From: "Deleganes, Ellen M" <ellen.m.deleganes@intel.com>
To: "'Bill Strahm'" <bill@strahm.net>, ipoverib@ietf.org
Subject: RE: [Ipoverib] Attempt for a Consensus call for LIDs in the ARP r
	esponse
Date: Fri, 25 Jan 2002 13:44:40 -0800
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="windows-1255"
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

I agree the LID should not be a part of the Link address.

Ellen Deleganes

-----Original Message-----
From: Bill Strahm [mailto:bill@strahm.net]
Sent: Friday, January 25, 2002 1:14 PM
To: ipoverib@ietf.org
Subject: RE: [Ipoverib] Attempt for a Consensus call for LIDs in the ARP
response


[WG Hat still on]
Just for the record,
Can I get people who agree that the LID should not be apart of the Link
Address to affirm this decision ?

Bill
[/Hat]

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Bill Strahm
Sent: Thursday, January 24, 2002 11:28 PM
To: ipoverib@ietf.org
Subject: [Ipoverib] Attempt for a Consensus call for LIDs in the ARP
response



[Working Group Chair Hat firmly in place]
I would like to make a concensus call on not having a LID in the ARP
response.  I believe the two main proponents for having the LID have found
other ways of getting the LID that they need and do not need it in the ARP
response.

Who has remaining technical reasons for maintaining the LID in the ARP
response
[End WG Chair message]

Bill


-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Yaron Haviv
Sent: Thursday, January 24, 2002 1:24 PM
To: ipoverib@ietf.org
Subject: [ipoverib] moving forward with the open Issues


Hi all

By now it looks to me like the LID issues is going to be resolved
By just dropping the LID from the ARP, and using the one in the LRH
And there is a consensus around it also from Vivek

For the capability bits, I suggested reserving the bits for future use (in
this RFC)
After we finish with this RFC we could start discussing the bits to be
defined in a separate RFC
This will enable every one to agree at least for the UD RFC on every thing

Are there any other open issues that are not in consensus right now ?
Does any one want to bring a new topic ?

If these are the last points it would really help close the standard quickly

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 25 16:53:10 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA15344
	for <ipoverib-archive@odin.ietf.org>; Fri, 25 Jan 2002 16:53:10 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id QAA12720
	for ipoverib-archive@odin.ietf.org; Fri, 25 Jan 2002 16:53:14 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA12700;
	Fri, 25 Jan 2002 16:52:18 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA12668
	for <ipoverib@ns.ietf.org>; Fri, 25 Jan 2002 16:52:16 -0500 (EST)
Received: from ztxmail04.ztx.compaq.com (ztxmail04.ztx.compaq.com [161.114.1.208])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA15307
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 16:52:11 -0500 (EST)
Received: from cceexg12.americas.cpqcorp.net (cceexg12.americas.cpqcorp.net [16.110.250.124])
	by ztxmail04.ztx.compaq.com (Postfix) with ESMTP
	id 3DAB21A6; Fri, 25 Jan 2002 15:51:43 -0600 (CST)
Received: from cceexc18.americas.cpqcorp.net ([16.110.250.64]) by cceexg12.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966);
	 Fri, 25 Jan 2002 15:51:43 -0600
X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="windows-1255"
Subject: RE: [ipoverib] resolving the Ethertype issue
Date: Fri, 25 Jan 2002 15:51:42 -0600
Message-ID: <4D027986353D1341ADA4F2A4F8A170760E317E@cceexc18.americas.cpqcorp.net>
Thread-Topic: [ipoverib] resolving the Ethertype issue
Thread-Index: AcGl4BAis5E4kMmySQ2SJGqJfq+iGgACF+/A
From: "Culley, Paul" <Paul.Culley@COMPAQ.com>
To: "Larsen, Roy K" <roy.k.larsen@intel.com>, <yaronh@voltaire.com>,
        <ipoverib@ietf.org>
X-OriginalArrivalTime: 25 Jan 2002 21:51:43.0037 (UTC) FILETIME=[79F8E6D0:01C1A5EA]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id QAA12669
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

embedded...

Paul R. Culley
Compaq Sr. Fellow
281-514-5543


-----Original Message-----
From: Larsen, Roy K [mailto:roy.k.larsen@intel.com]
Sent: Friday, January 25, 2002 2:36 PM
To: 'yaronh@voltaire.com'; ipoverib@ietf.org
Subject: RE: [ipoverib] resolving the Ethertype issue


Although my feelings aren't strong enough to drag the reflector into a
never
ending rat hole, I believe the IPoIB draft should specify a
protoco/"Ethertype" as the first field of the payload which is treated
as an
IPoIB protocol specific field.  The reasons are as follows:

1.  Interesting OSs require the link driver to identify the destination
protocol of received frames.
<prc>
True, but at the moment, only two protocols are in our charter to
differentiate.  These could be easily identified by examining the first
nibble of the packet; IP has 4 or 6 in the first nibble, surly we can
chose some other value to id ARP.  Or you can use the following QP
proposal.  Or both, if the implementation wants both protocols on one
QP.
</prc> 

2.  QPNs do not provide a clean mechanism to do this because, at a
minimum,
both ARP responses and IPv4 frames must share the same QPN.
<prc>
 I don't see why this must be so.  I seems perfectly possible to include
an ARP return QP in the ARP request that would be used in the ARP
response.  The ARP response can, in turn include the IP QP for use by
the IP traffic.  If gratuitous ARP and ARP snooping are
allowed/supported, then the ARP request should also include the separate
IP QP.  This solves the problem for the two protocols that our group is
chartered to solve.
</prc>

3.  Requiring the IPoIB driver to peek into an upper-level protocol
frame is
architecturally displeasing. (Read that as an euphemism for KLUDGE.)
<prc>
Whatever works...
</prc>

4.  Arguments that some day the IB header may contain such a field are
irrelevant in defining a protocol today.  (Same reasoning we've seen for
IB
subnet routing issues.)

Roy

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Friday, January 25, 2002 9:46 AM
To: ipoverib@ietf.org
Subject: [ipoverib] resolving the Ethertype issue



As Arvind mentioned the Ethertype field issue is one of the last items
to
agree up on

Currently there are 2 proposals:
1.	(Vivek) no need for ethertype, we could see its an ARP from the
first
fields in the Packet (IP Ver)
2.	put the Ethertype field in the first few bytes of each message

I personally don't see it as a critical decision choosing 1 or 2, the
only
thing that Vivek proposal might not enable is doing Ethernet over IB
which
is not in the scope of this RFC anyway
I would tend to agree with Vivek for the simplicity reasons (not the
performance ones)

If any one has a firm reason why to put the Ethertype field in the
payload
please make your arguments
And it would help in settling this matter too
Please try to give real reasons in this case too, and not things like
“it is
better”, etc'

Yaron



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 25 17:27:14 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA16500
	for <ipoverib-archive@odin.ietf.org>; Fri, 25 Jan 2002 17:27:13 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id RAA14595
	for ipoverib-archive@odin.ietf.org; Fri, 25 Jan 2002 17:27:18 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA14494;
	Fri, 25 Jan 2002 17:25:17 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA14460
	for <ipoverib@ns.ietf.org>; Fri, 25 Jan 2002 17:25:15 -0500 (EST)
Received: from mxout2.netvision.net.il (mxout2.netvision.net.il [194.90.9.21])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA16459
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 17:25:09 -0500 (EST)
Received: from vaio505he ([139.92.208.22]) by mxout2.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GQI000KIM92EY@mxout2.netvision.net.il> for ipoverib@ietf.org;
 Sat, 26 Jan 2002 00:24:41 +0200 (IST)
Date: Sat, 26 Jan 2002 00:21:28 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: RE: [ipoverib] resolving the Ethertype issue
In-reply-to: <D9223EB959A5D511A98F00508B68C20C21A882@ORSMSX108>
To: "'Larsen, Roy K'" <roy.k.larsen@intel.com>, ipoverib@ietf.org
Reply-to: yaronh@voltaire.com
Message-id: <001801c1a5ee$a30c5b20$a3d05c8b@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: text/plain; charset=windows-1255
Content-transfer-encoding: 7BIT
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7BIT

Roy, similar arguments like Paul, see embadded

 -----Original Message-----
From: 	ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]  On Behalf
Of Larsen, Roy K
Sent:	Friday, January 25, 2002 10:36 PM
To:	'yaronh@voltaire.com'; ipoverib@ietf.org
Subject:	RE: [ipoverib] resolving the Ethertype issue

Although my feelings aren't strong enough to drag the reflector into a never
ending rat hole, I believe the IPoIB draft should specify a
protoco/"Ethertype" as the first field of the payload which is treated as an
IPoIB protocol specific field.  The reasons are as follows:

1.  Interesting OSs require the link driver to identify the destination
protocol of received frames.
<yh> since you know its IP, you could fill it out

2.  QPNs do not provide a clean mechanism to do this because, at a minimum,
both ARP responses and IPv4 frames must share the same QPN.
<yh> the idea Vivek suggested is to put zero in the IP ver field in cases of
an ARP, this resolves the problem

3.  Requiring the IPoIB driver to peek into an upper-level protocol frame is
architecturally displeasing. (Read that as an euphemism for KLUDGE.)
<yh> also Ethernet IP drivers do some of it today, not sure I get what you
mean as upper-level in this case

4.  Arguments that some day the IB header may contain such a field are
irrelevant in defining a protocol today.  (Same reasoning we've seen for IB
subnet routing issues.)
<yh> this was not assumed, just was argued that it might also solve the
problem of limiting it to IP (& not also other Ethernet stuff)

Roy

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Friday, January 25, 2002 9:46 AM
To: ipoverib@ietf.org
Subject: [ipoverib] resolving the Ethertype issue



As Arvind mentioned the Ethertype field issue is one of the last items to
agree up on

Currently there are 2 proposals:
1.	(Vivek) no need for ethertype, we could see its an ARP from the
first
fields in the Packet (IP Ver)
2.	put the Ethertype field in the first few bytes of each message

I personally don't see it as a critical decision choosing 1 or 2, the only
thing that Vivek proposal might not enable is doing Ethernet over IB which
is not in the scope of this RFC anyway
I would tend to agree with Vivek for the simplicity reasons (not the
performance ones)

If any one has a firm reason why to put the Ethertype field in the payload
please make your arguments
And it would help in settling this matter too
Please try to give real reasons in this case too, and not things like "it is
better", etc'

Yaron



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 25 17:27:53 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA16533
	for <ipoverib-archive@odin.ietf.org>; Fri, 25 Jan 2002 17:27:48 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id RAA14614
	for ipoverib-archive@odin.ietf.org; Fri, 25 Jan 2002 17:27:53 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA14564;
	Fri, 25 Jan 2002 17:26:11 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA14510
	for <ipoverib@ns.ietf.org>; Fri, 25 Jan 2002 17:26:09 -0500 (EST)
Received: from mxout2.netvision.net.il (mxout2.netvision.net.il [194.90.9.21])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA16481
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 17:26:02 -0500 (EST)
Received: from vaio505he ([139.92.208.22]) by mxout2.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GQI000KJMAKE3@mxout2.netvision.net.il> for ipoverib@ietf.org;
 Sat, 26 Jan 2002 00:25:34 +0200 (IST)
Date: Sat, 26 Jan 2002 00:22:22 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: RE: [Ipoverib] Attempt for a Consensus call for LIDs in the ARP
 response
In-reply-to: <HHEDIJGOEFLPCPNILMPAGEHJCFAA.bill@strahm.net>
To: "'Bill Strahm'" <bill@strahm.net>, ipoverib@ietf.org
Reply-to: yaronh@voltaire.com
Message-id: <001901c1a5ee$c2c47920$a3d05c8b@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: text/plain; charset=windows-1255
Content-transfer-encoding: 7BIT
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7BIT

I agree that the LID should not be part of the link layer address.

Yaron

 -----Original Message-----
From: 	ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]  On Behalf
Of Bill Strahm
Sent:	Friday, January 25, 2002 11:14 PM
To:	ipoverib@ietf.org
Subject:	RE: [Ipoverib] Attempt for a Consensus call for LIDs in the ARP
response

[WG Hat still on]
Just for the record,
Can I get people who agree that the LID should not be apart of the Link
Address to affirm this decision ?

Bill
[/Hat]

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Bill Strahm
Sent: Thursday, January 24, 2002 11:28 PM
To: ipoverib@ietf.org
Subject: [Ipoverib] Attempt for a Consensus call for LIDs in the ARP
response



[Working Group Chair Hat firmly in place]
I would like to make a concensus call on not having a LID in the ARP
response.  I believe the two main proponents for having the LID have found
other ways of getting the LID that they need and do not need it in the ARP
response.

Who has remaining technical reasons for maintaining the LID in the ARP
response
[End WG Chair message]

Bill


-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Yaron Haviv
Sent: Thursday, January 24, 2002 1:24 PM
To: ipoverib@ietf.org
Subject: [ipoverib] moving forward with the open Issues


Hi all

By now it looks to me like the LID issues is going to be resolved
By just dropping the LID from the ARP, and using the one in the LRH
And there is a consensus around it also from Vivek

For the capability bits, I suggested reserving the bits for future use (in
this RFC)
After we finish with this RFC we could start discussing the bits to be
defined in a separate RFC
This will enable every one to agree at least for the UD RFC on every thing

Are there any other open issues that are not in consensus right now ?
Does any one want to bring a new topic ?

If these are the last points it would really help close the standard quickly

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 25 17:33:56 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA16788
	for <ipoverib-archive@odin.ietf.org>; Fri, 25 Jan 2002 17:33:55 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id RAA15169
	for ipoverib-archive@odin.ietf.org; Fri, 25 Jan 2002 17:33:55 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA14893;
	Fri, 25 Jan 2002 17:30:31 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA14847
	for <ipoverib@ns.ietf.org>; Fri, 25 Jan 2002 17:30:29 -0500 (EST)
Received: from zinfandel.topspincom.com ([209.237.59.50])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA16608
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 17:30:24 -0500 (EST)
Received: from anarchy.topspincom.com (lab2.topspincom.com [10.0.0.210]) by zinfandel.topspincom.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13)
	id ZH5HKPLF; Fri, 25 Jan 2002 14:34:13 -0800
Received: (from libor@localhost)
	by anarchy.topspincom.com (8.11.0/8.11.0) id g0PMYsT25941
	for ipoverib@ietf.org; Fri, 25 Jan 2002 14:34:54 -0800
Date: Fri, 25 Jan 2002 14:34:54 -0800
From: Libor Michalek <libor@pobox.com>
To: ipoverib@ietf.org
Subject: Re: [Ipoverib] Attempt for a Consensus call for LIDs in the ARP response
Message-ID: <20020125143454.E22989@topspincom.com>
References: <HHEDIJGOEFLPCPNILMPAIEHACFAA.bill@strahm.net> <HHEDIJGOEFLPCPNILMPAGEHJCFAA.bill@strahm.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <HHEDIJGOEFLPCPNILMPAGEHJCFAA.bill@strahm.net>; from bill@strahm.net on Fri, Jan 25, 2002 at 01:13:42PM -0800
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

On Fri, Jan 25, 2002 at 01:13:42PM -0800, Bill Strahm wrote:
> [WG Hat still on]
> Just for the record,
> Can I get people who agree that the LID should not be apart of the Link
> Address to affirm this decision ?

  I agree, the LID should not be a part of the link address.

-Libor

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 25 18:05:13 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA17530
	for <ipoverib-archive@odin.ietf.org>; Fri, 25 Jan 2002 18:05:08 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id SAA16737
	for ipoverib-archive@odin.ietf.org; Fri, 25 Jan 2002 18:05:01 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA16704;
	Fri, 25 Jan 2002 18:04:24 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA16669
	for <ipoverib@ns.ietf.org>; Fri, 25 Jan 2002 18:04:22 -0500 (EST)
Received: from mxout1.netvision.net.il (mxout1.netvision.net.il [194.90.9.20])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA17519
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 18:04:18 -0500 (EST)
Received: from vaio505he ([139.92.208.22]) by mxout1.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GQI00C7AO2BJD@mxout1.netvision.net.il> for ipoverib@ietf.org;
 Sat, 26 Jan 2002 01:03:49 +0200 (IST)
Date: Sat, 26 Jan 2002 01:00:27 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: RE: [ipoverib] moving forward with the open Issues - end node
 configuration
In-reply-to: 
 <2E33960095B58E40A4D3345AB9F65EC10285F690@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
To: "'Aditya Dube'" <adube@windows.microsoft.com>, ipoverib@ietf.org
Reply-to: yaronh@voltaire.com
Message-id: <001f01c1a5f4$15729a80$a3d05c8b@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7BIT


I agree that some method should be specified in the future
However it is not so trivial to do it right now
a. There are issues between IBTA & IETF on things like service records etc'
b. You could have few VLAN's on the same IB subnet, how would you
distinguish between them, etc' (maybe a service ID range)

I would tend to go for an RFC with a static configuration, with known
defaults like:
The default P_Key value, 1500 MTU
Work on a follow on proposal either in IBTA or IETF to address the
automation

You probably wonder why 1500, some of the reasons originally for determining
that IPoIB supporting devices will have minimum MTU of 2K was to maintain
similarity with Ethernet
Since in IB you have either 1K or 2K, 1500 falls in the 2K domain, so the
device must support 2K, not to say the MTU should by 2K

Why 1500 again, lets say you have an Ethernet to IB bridge/router it will
have to fragment/reassemble the packets
This will create a lot of overhead on both the hosts and the routers that
will significantly affect performance

(it might be 1400 & something)

Yaron

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Aditya Dube
Sent: Friday, January 25, 2002 9:17 PM
To: ipoverib@ietf.org
Subject: RE: [ipoverib] moving forward with the open Issues - end node
configuration



The current IPOverIB spec requires that the P_Key and MTU be configured
on each node.

The IPOverIB spec should provide a default method for an end-node to
come online with no IPoIB-specific configuration.

During initialization, a node could query an IPOverIB specific
ServiceRecord (Table 164, pg 685). This query will return a P_Key and
ServiceSpecificFlags. The ServiceSpecificFlags can specify whether 2048
or 4096 is the link MTU.

The ServiceRecord's ServiceID should have a default value specific to
IPOverIB. This number could be 0xFFFF:4000 in accordance with vol3
section on "ServiceID Namespace" (Table 128, pg 215).

The ServiceRecord needs to be created just as the all-node or broadcast
MCGroup is created, i.e. by the first node on the network.

If the Network Admin wishes to put the end-node on a different P_Key,
then the Admin may override the values returned by the ServiceRecord.
That is outside the scope of my proposal.


Aditya Dube



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 25 18:11:21 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA17604
	for <ipoverib-archive@odin.ietf.org>; Fri, 25 Jan 2002 18:11:02 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id SAA16874
	for ipoverib-archive@odin.ietf.org; Fri, 25 Jan 2002 18:11:02 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA16853;
	Fri, 25 Jan 2002 18:10:20 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA16816
	for <ipoverib@ns.ietf.org>; Fri, 25 Jan 2002 18:10:18 -0500 (EST)
Received: from thalia.fm.intel.com (fmfdns02.fm.intel.com [132.233.247.11])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA17591
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 18:10:14 -0500 (EST)
Received: from fmsmsxvs041.fm.intel.com (fmsmsxv041-1.fm.intel.com [132.233.48.109])
	by thalia.fm.intel.com (8.9.1a+p1/8.9.1/d: relay.m4,v 1.49 2002/01/25 02:16:58 root Exp $) with SMTP id XAA25601
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 23:10:16 GMT
Received: from fmsmsx29.FM.INTEL.COM ([132.233.42.29])
 by fmsmsxvs041.fm.intel.com (NAVGW 2.5.1.16) with SMTP id M2002012515111129340
 ; Fri, 25 Jan 2002 15:11:11 -0800
Received: by fmsmsx29.fm.intel.com with Internet Mail Service (5.5.2653.19)
	id <DVHBZ07P>; Fri, 25 Jan 2002 15:10:16 -0800
Message-ID: <FD2423AA68A7D511A5A20002A50729E1012062C7@orsmsx115.jf.intel.com>
From: "Grun, Paul" <paul.grun@intel.com>
To: "'Bill Strahm'" <bill@strahm.net>, ipoverib@ietf.org
Subject: RE: [Ipoverib] Attempt for a Consensus call for LIDs in the ARP r
	esponse
Date: Fri, 25 Jan 2002 15:10:15 -0800
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="windows-1255"
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

I strongly agree that the LID should not be included in the ARP response.
-paul

> -----Original Message-----
> From: Bill Strahm [mailto:bill@strahm.net]
> Sent: Friday, January 25, 2002 1:14 PM
> To: ipoverib@ietf.org
> Subject: RE: [Ipoverib] Attempt for a Consensus call for LIDs 
> in the ARP response
> 
> 
> [WG Hat still on]
> Just for the record,
> Can I get people who agree that the LID should not be apart 
> of the Link
> Address to affirm this decision ?
> 
> Bill
> [/Hat]
> 
> -----Original Message-----
> From: ipoverib-admin@ietf.org 
> [mailto:ipoverib-admin@ietf.org]On Behalf Of
> Bill Strahm
> Sent: Thursday, January 24, 2002 11:28 PM
> To: ipoverib@ietf.org
> Subject: [Ipoverib] Attempt for a Consensus call for LIDs in the ARP
> response
> 
> 
> 
> [Working Group Chair Hat firmly in place]
> I would like to make a concensus call on not having a LID in the ARP
> response.  I believe the two main proponents for having the 
> LID have found
> other ways of getting the LID that they need and do not need 
> it in the ARP
> response.
> 
> Who has remaining technical reasons for maintaining the LID in the ARP
> response
> [End WG Chair message]
> 
> Bill
> 
> 
> -----Original Message-----
> From: ipoverib-admin@ietf.org 
> [mailto:ipoverib-admin@ietf.org]On Behalf Of
> Yaron Haviv
> Sent: Thursday, January 24, 2002 1:24 PM
> To: ipoverib@ietf.org
> Subject: [ipoverib] moving forward with the open Issues
> 
> 
> Hi all
> 
> By now it looks to me like the LID issues is going to be resolved
> By just dropping the LID from the ARP, and using the one in the LRH
> And there is a consensus around it also from Vivek
> 
> For the capability bits, I suggested reserving the bits for 
> future use (in
> this RFC)
> After we finish with this RFC we could start discussing the bits to be
> defined in a separate RFC
> This will enable every one to agree at least for the UD RFC 
> on every thing
> 
> Are there any other open issues that are not in consensus right now ?
> Does any one want to bring a new topic ?
> 
> If these are the last points it would really help close the 
> standard quickly
> 
> Yaron
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 25 18:26:25 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA17842
	for <ipoverib-archive@odin.ietf.org>; Fri, 25 Jan 2002 18:26:20 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id SAA17324
	for ipoverib-archive@odin.ietf.org; Fri, 25 Jan 2002 18:26:23 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA17303;
	Fri, 25 Jan 2002 18:25:50 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA17269
	for <ipoverib@ns.ietf.org>; Fri, 25 Jan 2002 18:25:48 -0500 (EST)
Received: from omegaband.com (faust.seagullsemi.com [206.196.68.34])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA17828
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 18:25:44 -0500 (EST)
Received: from seahorsepc [207.8.52.226] by omegaband.com with ESMTP
  (SMTPD32-7.04) id A7D41019011C; Fri, 25 Jan 2002 17:18:44 -0600
From: "Ashwin Kamath" <kamath@omegaband.com>
To: "Bill Strahm" <bill@strahm.net>, <ipoverib@ietf.org>
Subject: RE: [Ipoverib] Attempt for a Consensus call for LIDs in the ARP response
Date: Fri, 25 Jan 2002 17:23:51 -0600
Message-ID: <IFEKJBDLPFOEGGPIIGMFMEHJCAAA.kamath@omegaband.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="windows-1255"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0)
Importance: Normal
In-Reply-To: <HHEDIJGOEFLPCPNILMPAGEHJCFAA.bill@strahm.net>
X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

I agree that LID should not be part of the link layer address.

-Ashwin.

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Bill Strahm
Sent: Friday, January 25, 2002 3:14 PM
To: ipoverib@ietf.org
Subject: RE: [Ipoverib] Attempt for a Consensus call for LIDs in the ARP
response


[WG Hat still on]
Just for the record,
Can I get people who agree that the LID should not be apart of the Link
Address to affirm this decision ?

Bill
[/Hat]

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Bill Strahm
Sent: Thursday, January 24, 2002 11:28 PM
To: ipoverib@ietf.org
Subject: [Ipoverib] Attempt for a Consensus call for LIDs in the ARP
response



[Working Group Chair Hat firmly in place]
I would like to make a concensus call on not having a LID in the ARP
response.  I believe the two main proponents for having the LID have found
other ways of getting the LID that they need and do not need it in the ARP
response.

Who has remaining technical reasons for maintaining the LID in the ARP
response
[End WG Chair message]

Bill


-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Yaron Haviv
Sent: Thursday, January 24, 2002 1:24 PM
To: ipoverib@ietf.org
Subject: [ipoverib] moving forward with the open Issues


Hi all

By now it looks to me like the LID issues is going to be resolved
By just dropping the LID from the ARP, and using the one in the LRH
And there is a consensus around it also from Vivek

For the capability bits, I suggested reserving the bits for future use (in
this RFC)
After we finish with this RFC we could start discussing the bits to be
defined in a separate RFC
This will enable every one to agree at least for the UD RFC on every thing

Are there any other open issues that are not in consensus right now ?
Does any one want to bring a new topic ?

If these are the last points it would really help close the standard quickly

Yaron


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 25 18:29:17 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA17894
	for <ipoverib-archive@odin.ietf.org>; Fri, 25 Jan 2002 18:29:12 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id SAA17447
	for ipoverib-archive@odin.ietf.org; Fri, 25 Jan 2002 18:29:16 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA17426;
	Fri, 25 Jan 2002 18:28:42 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA17390
	for <ipoverib@ns.ietf.org>; Fri, 25 Jan 2002 18:28:40 -0500 (EST)
Received: from pdxpo.dsl-only.net (pdxpo.dsl-only.net [63.105.16.3])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA17889
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 18:28:36 -0500 (EST)
Received: from bonfire (unverified [63.105.18.201]) by pdxpo.dsl-only.net
 (Rockliffe SMTPRA 4.5.6) with SMTP id <B0009392662@pdxpo.dsl-only.net>;
 Fri, 25 Jan 2002 15:27:18 -0800
From: "Bill Strahm" <bill@strahm.net>
To: <yaronh@voltaire.com>, "'Aditya Dube'" <adube@windows.microsoft.com>,
        <ipoverib@ietf.org>
Subject: RE: [ipoverib] moving forward with the open Issues - end node configuration
Date: Fri, 25 Jan 2002 15:37:17 -0800
Message-ID: <HHEDIJGOEFLPCPNILMPAMEHMCFAA.bill@strahm.net>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0)
Importance: Normal
In-Reply-To: <001f01c1a5f4$15729a80$a3d05c8b@vaio505he>
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

The fact that there are transports in the middle of the network that do not
support larger MTUs is not a reason to lower your MTU value...

TCP will find the correct Path MTU, and UDP doesn't care.  Therefor why
should I limit my MTU on the local subnet because someone MIGHT have an MTU
much lower.  I would rather state the largest local wire MTU (2K - overhead)
and let the protocols lower it automatically...

Bill

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Yaron Haviv
Sent: Friday, January 25, 2002 3:00 PM
To: 'Aditya Dube'; ipoverib@ietf.org
Subject: RE: [ipoverib] moving forward with the open Issues - end node
configuration



I agree that some method should be specified in the future
However it is not so trivial to do it right now
a. There are issues between IBTA & IETF on things like service records etc'
b. You could have few VLAN's on the same IB subnet, how would you
distinguish between them, etc' (maybe a service ID range)

I would tend to go for an RFC with a static configuration, with known
defaults like:
The default P_Key value, 1500 MTU
Work on a follow on proposal either in IBTA or IETF to address the
automation

You probably wonder why 1500, some of the reasons originally for determining
that IPoIB supporting devices will have minimum MTU of 2K was to maintain
similarity with Ethernet
Since in IB you have either 1K or 2K, 1500 falls in the 2K domain, so the
device must support 2K, not to say the MTU should by 2K

Why 1500 again, lets say you have an Ethernet to IB bridge/router it will
have to fragment/reassemble the packets
This will create a lot of overhead on both the hosts and the routers that
will significantly affect performance

(it might be 1400 & something)

Yaron

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Aditya Dube
Sent: Friday, January 25, 2002 9:17 PM
To: ipoverib@ietf.org
Subject: RE: [ipoverib] moving forward with the open Issues - end node
configuration



The current IPOverIB spec requires that the P_Key and MTU be configured
on each node.

The IPOverIB spec should provide a default method for an end-node to
come online with no IPoIB-specific configuration.

During initialization, a node could query an IPOverIB specific
ServiceRecord (Table 164, pg 685). This query will return a P_Key and
ServiceSpecificFlags. The ServiceSpecificFlags can specify whether 2048
or 4096 is the link MTU.

The ServiceRecord's ServiceID should have a default value specific to
IPOverIB. This number could be 0xFFFF:4000 in accordance with vol3
section on "ServiceID Namespace" (Table 128, pg 215).

The ServiceRecord needs to be created just as the all-node or broadcast
MCGroup is created, i.e. by the first node on the network.

If the Network Admin wishes to put the end-node on a different P_Key,
then the Admin may override the values returned by the ServiceRecord.
That is outside the scope of my proposal.


Aditya Dube



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 25 18:53:51 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA18523
	for <ipoverib-archive@odin.ietf.org>; Fri, 25 Jan 2002 18:53:51 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id SAA18532
	for ipoverib-archive@odin.ietf.org; Fri, 25 Jan 2002 18:53:54 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA18481;
	Fri, 25 Jan 2002 18:52:59 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA18449
	for <ipoverib@ns.ietf.org>; Fri, 25 Jan 2002 18:52:48 -0500 (EST)
Received: from calliope1.fm.intel.com (fmfdns01.fm.intel.com [132.233.247.10])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA18476
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 18:50:25 -0500 (EST)
Received: from fmsmsxvs043.fm.intel.com (fmsmsxv043-1.fm.intel.com [132.233.48.128])
	by calliope1.fm.intel.com (8.9.1a+p1/8.9.1/d: relay.m4,v 1.49 2002/01/25 02:16:58 root Exp $) with SMTP id XAA21556
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 23:50:28 GMT
Received: from fmsmsx28.fm.intel.com ([132.233.42.28])
 by fmsmsxvs043.fm.intel.com (NAVGW 2.5.1.16) with SMTP id M2002012515493725153
 ; Fri, 25 Jan 2002 15:49:37 -0800
Received: by fmsmsx28.fm.intel.com with Internet Mail Service (5.5.2653.19)
	id <DVG6TVTY>; Fri, 25 Jan 2002 15:50:27 -0800
Message-ID: <D9223EB959A5D511A98F00508B68C20C21A884@ORSMSX108>
From: "Larsen, Roy K" <roy.k.larsen@intel.com>
To: "'Culley, Paul'" <Paul.Culley@compaq.com>, yaronh@voltaire.com,
        ipoverib@ietf.org
Subject: RE: [ipoverib] resolving the Ethertype issue
Date: Fri, 25 Jan 2002 15:50:26 -0800
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="windows-1255"
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

embedded <rkl>

-----Original Message-----
From: Culley, Paul [mailto:Paul.Culley@compaq.com]
Sent: Friday, January 25, 2002 1:52 PM
To: Larsen, Roy K; yaronh@voltaire.com; ipoverib@ietf.org
Subject: RE: [ipoverib] resolving the Ethertype issue


embedded...

Paul R. Culley
Compaq Sr. Fellow
281-514-5543


-----Original Message-----
From: Larsen, Roy K [mailto:roy.k.larsen@intel.com]
Sent: Friday, January 25, 2002 2:36 PM
To: 'yaronh@voltaire.com'; ipoverib@ietf.org
Subject: RE: [ipoverib] resolving the Ethertype issue


Although my feelings aren't strong enough to drag the reflector into a never
ending rat hole, I believe the IPoIB draft should specify a
protoco/"Ethertype" as the first field of the payload which is treated as an
IPoIB protocol specific field.  The reasons are as follows:

1.  Interesting OSs require the link driver to identify the destination
protocol of received frames.
<prc>
True, but at the moment, only two protocols are in our charter to
differentiate.  These could be easily identified by examining the first
nibble of the packet; IP has 4 or 6 in the first nibble, surly we can chose
some other value to id ARP.  Or you can use the following QP proposal.  Or
both, if the implementation wants both protocols on one QP.
</prc> 
<rkl>
Actually, there are three, ARP, IPv4, IPv6.  Each are treated as distinct
protocols in the OSs I'm interested in.  I've already expressed my view on
the integrity of peeking into protocol headers.
</rkl>

2.  QPNs do not provide a clean mechanism to do this because, at a minimum,
both ARP responses and IPv4 frames must share the same QPN.
<prc>
 I don't see why this must be so.  I seems perfectly possible to include an
ARP return QP in the ARP request that would be used in the ARP response.
The ARP response can, in turn include the IP QP for use by the IP traffic.
If gratuitous ARP and ARP snooping are allowed/supported, then the ARP
request should also include the separate IP QP.  This solves the problem for
the two protocols that our group is chartered to solve.
</prc>
<rkl>
Are you suggesting that a single request/response transaction should not
resolve the IPv4 endpoints on both sides?  If so, we need to explicitly
state that the QPN in the source hardware address field of the ARP request
is only valid for ARP responses and MUST NOT be assumed valid for IPv4
frames.  I assumed this was not the case since it would increase ARP traffic
by a factor of 2.  Surely a protocol type field is a much smaller price to
pay than requiring every node to initiate an ARP even for nodes it has
already been in ARP-replay contact with.  I'm not even sure my OSs can deal
with this..... 
</rkl>

3.  Requiring the IPoIB driver to peek into an upper-level protocol frame is
architecturally displeasing. (Read that as an euphemism for KLUDGE.)
<prc>
Whatever works...
</prc>
<rkl>
An engineering approach I've used in the past, but only when left with no
cleaner solution.
</rkl>

4.  Arguments that some day the IB header may contain such a field are
irrelevant in defining a protocol today.  (Same reasoning we've seen for IB
subnet routing issues.)

Roy

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Friday, January 25, 2002 9:46 AM
To: ipoverib@ietf.org
Subject: [ipoverib] resolving the Ethertype issue



As Arvind mentioned the Ethertype field issue is one of the last items to
agree up on

Currently there are 2 proposals:
1.	(Vivek) no need for ethertype, we could see its an ARP from the
first
fields in the Packet (IP Ver)
2.	put the Ethertype field in the first few bytes of each message

I personally don't see it as a critical decision choosing 1 or 2, the only
thing that Vivek proposal might not enable is doing Ethernet over IB which
is not in the scope of this RFC anyway
I would tend to agree with Vivek for the simplicity reasons (not the
performance ones)

If any one has a firm reason why to put the Ethertype field in the payload
please make your arguments
And it would help in settling this matter too
Please try to give real reasons in this case too, and not things like “it is
better”, etc'

Yaron



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 25 19:17:47 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA18728
	for <ipoverib-archive@odin.ietf.org>; Fri, 25 Jan 2002 19:17:47 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id TAA19372
	for ipoverib-archive@odin.ietf.org; Fri, 25 Jan 2002 19:17:48 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA19335;
	Fri, 25 Jan 2002 19:16:11 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA19305
	for <ipoverib@ns.ietf.org>; Fri, 25 Jan 2002 19:16:08 -0500 (EST)
Received: from thalia.fm.intel.com (fmfdns02.fm.intel.com [132.233.247.11])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA18721
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 19:16:05 -0500 (EST)
Received: from fmsmsxvs041.fm.intel.com (fmsmsxv041-1.fm.intel.com [132.233.48.109])
	by thalia.fm.intel.com (8.9.1a+p1/8.9.1/d: relay.m4,v 1.49 2002/01/25 02:16:58 root Exp $) with SMTP id AAA17488
	for <ipoverib@ietf.org>; Sat, 26 Jan 2002 00:16:06 GMT
Received: from FMSMSX016.fm.intel.com ([132.233.42.195])
 by fmsmsxvs041.fm.intel.com (NAVGW 2.5.1.16) with SMTP id M2002012516170111137
 ; Fri, 25 Jan 2002 16:17:01 -0800
Received: by fmsmsx016.fm.intel.com with Internet Mail Service (5.5.2653.19)
	id <DVGXCARF>; Fri, 25 Jan 2002 16:16:05 -0800
Message-ID: <D9223EB959A5D511A98F00508B68C20C21A885@ORSMSX108>
From: "Larsen, Roy K" <roy.k.larsen@intel.com>
To: "'yaronh@voltaire.com'" <yaronh@voltaire.com>, ipoverib@ietf.org
Subject: RE: [ipoverib] resolving the Ethertype issue
Date: Fri, 25 Jan 2002 16:16:03 -0800
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="windows-1255"
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Yaron,

I believe I addressed your comments in a response to Paul with the exception
of #3.

In the OSs I work with, none of the link drivers wonder into the buffer
beyond their own header.  For example, the Ethernet header includes the
len/type field which we have been referring to as the "ethertype".  It
belongs to the link layer to specify the destination protocol of the
payload.  Supplying destination protocol identifiers so a packet can be
routed from service provider to service user within the network stack is
basic networking 101.  All protocol layers use this mechanism which is why
I'm a little surprised at the resistance this concept is getting from the
list.  The network stack of interesting OSs have simple and elegant
mechanisms to "strip" protocol headers as it traverses the stack so it can't
be a complexity issue.

Roy

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Friday, January 25, 2002 2:21 PM
To: 'Larsen, Roy K'; ipoverib@ietf.org
Subject: RE: [ipoverib] resolving the Ethertype issue


Roy, similar arguments like Paul, see embadded

 -----Original Message-----
From: 	ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]  On Behalf
Of Larsen, Roy K
Sent:	Friday, January 25, 2002 10:36 PM
To:	'yaronh@voltaire.com'; ipoverib@ietf.org
Subject:	RE: [ipoverib] resolving the Ethertype issue

Although my feelings aren't strong enough to drag the reflector into a never
ending rat hole, I believe the IPoIB draft should specify a
protoco/"Ethertype" as the first field of the payload which is treated as an
IPoIB protocol specific field.  The reasons are as follows:

1.  Interesting OSs require the link driver to identify the destination
protocol of received frames.
<yh> since you know its IP, you could fill it out

2.  QPNs do not provide a clean mechanism to do this because, at a minimum,
both ARP responses and IPv4 frames must share the same QPN.
<yh> the idea Vivek suggested is to put zero in the IP ver field in cases of
an ARP, this resolves the problem

3.  Requiring the IPoIB driver to peek into an upper-level protocol frame is
architecturally displeasing. (Read that as an euphemism for KLUDGE.)
<yh> also Ethernet IP drivers do some of it today, not sure I get what you
mean as upper-level in this case

4.  Arguments that some day the IB header may contain such a field are
irrelevant in defining a protocol today.  (Same reasoning we've seen for IB
subnet routing issues.)
<yh> this was not assumed, just was argued that it might also solve the
problem of limiting it to IP (& not also other Ethernet stuff)

Roy

-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: Friday, January 25, 2002 9:46 AM
To: ipoverib@ietf.org
Subject: [ipoverib] resolving the Ethertype issue



As Arvind mentioned the Ethertype field issue is one of the last items to
agree up on

Currently there are 2 proposals:
1.	(Vivek) no need for ethertype, we could see its an ARP from the
first
fields in the Packet (IP Ver)
2.	put the Ethertype field in the first few bytes of each message

I personally don't see it as a critical decision choosing 1 or 2, the only
thing that Vivek proposal might not enable is doing Ethernet over IB which
is not in the scope of this RFC anyway
I would tend to agree with Vivek for the simplicity reasons (not the
performance ones)

If any one has a firm reason why to put the Ethertype field in the payload
please make your arguments
And it would help in settling this matter too
Please try to give real reasons in this case too, and not things like "it is
better", etc'

Yaron



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Fri Jan 25 20:21:53 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA19402
	for <ipoverib-archive@odin.ietf.org>; Fri, 25 Jan 2002 20:21:53 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id UAA21444
	for ipoverib-archive@odin.ietf.org; Fri, 25 Jan 2002 20:21:55 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA21415;
	Fri, 25 Jan 2002 20:20:49 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA21383
	for <ipoverib@ns.ietf.org>; Fri, 25 Jan 2002 20:20:47 -0500 (EST)
Received: from mail5.microsoft.com (mail5.microsoft.com [131.107.3.121])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA19373
	for <ipoverib@ietf.org>; Fri, 25 Jan 2002 20:20:44 -0500 (EST)
Received: from inet-vrs-05.redmond.corp.microsoft.com ([157.54.6.148]) by mail5.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Fri, 25 Jan 2002 17:20:15 -0800
Received: from 157.54.8.155 by inet-vrs-05.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Fri, 25 Jan 2002 17:20:14 -0800
Received: from red-imc-02.redmond.corp.microsoft.com ([157.54.9.107]) by inet-hub-04.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Fri, 25 Jan 2002 17:20:14 -0800
Received: from win-imc-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.39]) by red-imc-02.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Fri, 25 Jan 2002 17:20:13 -0800
Received: from win-msg-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.52]) by win-imc-01.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3588.0);
	 Fri, 25 Jan 2002 17:17:58 -0800
X-MimeOLE: Produced By Microsoft Exchange V6.0.6132.0
Content-Class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Subject: RE: [ipoverib] moving forward with the open Issues - end node configuration
Date: Fri, 25 Jan 2002 17:17:58 -0800
Message-ID: <2E33960095B58E40A4D3345AB9F65EC10285F6A3@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
Thread-Topic: [ipoverib] moving forward with the open Issues - end node configuration
Thread-Index: AcGl22M+PI2BOhwfQySVQlqCSGNstwADH71Q
From: "Aditya Dube" <adube@windows.microsoft.com>
To: <ipoverib@ietf.org>
X-OriginalArrivalTime: 26 Jan 2002 01:17:58.0411 (UTC) FILETIME=[4A451DB0:01C1A607]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id UAA21384
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

Hi folks

The Software Transport Verbs have this limitation:

1) To Attach to an MCGroup, one needs a Queue Pair
2) To create a Queue Pair, one needs a P_Key.

Therefore, a node needs a P_Key before it can join any MCGroup or create
a Queue Pair. I propose that there be a well-known default ServiceRecord
that we use to configure the P_Key and MTU.

On another note, the Administrator is welcome to override our defaults
and set up as many subnets, VLANs or P_Keys as he or she desires. That
is outside the scope of my proposal.


Aditya Dube

-----Original Message-----
From: Vivek Kashyap [mailto:vivk@us.ibm.com] 
Sent: Friday, January 25, 2002 12:04 PM
To: ipoverib@ietf.org
Subject: RE: [ipoverib] moving forward with the open Issues - end node
configuration


<VK> I am not sure what you gain by that. The P_Key is programmed into
the
port by the SM on startup. You already have it in the P_Key tables.  How
you determine the P_Key to use is upto your administrator.

If there are multiple IPoIB subnets over the fabric - how do you know
which
serviceID to choose ? These will have to be specified by the
administrator.
So the administrator might as well specify the P_Key per IP interface.
You
may automate the process by ensuring that your interface tries all the
P_Keys in the relevant port.

The P_Key then allows you to determine the all-nodes multicast/broadcast
address (as defined in the multicast draft). This address that has to be
setup by the administrator (first node/other method). Every interface is
required to join this address. The MTU returned on the join request is
the
MTU of the interface. So the MTU is automatically configured. The join
request is less than 256 bytes and hence will be received by the SM
whatever be the link MTU/path MTU.

Zero-conf can be done in the following way though:

      The P_Keys are in the P_Key tables. Let, by default, there exist
an
MGID formed by zeroing out the link-id (P_Key) field in the MGID. So for
IPv4 it will be FF12::255.255.255.255. This group is joined by the
interfaces. If they are in the partition of this MGID they will receive
the
right P_Key to use for the subnet. They will also get the subnet MTU.
There
can be only one such subnet in the IB subnet. If multiple IPoIB links
are
defined then this scheme won't be universal but it allows complete
automation in the default case.

For multiple IPoIB links:

      If the IPoIB ports can support at least two P_Keys (since we are
looking at multiple IPoIB links) we can always define a default subnet
(as
above). A DHCP server can be put on this IPoIB link. A node then
enquires
this DHCP server for the P_Key. I can suggest the DHCP
extensions/process
required if needed.

Comments ?

Vivek
<VK>


The current IPOverIB spec requires that the P_Key and MTU be configured
on each node.

The IPOverIB spec should provide a default method for an end-node to
come online with no IPoIB-specific configuration.

During initialization, a node could query an IPOverIB specific
ServiceRecord (Table 164, pg 685). This query will return a P_Key and
ServiceSpecificFlags. The ServiceSpecificFlags can specify whether 2048
or 4096 is the link MTU.

The ServiceRecord's ServiceID should have a default value specific to
IPOverIB. This number could be 0xFFFF:4000 in accordance with vol3
section on "ServiceID Namespace" (Table 128, pg 215).

The ServiceRecord needs to be created just as the all-node or broadcast
MCGroup is created, i.e. by the first node on the network.

If the Network Admin wishes to put the end-node on a different P_Key,
then the Admin may override the values returned by the ServiceRecord.
That is outside the scope of my proposal.


Aditya Dube



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib





_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Sat Jan 26 13:19:23 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA09408
	for <ipoverib-archive@odin.ietf.org>; Sat, 26 Jan 2002 13:19:18 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id NAA01199
	for ipoverib-archive@odin.ietf.org; Sat, 26 Jan 2002 13:19:21 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA01178;
	Sat, 26 Jan 2002 13:17:59 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA01148
	for <ipoverib@optimus.ietf.org>; Sat, 26 Jan 2002 13:17:57 -0500 (EST)
Received: from host32.cedant.com (host32.cedant.com [209.239.32.20])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA09391
	for <ipoverib@ietf.org>; Sat, 26 Jan 2002 13:17:54 -0500 (EST)
Received: from sahara ([12.81.9.91])
	by host32.cedant.com (8.10.2/8.10.2) with ESMTP id g0QIHde29992;
	Sat, 26 Jan 2002 13:17:39 -0500
Message-ID: <200201261033070333.4D9E242A@opulentsystems.com>
In-Reply-To: <HHEDIJGOEFLPCPNILMPAMEHMCFAA.bill@strahm.net>
References: <HHEDIJGOEFLPCPNILMPAMEHMCFAA.bill@strahm.net>
X-Mailer: Calypso Version 3.30.00.00 (4)
Date: Sat, 26 Jan 2002 10:33:07 -0800
Reply-To: sganguly@opulentsystems.com
From: "Sukanta Ganguly" <sganguly@opulentsystems.com>
To: "Bill Strahm" <bill@strahm.net>, "Yaron Haviv" <yaronh@voltaire.com>,
        "'Aditya Dube'" <adube@windows.microsoft.com>, ipoverib@ietf.org
Subject: RE: [ipoverib] moving forward with the open Issues - end node
  configuration
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id NAA01149
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

Bill,
  By forcing a larger MTU value an overhead in injected in the path, where nodes not in the position to support larger MTU have to fragment/defrag the packets. This is processing overhead. From the position of sending the packet it may not seem like one. But as far as packet being delivered from one end to the other the overhead is injected. The Path MTU mechanism is to determine the lowest MTU for all the individual links and setting that to be your MTU. No packet frag/defrag will happen and hence not buffereing expected in the path for any intermediate nodes. 

SG

*********** REPLY SEPARATOR  ***********

On 1/25/2002 at 3:37 PM Bill Strahm wrote:

>The fact that there are transports in the middle of the network that do
>not
>support larger MTUs is not a reason to lower your MTU value...
>
>TCP will find the correct Path MTU, and UDP doesn't care.  Therefor why
>should I limit my MTU on the local subnet because someone MIGHT have an MTU
>much lower.  I would rather state the largest local wire MTU (2K -
>overhead)
>and let the protocols lower it automatically...
>
>Bill
>
>-----Original Message-----
>From: ipove  rib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
>Of Yaron Haviv
>Sent: Friday, January 25, 2002 3:00 PM
>To: 'Aditya Dube'; ipoverib@ietf.org
>Subject: RE: [ipoverib] moving forward with the open Issues - end node
>configuration
>
>
>
>I agree that some method should be specified in the future
>However it is not so trivial to do it right now
>a. There are issues between IBTA & IETF on things like service records etc'
>b. You could have few VLAN's on the same IB subnet, how would you
>distinguish between them, etc' (maybe a service ID range)
>
>I would tend to go for an RFC with a static configuration, with known
>defaults like:
>The default P_Key value, 1500 MTU
>Work on a follow on proposal either in IBTA or IETF to address the
>automation
>
>You probably wonder why 1500, some of the reasons originally for
>determining
>that IPoIB supporting devices will have minimum MTU of 2K was to maintain
>similarity with Ethernet
>Since in IB you have either 1K or 2K, 1500 falls in the 2K domain, so the
>device must support 2K, not to say the MTU should by 2K
>
>Why 1500 again, lets say you have an Ethernet to IB bridge/router it will
>have to fragment/reassemble the packets
>This will create a lot of overhead on both the hosts and the routers that
>will significantly affect performance
>
>(it might be 1400 & something)
>
>Yaron
>
>-----Original Message-----
>From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
>Aditya Dube
>Sent: Friday, January 25, 2002 9:17 PM
>To: ipoverib@ietf.org
>Subject: RE: [ipoverib] moving forward with the open Issues - end node
>configuration
>
>
>
>The current IPOverIB spec requires that the P_Key and MTU be configured
>on each node.
>
>The IPOverIB spec should provide a default method for an end-node to
>come online with no IPoIB-specific configuration.
>
>During initialization, a node could query an IPOverIB specific
>ServiceRecord (Table 164, pg 685). This query will return a P_Key and
>ServiceSpecificFlags. The ServiceSpecificFlags can specify whether 2048
>or 4096 is the link MTU.
>
>The ServiceRecord's ServiceID should have a default value specific to
>IPOverIB. This number could be 0xFFFF:4000 in accordance with vol3
>section on "ServiceID Namespace" (Table 128, pg 215).
>
>The ServiceRecord needs to be created just as the all-node or broadcast
>MCGroup is created, i.e. by the first node on the network.
>
>If the Network Admin wishes to put the end-node on a different P_Key,
>then the Admin may override the values returned by the ServiceRecord.
>That is outside the scope of my proposal.
>
>
>Aditya Dube
>
>
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>https://www1.ietf.org/mailman/listinfo/ipoverib
>
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>https://www1.ietf.org/mailman/listinfo/ipoverib
>
>
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>https://www1.ietf.org/mailman/listinfo/ipoverib




_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Sat Jan 26 16:54:51 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA11110
	for <ipoverib-archive@odin.ietf.org>; Sat, 26 Jan 2002 16:54:51 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id QAA06993
	for ipoverib-archive@odin.ietf.org; Sat, 26 Jan 2002 16:54:56 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA06976;
	Sat, 26 Jan 2002 16:54:20 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id QAA06947
	for <ipoverib@optimus.ietf.org>; Sat, 26 Jan 2002 16:54:18 -0500 (EST)
Received: from pdxpo.dsl-only.net (pdxpo.dsl-only.net [63.105.16.3])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA11105
	for <ipoverib@ietf.org>; Sat, 26 Jan 2002 16:54:12 -0500 (EST)
Received: from bonfire (unverified [63.105.18.201]) by pdxpo.dsl-only.net
 (Rockliffe SMTPRA 4.5.6) with SMTP id <B0009406695@pdxpo.dsl-only.net>;
 Sat, 26 Jan 2002 13:52:56 -0800
From: "Bill Strahm" <bill@strahm.net>
To: <sganguly@opulentsystems.com>, "Yaron Haviv" <yaronh@voltaire.com>,
        "'Aditya Dube'" <adube@windows.microsoft.com>, <ipoverib@ietf.org>
Subject: RE: [ipoverib] moving forward with the open Issues - end node  configuration
Date: Sat, 26 Jan 2002 14:02:28 -0800
Message-ID: <HHEDIJGOEFLPCPNILMPAMEICCFAA.bill@strahm.net>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0)
Importance: Normal
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
In-Reply-To: <200201261033070333.4D9E242A@opulentsystems.com>
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

I know all of this...

No answer this.

In the case of TCP, PMTU will correctly determine the PATH MTU, not the
local link MTU and you don't have frag/reassembly problems in the network

In the case of UDP, you will have to handle these anyway, because UDP
packets are just as big as they are...

Now tell me what you have bought me by reducing the local MTU below its
maximum ?

Bill

-----Original Message-----
From: Sukanta Ganguly [mailto:sganguly@opulentsystems.com]
Sent: Saturday, January 26, 2002 10:33 AM
To: Bill Strahm; Yaron Haviv; 'Aditya Dube'; ipoverib@ietf.org
Subject: RE: [ipoverib] moving forward with the open Issues - end node
configuration


Bill,
  By forcing a larger MTU value an overhead in injected in the path, where
nodes not in the position to support larger MTU have to fragment/defrag the
packets. This is processing overhead. From the position of sending the
packet it may not seem like one. But as far as packet being delivered from
one end to the other the overhead is injected. The Path MTU mechanism is to
determine the lowest MTU for all the individual links and setting that to be
your MTU. No packet frag/defrag will happen and hence not buffereing
expected in the path for any intermediate nodes.

SG

*********** REPLY SEPARATOR  ***********

On 1/25/2002 at 3:37 PM Bill Strahm wrote:

>The fact that there are transports in the middle of the network that do
>not
>support larger MTUs is not a reason to lower your MTU value...
>
>TCP will find the correct Path MTU, and UDP doesn't care.  Therefor why
>should I limit my MTU on the local subnet because someone MIGHT have an MTU
>much lower.  I would rather state the largest local wire MTU (2K -
>overhead)
>and let the protocols lower it automatically...
>
>Bill
>
>-----Original Message-----
>From: ipove  rib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
>Of Yaron Haviv
>Sent: Friday, January 25, 2002 3:00 PM
>To: 'Aditya Dube'; ipoverib@ietf.org
>Subject: RE: [ipoverib] moving forward with the open Issues - end node
>configuration
>
>
>
>I agree that some method should be specified in the future
>However it is not so trivial to do it right now
>a. There are issues between IBTA & IETF on things like service records etc'
>b. You could have few VLAN's on the same IB subnet, how would you
>distinguish between them, etc' (maybe a service ID range)
>
>I would tend to go for an RFC with a static configuration, with known
>defaults like:
>The default P_Key value, 1500 MTU
>Work on a follow on proposal either in IBTA or IETF to address the
>automation
>
>You probably wonder why 1500, some of the reasons originally for
>determining
>that IPoIB supporting devices will have minimum MTU of 2K was to maintain
>similarity with Ethernet
>Since in IB you have either 1K or 2K, 1500 falls in the 2K domain, so the
>device must support 2K, not to say the MTU should by 2K
>
>Why 1500 again, lets say you have an Ethernet to IB bridge/router it will
>have to fragment/reassemble the packets
>This will create a lot of overhead on both the hosts and the routers that
>will significantly affect performance
>
>(it might be 1400 & something)
>
>Yaron
>
>-----Original Message-----
>From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
>Aditya Dube
>Sent: Friday, January 25, 2002 9:17 PM
>To: ipoverib@ietf.org
>Subject: RE: [ipoverib] moving forward with the open Issues - end node
>configuration
>
>
>
>The current IPOverIB spec requires that the P_Key and MTU be configured
>on each node.
>
>The IPOverIB spec should provide a default method for an end-node to
>come online with no IPoIB-specific configuration.
>
>During initialization, a node could query an IPOverIB specific
>ServiceRecord (Table 164, pg 685). This query will return a P_Key and
>ServiceSpecificFlags. The ServiceSpecificFlags can specify whether 2048
>or 4096 is the link MTU.
>
>The ServiceRecord's ServiceID should have a default value specific to
>IPOverIB. This number could be 0xFFFF:4000 in accordance with vol3
>section on "ServiceID Namespace" (Table 128, pg 215).
>
>The ServiceRecord needs to be created just as the all-node or broadcast
>MCGroup is created, i.e. by the first node on the network.
>
>If the Network Admin wishes to put the end-node on a different P_Key,
>then the Admin may override the values returned by the ServiceRecord.
>That is outside the scope of my proposal.
>
>
>Aditya Dube
>
>
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>https://www1.ietf.org/mailman/listinfo/ipoverib
>
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>https://www1.ietf.org/mailman/listinfo/ipoverib
>
>
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>https://www1.ietf.org/mailman/listinfo/ipoverib





_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Sat Jan 26 17:46:37 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA11564
	for <ipoverib-archive@odin.ietf.org>; Sat, 26 Jan 2002 17:46:37 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id RAA08688
	for ipoverib-archive@odin.ietf.org; Sat, 26 Jan 2002 17:46:40 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA08635;
	Sat, 26 Jan 2002 17:42:57 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA08606
	for <ipoverib@optimus.ietf.org>; Sat, 26 Jan 2002 17:42:56 -0500 (EST)
Received: from patan.sun.com (patan.Sun.COM [192.18.98.43])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA11524
	for <ipoverib@ietf.org>; Sat, 26 Jan 2002 17:42:49 -0500 (EST)
Received: from sunmail1.Sun.COM ([129.145.1.2])
	by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id PAA26368;
	Sat, 26 Jan 2002 15:42:52 -0700 (MST)
Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.84.45])
	by sunmail1.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1-Sun.COM.mod.2) with ESMTP id OAA02343;
	Sat, 26 Jan 2002 14:44:16 -0800 (PST)
Received: from sweethome (dsl-194-138.Eng.Sun.COM [129.146.194.138])
	by jurassic.eng.sun.com (8.12.2+Sun/8.12.2) with SMTP id g0QMgm3g283480;
	Sat, 26 Jan 2002 14:42:51 -0800 (PST)
Message-Id: <200201262242.g0QMgm3g283480@jurassic.eng.sun.com>
Date: Sat, 26 Jan 2002 14:43:18 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Subject: RE: [ipoverib] moving forward with the open Issues - end node configuration
To: adube@windows.microsoft.com, ipoverib@ietf.org
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: oVmbQFCX+i53HkDddYgGzw==
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.2 SunOS 5.8 sun4u sparc 
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

There are two levels of configuration tasks needed for IPoIB. One at the
link level the other at IP level. The link and multicast draft addresses
the configuration steps required to set up IPoIB "links". The equivalent
in Ethenet network is setting up VLANs. In Ethernet, configuring VLANs
to run IP is much simpler because there is no MTU issue, no special
steps required to enable broadcast/multicast... In a way you can use
the link and multicast draft to apply it to other, non-IP networks
too. It contains all the generic steps for setting up a link for
a broadcast/multicast capable network.

You should consider P_Keys of a CA port as defining "virtual" network
interfaces for the local node. E.g. if an HCA port has been assigned
ten P_keys, when starting up the network subsystem, the OS must lookup
from SA for the special all-node MC group with MGIDs defined in the
draft. If only three are found, then only three "virtual" network
interfaces will be enabled to run IP.

Once the link layer is fully set up, the configuration of IP subnets
on top of these IPoIB links are really no different from other IPoverFoo
(e.g. Ethernet). It is my hope that all the nice work IETF has done in
the area of autoconfiguration/zeroconf can be applied to IPoIB with
no (or minimal) changes.

Do you still see a need for special ServiceRecords, given all the info
readily available from MCGroupRecords?

Jerry 

>
>Quick comment,
>
>I thought Vol 3 was never an approved IBTA specification and was dropped...
>
>I definately agree with you that a zero-config type environment would be
>nice for IPoIB traffic, however there might be some other alternatives.
>
>I can see DHCP extensions
>I can see Service Records (can the IETF define these Mike Krause ?) like you
>suggested
>I can see manual configuration - technically not zero config at this point
>I can see defaults in the protocol - Though SHOULD use 2K MTU and a P_Key of
>mikimitdigoat or I'll even let you use a ascii representation of your
>initials if you are vain <G> (Just don't tell anyone)
>
>Do you think you could write a proposal describing how any/all of these
>methods that you think will work and any others you might come across...
>
>Thank you
>Bill
>
>-----Original Message-----
>From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
>Of Aditya Dube
>Sent: Friday, January 25, 2002 11:17 AM
>To: ipoverib@ietf.org
>Subject: RE: [ipoverib] moving forward with the open Issues - end node
>configuration
>
>
>
>
>The current IPOverIB spec requires that the P_Key and MTU be configured
>on each node.
>
>The IPOverIB spec should provide a default method for an end-node to
>come online with no IPoIB-specific configuration.
>
>During initialization, a node could query an IPOverIB specific
>ServiceRecord (Table 164, pg 685). This query will return a P_Key and
>ServiceSpecificFlags. The ServiceSpecificFlags can specify whether 2048
>or 4096 is the link MTU.
>
>The ServiceRecord's ServiceID should have a default value specific to
>IPOverIB. This number could be 0xFFFF:4000 in accordance with vol3
>section on "ServiceID Namespace" (Table 128, pg 215).
>
>The ServiceRecord needs to be created just as the all-node or broadcast
>MCGroup is created, i.e. by the first node on the network.
>
>If the Network Admin wishes to put the end-node on a different P_Key,
>then the Admin may override the values returned by the ServiceRecord.
>That is outside the scope of my proposal.
>
>
>Aditya Dube
>
>
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>https://www1.ietf.org/mailman/listinfo/ipoverib
>
>
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Sat Jan 26 18:28:59 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA11857
	for <ipoverib-archive@odin.ietf.org>; Sat, 26 Jan 2002 18:28:54 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id SAA09665
	for ipoverib-archive@odin.ietf.org; Sat, 26 Jan 2002 18:28:58 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA09595;
	Sat, 26 Jan 2002 18:24:58 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id SAA09564
	for <ipoverib@optimus.ietf.org>; Sat, 26 Jan 2002 18:24:56 -0500 (EST)
Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA11820
	for <ipoverib@ietf.org>; Sat, 26 Jan 2002 18:24:51 -0500 (EST)
Received: from sunmail1.Sun.COM ([129.145.1.2])
	by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id PAA28297;
	Sat, 26 Jan 2002 15:24:55 -0800 (PST)
Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.82.166])
	by sunmail1.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1-Sun.COM.mod.2) with ESMTP id PAA09038;
	Sat, 26 Jan 2002 15:26:19 -0800 (PST)
Received: from sweethome (dsl-194-138.Eng.Sun.COM [129.146.194.138])
	by jurassic.eng.sun.com (8.12.2+Sun/8.12.2) with SMTP id g0QNOo3g285308;
	Sat, 26 Jan 2002 15:24:54 -0800 (PST)
Message-Id: <200201262324.g0QNOo3g285308@jurassic.eng.sun.com>
Date: Sat, 26 Jan 2002 15:25:20 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Subject: RE: [ipoverib] moving forward with the open Issues - end node configuration
To: ipoverib@ietf.org, yaronh@voltaire.com
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: Qfw5PNb/xejkyP3oXE1+sQ==
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.2 SunOS 5.8 sun4u sparc 
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Yaron,

>I agree that some method should be specified in the future
>However it is not so trivial to do it right now
>a. There are issues between IBTA & IETF on things like service records etc'
>b. You could have few VLAN's on the same IB subnet, how would you
>distinguish between them, etc' (maybe a service ID range)

For each P_Key in a port's P_Key table, simply do a lookup to SA for
an IPoIB MGID defined in the link and multicast draft. If the lookup
succeeds, the IB partition defined by the P_Key will be used to host
IPoIB, otherwise it won't. Isn't that pretty simple?

>
>I would tend to go for an RFC with a static configuration, with known
>defaults like:
>The default P_Key value, 1500 MTU
>Work on a follow on proposal either in IBTA or IETF to address the
>automation

Static configuration is a thing of the past. Perhaps the motivation here
is to bypass SM/SA again (like the LID proposal)? I can sympathize with
the concern that SM/SA may be so complex that having a dependency on
it is a bit risky. But let's see how feasible a bypass is. You can
hardwire MTU, Q_Key, TClass... But you really need a broadcast mechanism
(simulating broadcast by unicast is real messy) so you've got to have a
MLID. Are you going to hardwire that too?

If the concern is about the complexity of SM/SA, why not just dummy up
a SM/SA implementation and hardwire all your favorite parameters there?
This way we don't need to muck with IPoIB standard.

>
>You probably wonder why 1500, some of the reasons originally for determining
>that IPoIB supporting devices will have minimum MTU of 2K was to maintain
>similarity with Ethernet
>Since in IB you have either 1K or 2K, 1500 falls in the 2K domain, so the
>device must support 2K, not to say the MTU should by 2K
>
>Why 1500 again, lets say you have an Ethernet to IB bridge/router it will
>have to fragment/reassemble the packets
>This will create a lot of overhead on both the hosts and the routers that
>will significantly affect performance

There are two very different cases that you mentioned here, bridge and
router. For the latter, no sweat, PMTU discovery takes care of it so
you don't need to muck with individual link MTU. (That's the whole
purpose of PMTU.) For bridges, no problem, just put your favorite
MTU value in the MCGroupRecord when you set up an IPoIB link to bridge
with an Ether segment.

Jerry

>
>(it might be 1400 & something)
>
>Yaron
>
>-----Original Message-----
>From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
>Aditya Dube
>Sent: Friday, January 25, 2002 9:17 PM
>To: ipoverib@ietf.org
>Subject: RE: [ipoverib] moving forward with the open Issues - end node
>configuration
>
>
>
>The current IPOverIB spec requires that the P_Key and MTU be configured
>on each node.
>
>The IPOverIB spec should provide a default method for an end-node to
>come online with no IPoIB-specific configuration.
>
>During initialization, a node could query an IPOverIB specific
>ServiceRecord (Table 164, pg 685). This query will return a P_Key and
>ServiceSpecificFlags. The ServiceSpecificFlags can specify whether 2048
>or 4096 is the link MTU.
>
>The ServiceRecord's ServiceID should have a default value specific to
>IPOverIB. This number could be 0xFFFF:4000 in accordance with vol3
>section on "ServiceID Namespace" (Table 128, pg 215).
>
>The ServiceRecord needs to be created just as the all-node or broadcast
>MCGroup is created, i.e. by the first node on the network.
>
>If the Network Admin wishes to put the end-node on a different P_Key,
>then the Admin may override the values returned by the ServiceRecord.
>That is outside the scope of my proposal.
>
>
>Aditya Dube
>
>
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>https://www1.ietf.org/mailman/listinfo/ipoverib
>
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Sat Jan 26 22:23:11 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA14925
	for <ipoverib-archive@odin.ietf.org>; Sat, 26 Jan 2002 22:23:10 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id WAA16222
	for ipoverib-archive@odin.ietf.org; Sat, 26 Jan 2002 22:23:15 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA16116;
	Sat, 26 Jan 2002 22:19:29 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA16084
	for <ipoverib@optimus.ietf.org>; Sat, 26 Jan 2002 22:19:27 -0500 (EST)
Received: from host32.cedant.com (host32.cedant.com [209.239.32.20])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA14836
	for <ipoverib@ietf.org>; Sat, 26 Jan 2002 22:19:22 -0500 (EST)
Received: from sahara (15.sanjose-08-09rs16rt.ca.dial-access.att.net [12.81.3.15])
	by host32.cedant.com (8.10.2/8.10.2) with ESMTP id g0R3Ixe24717;
	Sat, 26 Jan 2002 22:19:01 -0500
Message-ID: <200201261934390895.4F8DF03C@opulentsystems.com>
In-Reply-To: <HHEDIJGOEFLPCPNILMPAMEICCFAA.bill@strahm.net>
References: <HHEDIJGOEFLPCPNILMPAMEICCFAA.bill@strahm.net>
X-Mailer: Calypso Version 3.30.00.00 (4)
Date: Sat, 26 Jan 2002 19:34:39 -0800
Reply-To: sganguly@opulentsystems.com
From: "Sukanta Ganguly" <sganguly@opulentsystems.com>
To: "Bill Strahm" <bill@strahm.net>, "Yaron Haviv" <yaronh@voltaire.com>,
        "'Aditya Dube'" <adube@windows.microsoft.com>, ipoverib@ietf.org
Subject: RE: [ipoverib] moving forward with the open Issues - end node
   configuration
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id WAA16085
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

Response is embedded between <B-SG> and <E-SG>. Hope my explanation and reasonings are made clear.

Thanks
SG

*********** REPLY SEPARATOR  ***********

On 1/26/2002 at 2:02 PM Bill Strahm wrote:

>I know all of this...
>
>No answer this.

<B-SG>
What do you mean by the above ???
<E-SG>

>
>In the case of TCP, PMTU will correctly determine the PATH MTU, not the
>local link MTU and you don't have frag/reassembly problems in the network

<B-SG>
We both agree on this along with the world. 
<E-SG>

>
>In the case of UDP, you will have to handle these anyway, because UDP
>packets are just as big as they are...
>

<B-SG>
OK.
<E-SG>

>Now tell me what you have bought me by reducing the local MTU below its
>maximum ?
>

<B-SG>
So instead of asking me the question of what I have bought you by allowing the MTU selection to comply with the PMTU like approach, I would rather think that you should have concluded that just like UDP dumps packets on to the IP layer, IPoIB will do the same, then there is a point I see. I do not agree with the reasoning, but I do acknowledge that there must be some reasons why such a conclusion was made and I guess, not only me but others too would be asking this question as to why such a conclusion.

Coming back to the issue in hand, are we concluding that we don't really care about the potential issues of frag/defrag that may happen at layer three and dump what ever size of the IB PDU, that suits our need due to our IB protocol constraints/design/choices etc?

We will definitely see fragramentation at layer three for PDU sizes over 1500 bytes.  
<E-SG>

>Bill
>
>-----Original Message-----
>From: Sukanta Ganguly [mailto:sganguly@opulentsystems.com]
>Sent: Saturday, January 26, 2002 10:33 AM
>To: Bill Strahm; Yaron Haviv; 'Aditya Dube'; ipoverib@ietf.org
>Subject: RE: [ipoverib] moving forward with the open Issues - end node
>configuration
>
>
>Bill,
>  By forcing a larger MTU value an overhead in injected in the path, where
>nodes not in the position to support larger MTU have to fragment/defrag the
>packets. This is processing overhead. From the position of sending the
>packet it may not seem like one. But as far as packet being delivered from
>one end to the other the overhead is injected. The Path MTU mechanism is to
>determine the lowest MTU for all the individual links and setting that to
>be
>your MTU. No packet frag/defrag will happen and hence not buffereing
>expected in the path for any intermediate nodes.
>
>SG
>
>*********** REPLY SEPARATOR  ***********
>
>On 1/25/2002 at 3:37 PM Bill Strahm wrote:
>
>>The fact that there are transports in the middle of the network that do
>>not
>>support larger MTUs is not a reason to lower your MTU value...
>>
>>TCP will find the correct Path MTU, and UDP doesn't care.  Therefor why
>>should I limit my MTU on the local subnet because someone MIGHT have an
>MTU
>>much lower.  I would rather state the largest local wire MTU (2K -
>>overhead)
>>and let the protocols lower it automatically...
>>
>>Bill
>>
>>-----Original Message-----
>>From: ipove  rib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
>>Of Yaron Haviv
>>Sent: Friday, January 25, 2002 3:00 PM
>>To: 'Aditya Dube'; ipoverib@ietf.org
>>Subject: RE: [ipoverib] moving forward with the open Issues - end node
>>configuration
>>
>>
>>
>>I agree that some method should be specified in the future
>>However it is not so trivial to do it right now
>>a. There are issues between IBTA & IETF on things like service records
>etc'
>>b. You could have few VLAN's on the same IB subnet, how would you
>>distinguish between them, etc' (maybe a service ID range)
>>
>>I would tend to go for an RFC with a static configuration, with known
>>defaults like:
>>The default P_Key value, 1500 MTU
>>Work on a follow on proposal either in IBTA or IETF to address the
>>automation
>>
>>You probably wonder why 1500, some of the reasons originally for
>>determining
>>that IPoIB supporting devices will have minimum MTU of 2K was to maintain
>>similarity with Ethernet
>>Since in IB you have either 1K or 2K, 1500 falls in the 2K domain, so the
>>device must support 2K, not to say the MTU should by 2K
>>
>>Why 1500 again, lets say you have an Ethernet to IB bridge/router it will
>>have to fragment/reassemble the packets
>>This will create a lot of overhead on both the hosts and the routers that
>>will significantly affect performance
>>
>>(it might be 1400 & something)
>>
>>Yaron
>>
>>-----Original Message-----
>>From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
>>Aditya Dube
>>Sent: Friday, January 25, 2002 9:17 PM
>>To: ipoverib@ietf.org
>>Subject: RE: [ipoverib] moving forward with the open Issues - end node
>>configuration
>>
>>
>>
>>The current IPOverIB spec requires that the P_Key and MTU be configured
>>on each node.
>>
>>The IPOverIB spec should provide a default method for an end-node to
>>come online with no IPoIB-specific configuration.
>>
>>During initialization, a node could query an IPOverIB specific
>>ServiceRecord (Table 164, pg 685). This query will return a P_Key and
>>ServiceSpecificFlags. The ServiceSpecificFlags can specify whether 2048
>>or 4096 is the link MTU.
>>
>>The ServiceRecord's ServiceID should have a default value specific to
>>IPOverIB. This number could be 0xFFFF:4000 in accordance with vol3
>>section on "ServiceID Namespace" (Table 128, pg 215).
>>
>>The ServiceRecord needs to be created just as the all-node or broadcast
>>MCGroup is created, i.e. by the first node on the network.
>>
>>If the Network Admin wishes to put the end-node on a different P_Key,
>>then the Admin may override the values returned by the ServiceRecord.
>>That is outside the scope of my proposal.
>>
>>
>>Aditya Dube
>>
>>
>>
>>_______________________________________________
>>IPoverIB mailing list
>>IPoverIB@ietf.org
>>https://www1.ietf.org/mailman/listinfo/ipoverib
>>
>>
>>_______________________________________________
>>IPoverIB mailing list
>>IPoverIB@ietf.org
>>https://www1.ietf.org/mailman/listinfo/ipoverib
>>
>>
>>
>>_______________________________________________
>>IPoverIB mailing list
>>IPoverIB@ietf.org
>>https://www1.ietf.org/mailman/listinfo/ipoverib




_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Sat Jan 26 23:03:04 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA16190
	for <ipoverib-archive@odin.ietf.org>; Sat, 26 Jan 2002 23:03:04 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id XAA17554
	for ipoverib-archive@odin.ietf.org; Sat, 26 Jan 2002 23:03:08 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id XAA17524;
	Sat, 26 Jan 2002 23:02:35 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id XAA17467
	for <ipoverib@optimus.ietf.org>; Sat, 26 Jan 2002 23:02:33 -0500 (EST)
Received: from pdxpo.dsl-only.net (pdxpo.dsl-only.net [63.105.16.3])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA16181
	for <ipoverib@ietf.org>; Sat, 26 Jan 2002 23:02:27 -0500 (EST)
Received: from bonfire (unverified [63.105.18.201]) by pdxpo.dsl-only.net
 (Rockliffe SMTPRA 4.5.6) with SMTP id <B0009411196@pdxpo.dsl-only.net>;
 Sat, 26 Jan 2002 20:01:14 -0800
From: "Bill Strahm" <bill@strahm.net>
To: <sganguly@opulentsystems.com>, <ipoverib@ietf.org>
Subject: RE: [ipoverib] moving forward with the open Issues - end node   configuration
Date: Sat, 26 Jan 2002 20:10:08 -0800
Message-ID: <HHEDIJGOEFLPCPNILMPAGEIFCFAA.bill@strahm.net>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0)
Importance: Normal
In-Reply-To: <200201261934390895.4F8DF03C@opulentsystems.com>
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit


Ok... Some comments inline
-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Sukanta Ganguly
Sent: Saturday, January 26, 2002 7:35 PM
To: Bill Strahm; Yaron Haviv; 'Aditya Dube'; ipoverib@ietf.org
Subject: RE: [ipoverib] moving forward with the open Issues - end node
configuration


Response is embedded between <B-SG> and <E-SG>. Hope my explanation and
reasonings are made clear.

Thanks
SG

*********** REPLY SEPARATOR  ***********

On 1/26/2002 at 2:02 PM Bill Strahm wrote:

>I know all of this...
>
>No answer this.

<B-SG>
What do you mean by the above ???
<E-SG>
<Bill>
Missing w, should be Now answer this
</Bill>

>
>In the case of TCP, PMTU will correctly determine the PATH MTU, not the
>local link MTU and you don't have frag/reassembly problems in the network

<B-SG>
We both agree on this along with the world.
<E-SG>

>
>In the case of UDP, you will have to handle these anyway, because UDP
>packets are just as big as they are...
>

<B-SG>
OK.
<E-SG>

>Now tell me what you have bought me by reducing the local MTU below its
>maximum ?
>

<B-SG>
So instead of asking me the question of what I have bought you by allowing
the MTU selection to comply with the PMTU like approach, I would rather
think that you should have concluded that just like UDP dumps packets on to
the IP layer, IPoIB will do the same, then there is a
<Bill>
IPoIB DOES NOT Dump packets onto IP, it dumps IP packets onto a layer 2.
For this reason, why should I limit the size of my layer 3 packet because
some other layer 2 happens to like a 1500 byte MTU.
</Bill>
point I see. I do not agree with the reasoning, but I do acknowledge that
there must be some reasons why such a conclusion was made and I guess, not
only me but others too would be asking this question as to why such a
conclusion.

Coming back to the issue in hand, are we concluding that we don't really
care about the potential issues of frag/defrag that may happen at layer
three and dump what ever size of the IB PDU, that suits our need due to our
IB protocol constraints/design/choices etc?

We will definitely see fragramentation at layer three for PDU sizes over
1500 bytes.
<E-SG>
<Bill>
I disagree with this statement.  There are MANY other protocols that use an
MTU larger than 1540 bytes.  If we route over one of those networks we will
have fragmentation, TCPs PMTU will handle this for us, if we do not (we go
from one IB subnet, over a WAN protocol that has a large MTU onto another IB
subnet that has a 2K MTU) we will not have any additional fragmentation.
Why should we limit IB as a layer 2 because Ethernet has a 1500 byte MTU
limitation ?

Also note that in IPv6 the network is NOT allowed to fragment packets.
</Bill>
>Bill
>
>-----Original Message-----
>From: Sukanta Ganguly [mailto:sganguly@opulentsystems.com]
>Sent: Saturday, January 26, 2002 10:33 AM
>To: Bill Strahm; Yaron Haviv; 'Aditya Dube'; ipoverib@ietf.org
>Subject: RE: [ipoverib] moving forward with the open Issues - end node
>configuration
>
>
>Bill,
>  By forcing a larger MTU value an overhead in injected in the path, where
>nodes not in the position to support larger MTU have to fragment/defrag the
>packets. This is processing overhead. From the position of sending the
>packet it may not seem like one. But as far as packet being delivered from
>one end to the other the overhead is injected. The Path MTU mechanism is to
>determine the lowest MTU for all the individual links and setting that to
>be
>your MTU. No packet frag/defrag will happen and hence not buffereing
>expected in the path for any intermediate nodes.
>
>SG
>
>*********** REPLY SEPARATOR  ***********
>
>On 1/25/2002 at 3:37 PM Bill Strahm wrote:
>
>>The fact that there are transports in the middle of the network that do
>>not
>>support larger MTUs is not a reason to lower your MTU value...
>>
>>TCP will find the correct Path MTU, and UDP doesn't care.  Therefor why
>>should I limit my MTU on the local subnet because someone MIGHT have an
>MTU
>>much lower.  I would rather state the largest local wire MTU (2K -
>>overhead)
>>and let the protocols lower it automatically...
>>
>>Bill
>>
>>-----Original Message-----
>>From: ipove  rib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
>>Of Yaron Haviv
>>Sent: Friday, January 25, 2002 3:00 PM
>>To: 'Aditya Dube'; ipoverib@ietf.org
>>Subject: RE: [ipoverib] moving forward with the open Issues - end node
>>configuration
>>
>>
>>
>>I agree that some method should be specified in the future
>>However it is not so trivial to do it right now
>>a. There are issues between IBTA & IETF on things like service records
>etc'
>>b. You could have few VLAN's on the same IB subnet, how would you
>>distinguish between them, etc' (maybe a service ID range)
>>
>>I would tend to go for an RFC with a static configuration, with known
>>defaults like:
>>The default P_Key value, 1500 MTU
>>Work on a follow on proposal either in IBTA or IETF to address the
>>automation
>>
>>You probably wonder why 1500, some of the reasons originally for
>>determining
>>that IPoIB supporting devices will have minimum MTU of 2K was to maintain
>>similarity with Ethernet
>>Since in IB you have either 1K or 2K, 1500 falls in the 2K domain, so the
>>device must support 2K, not to say the MTU should by 2K
>>
>>Why 1500 again, lets say you have an Ethernet to IB bridge/router it will
>>have to fragment/reassemble the packets
>>This will create a lot of overhead on both the hosts and the routers that
>>will significantly affect performance
>>
>>(it might be 1400 & something)
>>
>>Yaron
>>
>>-----Original Message-----
>>From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
>>Aditya Dube
>>Sent: Friday, January 25, 2002 9:17 PM
>>To: ipoverib@ietf.org
>>Subject: RE: [ipoverib] moving forward with the open Issues - end node
>>configuration
>>
>>
>>
>>The current IPOverIB spec requires that the P_Key and MTU be configured
>>on each node.
>>
>>The IPOverIB spec should provide a default method for an end-node to
>>come online with no IPoIB-specific configuration.
>>
>>During initialization, a node could query an IPOverIB specific
>>ServiceRecord (Table 164, pg 685). This query will return a P_Key and
>>ServiceSpecificFlags. The ServiceSpecificFlags can specify whether 2048
>>or 4096 is the link MTU.
>>
>>The ServiceRecord's ServiceID should have a default value specific to
>>IPOverIB. This number could be 0xFFFF:4000 in accordance with vol3
>>section on "ServiceID Namespace" (Table 128, pg 215).
>>
>>The ServiceRecord needs to be created just as the all-node or broadcast
>>MCGroup is created, i.e. by the first node on the network.
>>
>>If the Network Admin wishes to put the end-node on a different P_Key,
>>then the Admin may override the values returned by the ServiceRecord.
>>That is outside the scope of my proposal.
>>
>>
>>Aditya Dube
>>
>>
>>
>>_______________________________________________
>>IPoverIB mailing list
>>IPoverIB@ietf.org
>>https://www1.ietf.org/mailman/listinfo/ipoverib
>>
>>
>>_______________________________________________
>>IPoverIB mailing list
>>IPoverIB@ietf.org
>>https://www1.ietf.org/mailman/listinfo/ipoverib
>>
>>
>>
>>_______________________________________________
>>IPoverIB mailing list
>>IPoverIB@ietf.org
>>https://www1.ietf.org/mailman/listinfo/ipoverib




_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Sun Jan 27 15:09:23 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA02902
	for <ipoverib-archive@odin.ietf.org>; Sun, 27 Jan 2002 15:09:23 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id PAA23879
	for ipoverib-archive@odin.ietf.org; Sun, 27 Jan 2002 15:09:25 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA23849;
	Sun, 27 Jan 2002 15:08:10 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA23817
	for <ipoverib@optimus.ietf.org>; Sun, 27 Jan 2002 15:08:07 -0500 (EST)
Received: from mxout1.netvision.net.il (mxout1.netvision.net.il [194.90.9.20])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA02882
	for <ipoverib@ietf.org>; Sun, 27 Jan 2002 15:08:02 -0500 (EST)
Received: from vaio505he ([139.92.208.141]) by mxout1.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GQM0000C58IOA@mxout1.netvision.net.il> for ipoverib@ietf.org;
 Sun, 27 Jan 2002 22:07:33 +0200 (IST)
Date: Sun, 27 Jan 2002 22:08:11 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: RE: [ipoverib] moving forward with the open Issues - end node
 configuration
In-reply-to: <20083A3BAEF9D211BDB600805F8B14F301F24078@NTSERVER>
To: "'H.K. Jerry Chu'" <Jerry.Chu@eng.sun.com>, ipoverib@ietf.org
Reply-to: yaronh@voltaire.com
Message-id: <000001c1a76e$5a05df60$8dd05c8b@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7BIT

Jerry,

I guess you just missed my point; it was not to say that manual
configuration is good, and we need to avoid the SM
I just said that the process of agreeing on an automatic solution is not
simple, and I would like to get an RFC first, than quarrel over this
mechanism
Not to mention that it might involve additional standardization from IBTA
(e.g. determining ServiceID's, etc') which wont make it faster
This is the same reasoning like abandoning the capability bits issue today

As for your proposal I'm not so sure that it is so efficient going over all
the P_Key's defined, in some cases there could be more than a few
I don't have strong feelings against it, just rather see something more
efficient, if possible
On the same query to the SM we might also get more info on that IP subnet
that will ease the configuration, and shorten the booting process

Regarding the 1500 MTU, I know all about PMTU
The 2K adds some load on the network, which also depends on the topology and
traffic characteristics
It could be acceptable by many people
I would just want to enable setting the values to a number lower than 2K, if
the admin requires so
I'm not against setting 2K values for anyone that wants it, or even as the
default magic number for IPoIB as long as smaller numbers are also valid
And from your answer it looks like you agree to that too

Yaron

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
H.K. Jerry Chu
Sent: Sunday, January 27, 2002 1:25 AM
To: ipoverib@ietf.org; Yaron Haviv
Subject: RE: [ipoverib] moving forward with the open Issues - end node
configuration

Yaron,

>I agree that some method should be specified in the future
>However it is not so trivial to do it right now
>a. There are issues between IBTA & IETF on things like service records etc'
>b. You could have few VLAN's on the same IB subnet, how would you
>distinguish between them, etc' (maybe a service ID range)

For each P_Key in a port's P_Key table, simply do a lookup to SA for
an IPoIB MGID defined in the link and multicast draft. If the lookup
succeeds, the IB partition defined by the P_Key will be used to host
IPoIB, otherwise it won't. Isn't that pretty simple?

>
>I would tend to go for an RFC with a static configuration, with known
>defaults like:
>The default P_Key value, 1500 MTU
>Work on a follow on proposal either in IBTA or IETF to address the
>automation

Static configuration is a thing of the past. Perhaps the motivation here
is to bypass SM/SA again (like the LID proposal)? I can sympathize with
the concern that SM/SA may be so complex that having a dependency on
it is a bit risky. But let's see how feasible a bypass is. You can
hardwire MTU, Q_Key, TClass... But you really need a broadcast mechanism
(simulating broadcast by unicast is real messy) so you've got to have a
MLID. Are you going to hardwire that too?

If the concern is about the complexity of SM/SA, why not just dummy up
a SM/SA implementation and hardwire all your favorite parameters there?
This way we don't need to muck with IPoIB standard.

>
>You probably wonder why 1500, some of the reasons originally for
determining
>that IPoIB supporting devices will have minimum MTU of 2K was to maintain
>similarity with Ethernet
>Since in IB you have either 1K or 2K, 1500 falls in the 2K domain, so the
>device must support 2K, not to say the MTU should by 2K
>
>Why 1500 again, lets say you have an Ethernet to IB bridge/router it will
>have to fragment/reassemble the packets
>This will create a lot of overhead on both the hosts and the routers that
>will significantly affect performance

There are two very different cases that you mentioned here, bridge and
router. For the latter, no sweat, PMTU discovery takes care of it so
you don't need to muck with individual link MTU. (That's the whole
purpose of PMTU.) For bridges, no problem, just put your favorite
MTU value in the MCGroupRecord when you set up an IPoIB link to bridge
with an Ether segment.

Jerry

>
>(it might be 1400 & something)
>
>Yaron
>
>-----Original Message-----
>From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
>Aditya Dube
>Sent: Friday, January 25, 2002 9:17 PM
>To: ipoverib@ietf.org
>Subject: RE: [ipoverib] moving forward with the open Issues - end node
>configuration
>
>
>
>The current IPOverIB spec requires that the P_Key and MTU be configured
>on each node.
>
>The IPOverIB spec should provide a default method for an end-node to
>come online with no IPoIB-specific configuration.
>
>During initialization, a node could query an IPOverIB specific
>ServiceRecord (Table 164, pg 685). This query will return a P_Key and
>ServiceSpecificFlags. The ServiceSpecificFlags can specify whether 2048
>or 4096 is the link MTU.
>
>The ServiceRecord's ServiceID should have a default value specific to
>IPOverIB. This number could be 0xFFFF:4000 in accordance with vol3
>section on "ServiceID Namespace" (Table 128, pg 215).
>
>The ServiceRecord needs to be created just as the all-node or broadcast
>MCGroup is created, i.e. by the first node on the network.
>
>If the Network Admin wishes to put the end-node on a different P_Key,
>then the Admin may override the values returned by the ServiceRecord.
>That is outside the scope of my proposal.
>
>
>Aditya Dube
>
>
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>https://www1.ietf.org/mailman/listinfo/ipoverib
>
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Sun Jan 27 15:57:00 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA03283
	for <ipoverib-archive@odin.ietf.org>; Sun, 27 Jan 2002 15:57:00 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id PAA25170
	for ipoverib-archive@odin.ietf.org; Sun, 27 Jan 2002 15:57:03 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA25155;
	Sun, 27 Jan 2002 15:56:41 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id PAA25123
	for <ipoverib@optimus.ietf.org>; Sun, 27 Jan 2002 15:56:39 -0500 (EST)
Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA03277
	for <ipoverib@ietf.org>; Sun, 27 Jan 2002 15:56:35 -0500 (EST)
Received: from sunmail1.Sun.COM ([129.145.1.2])
	by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id MAA04261;
	Sun, 27 Jan 2002 12:56:37 -0800 (PST)
Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.88.130])
	by sunmail1.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1-Sun.COM.mod.2) with ESMTP id MAA02611;
	Sun, 27 Jan 2002 12:58:03 -0800 (PST)
Received: from sweethome (dsl-194-138.Eng.Sun.COM [129.146.194.138])
	by jurassic.eng.sun.com (8.12.2+Sun/8.12.2) with SMTP id g0RKuW3g338956;
	Sun, 27 Jan 2002 12:56:36 -0800 (PST)
Message-Id: <200201272056.g0RKuW3g338956@jurassic.eng.sun.com>
Date: Sun, 27 Jan 2002 12:57:02 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Subject: RE: [ipoverib] moving forward with the open Issues - end node configuration
To: ipoverib@ietf.org, yaronh@voltaire.com
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: aBgOu/GlC+YReUw76BU9sg==
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.2 SunOS 5.8 sun4u sparc 
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Yaron,

>I guess you just missed my point; it was not to say that manual
>configuration is good, and we need to avoid the SM
>I just said that the process of agreeing on an automatic solution is not
>simple, and I would like to get an RFC first, than quarrel over this
>mechanism

Disagree. See my other mail. IETF already provides a number of mechanisms
for autoconfiguration that apply to IPoIB as well as any IPoverFoo. All
we need to solve here is to autoconfigure IPoIB links.

>Not to mention that it might involve additional standardization from IBTA
>(e.g. determining ServiceID's, etc') which wont make it faster
>This is the same reasoning like abandoning the capability bits issue today
>
>As for your proposal I'm not so sure that it is so efficient going over all
>the P_Key's defined, in some cases there could be more than a few
>I don't have strong feelings against it, just rather see something more
>efficient, if possible

Note first that this link configuration process only occur once per node
reboot. The efficiency concern IMHO is even less convincing here than the
one that argues for inlcuding LID in ARP. Second, there is nothing that
prevents implementations from supplementing additional configuration
files on an endnode that cache/specify IPoIB P_Keys, for example. These
local configuration practices don't present any interoperability problem,
therefore does not need to be standardized.

>On the same query to the SM we might also get more info on that IP subnet
>that will ease the configuration, and shorten the booting process
>
>Regarding the 1500 MTU, I know all about PMTU
>The 2K adds some load on the network, which also depends on the topology and
>traffic characteristics
>It could be acceptable by many people
>I would just want to enable setting the values to a number lower than 2K, if
>the admin requires so
>I'm not against setting 2K values for anyone that wants it, or even as the
>default magic number for IPoIB as long as smaller numbers are also valid
>And from your answer it looks like you agree to that too

Yup. I've been against some earlier proposals to designate a fixed MTU size
(e.g. 2K) for IPoIB because I don't think that is necessary. This is covered
in the link and multicast draft by using the MTU field in the MCGroupRecord.
But an issue just came up to me if one wants to support 1500 MTU. Contrary
to the "IP MTU" that allows one to specify an arbitrary size as long as
it's not greater than the link MTU, the "IB MTU" in the MCGroupRecord
field only allows fives choices from 256 to 4096.

We can either
1. punt it, since other IPoverFoo doesn't support the use of arbitrarily
configured MTU size either. (How does bridge vendors solve this problem when
bridging FDDI with Ethernet?)

or
2. find some place to store the IP MTU (ServiceRecord? I prefer not.)

Any suggestion?

Jerry

>
>Yaron
>
>-----Original Message-----
>From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
>H.K. Jerry Chu
>Sent: Sunday, January 27, 2002 1:25 AM
>To: ipoverib@ietf.org; Yaron Haviv
>Subject: RE: [ipoverib] moving forward with the open Issues - end node
>configuration
>
>Yaron,
>
>>I agree that some method should be specified in the future
>>However it is not so trivial to do it right now
>>a. There are issues between IBTA & IETF on things like service records etc'
>>b. You could have few VLAN's on the same IB subnet, how would you
>>distinguish between them, etc' (maybe a service ID range)
>
>For each P_Key in a port's P_Key table, simply do a lookup to SA for
>an IPoIB MGID defined in the link and multicast draft. If the lookup
>succeeds, the IB partition defined by the P_Key will be used to host
>IPoIB, otherwise it won't. Isn't that pretty simple?
>
>>
>>I would tend to go for an RFC with a static configuration, with known
>>defaults like:
>>The default P_Key value, 1500 MTU
>>Work on a follow on proposal either in IBTA or IETF to address the
>>automation
>
>Static configuration is a thing of the past. Perhaps the motivation here
>is to bypass SM/SA again (like the LID proposal)? I can sympathize with
>the concern that SM/SA may be so complex that having a dependency on
>it is a bit risky. But let's see how feasible a bypass is. You can
>hardwire MTU, Q_Key, TClass... But you really need a broadcast mechanism
>(simulating broadcast by unicast is real messy) so you've got to have a
>MLID. Are you going to hardwire that too?
>
>If the concern is about the complexity of SM/SA, why not just dummy up
>a SM/SA implementation and hardwire all your favorite parameters there?
>This way we don't need to muck with IPoIB standard.
>
>>
>>You probably wonder why 1500, some of the reasons originally for
>determining
>>that IPoIB supporting devices will have minimum MTU of 2K was to maintain
>>similarity with Ethernet
>>Since in IB you have either 1K or 2K, 1500 falls in the 2K domain, so the
>>device must support 2K, not to say the MTU should by 2K
>>
>>Why 1500 again, lets say you have an Ethernet to IB bridge/router it will
>>have to fragment/reassemble the packets
>>This will create a lot of overhead on both the hosts and the routers that
>>will significantly affect performance
>
>There are two very different cases that you mentioned here, bridge and
>router. For the latter, no sweat, PMTU discovery takes care of it so
>you don't need to muck with individual link MTU. (That's the whole
>purpose of PMTU.) For bridges, no problem, just put your favorite
>MTU value in the MCGroupRecord when you set up an IPoIB link to bridge
>with an Ether segment.
>
>Jerry
>
>>
>>(it might be 1400 & something)
>>
>>Yaron
>>
>>-----Original Message-----
>>From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
>>Aditya Dube
>>Sent: Friday, January 25, 2002 9:17 PM
>>To: ipoverib@ietf.org
>>Subject: RE: [ipoverib] moving forward with the open Issues - end node
>>configuration
>>
>>
>>
>>The current IPOverIB spec requires that the P_Key and MTU be configured
>>on each node.
>>
>>The IPOverIB spec should provide a default method for an end-node to
>>come online with no IPoIB-specific configuration.
>>
>>During initialization, a node could query an IPOverIB specific
>>ServiceRecord (Table 164, pg 685). This query will return a P_Key and
>>ServiceSpecificFlags. The ServiceSpecificFlags can specify whether 2048
>>or 4096 is the link MTU.
>>
>>The ServiceRecord's ServiceID should have a default value specific to
>>IPOverIB. This number could be 0xFFFF:4000 in accordance with vol3
>>section on "ServiceID Namespace" (Table 128, pg 215).
>>
>>The ServiceRecord needs to be created just as the all-node or broadcast
>>MCGroup is created, i.e. by the first node on the network.
>>
>>If the Network Admin wishes to put the end-node on a different P_Key,
>>then the Admin may override the values returned by the ServiceRecord.
>>That is outside the scope of my proposal.
>>
>>
>>Aditya Dube
>>
>>
>>
>>_______________________________________________
>>IPoverIB mailing list
>>IPoverIB@ietf.org
>>https://www1.ietf.org/mailman/listinfo/ipoverib
>>
>>
>>_______________________________________________
>>IPoverIB mailing list
>>IPoverIB@ietf.org
>>https://www1.ietf.org/mailman/listinfo/ipoverib
>
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>https://www1.ietf.org/mailman/listinfo/ipoverib
>


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 28 03:05:10 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA20088
	for <ipoverib-archive@odin.ietf.org>; Mon, 28 Jan 2002 03:05:10 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id DAA26392
	for ipoverib-archive@odin.ietf.org; Mon, 28 Jan 2002 03:05:11 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id DAA26257;
	Mon, 28 Jan 2002 03:03:06 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id AAA10864
	for <ipoverib@optimus.ietf.org>; Mon, 28 Jan 2002 00:07:00 -0500 (EST)
Received: from fgwmail7.fujitsu.co.jp (fgwmail7.fujitsu.co.jp [192.51.44.37])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA10088
	for <ipoverib@ietf.org>; Mon, 28 Jan 2002 00:06:58 -0500 (EST)
Received: from m6.gw.fujitsu.co.jp by fgwmail7.fujitsu.co.jp (8.9.3/3.7W-MX0201-Fujitsu Gateway)
	id OAA27198 for <ipoverib@ietf.org>; Mon, 28 Jan 2002 14:06:25 +0900 (JST)
	(envelope-from koji@lp.nm.fujitsu.co.jp)
Received: from lpdm.lp.nm.fujitsu.co.jp by m6.gw.fujitsu.co.jp (8.9.3/3.7W-0112-Fujitsu Domain Master)
	id OAA31620 for <ipoverib@ietf.org>; Mon, 28 Jan 2002 14:06:24 +0900
	(envelope-from koji@lp.nm.fujitsu.co.jp)
Received: from ebara.lp.nm.fujitsu.co.jp (ebara [10.124.76.64])
	by lpdm.lp.nm.fujitsu.co.jp (8.9.3/3.6W) with SMTP id OAA15959
	for <ipoverib@ietf.org>; Mon, 28 Jan 2002 14:06:24 +0900 (JST)
Received: by ebara.lp.nm.fujitsu.co.jp (SMI-8.6/6.4J.6-lpslave03)
	id OAA18300; Mon, 28 Jan 2002 14:08:20 +0900
Date: Mon, 28 Jan 2002 14:08:20 +0900
From: Koji takahara <koji@lp.nm.fujitsu.co.jp>
Message-Id: <200201280508.OAA18300@ebara.lp.nm.fujitsu.co.jp>
To: ipoverib@ietf.org
Subject: RE: [ipoverib] resolving the Ethertype issue
In-Reply-To: <D9223EB959A5D511A98F00508B68C20C21A885@ORSMSX108>
References: <D9223EB959A5D511A98F00508B68C20C21A885@ORSMSX108>
Reply-To: koji@cs.fujitsu.co.jp
Mime-Version: 1.0 (generated by tm-edit 7.66)
Content-Type: text/plain; charset=US-ASCII
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

I strongly agree with Roy.

I think the best method to identify the destination protocol is
the following.
(1) to specify a Ethertype as the first field of the payload

If we do not use method (1), we have to use one of the following
methods.
(2) to use different QPNs for different Ethertypes, or
(3) to peek into an upper-level protocol frame in link layer

I also think both (2) and (3) are not clean.

And I think we should not assume Ethertypes are only ARP, IPv4,
IPv6.
That assumption restricts the expantion of ther upper-layer of
the ipoverib link layer driver.

_______________________________________________________________
Koji Takahara  Email: koji@cs.fujitsu.co.jp
Development Dept.5,  Infrastructure Systems division,
Computer Group, Fujitsu Limitted


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 28 03:17:13 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA20225
	for <ipoverib-archive@odin.ietf.org>; Mon, 28 Jan 2002 03:17:13 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id DAA26754
	for ipoverib-archive@odin.ietf.org; Mon, 28 Jan 2002 03:17:15 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id DAA26700;
	Mon, 28 Jan 2002 03:15:32 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id DAA26663
	for <ipoverib@optimus.ietf.org>; Mon, 28 Jan 2002 03:15:30 -0500 (EST)
Received: from pdxpo.dsl-only.net (pdxpo.dsl-only.net [63.105.16.3])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA20212
	for <ipoverib@ietf.org>; Mon, 28 Jan 2002 03:15:26 -0500 (EST)
Received: from bonfire (unverified [63.105.18.201]) by pdxpo.dsl-only.net
 (Rockliffe SMTPRA 4.5.6) with SMTP id <B0009427245@pdxpo.dsl-only.net>;
 Mon, 28 Jan 2002 00:14:09 -0800
From: "Bill Strahm" <bill@strahm.net>
To: <koji@cs.fujitsu.co.jp>, <ipoverib@ietf.org>
Subject: RE: [ipoverib] resolving the Ethertype issue
Date: Mon, 28 Jan 2002 00:24:20 -0800
Message-ID: <HHEDIJGOEFLPCPNILMPAIEIKCFAA.bill@strahm.net>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0)
In-Reply-To: <200201280508.OAA18300@ebara.lp.nm.fujitsu.co.jp>
Importance: Normal
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Comments inline...

I guess my big question is why are people afraid of allocating QPN for a
given use of Infiniband.  I thought that was the purpose of QPN.

Can someone tell me how they can redirect packets to the correct Layer 3
(with IB UD being a Layer 2 protocol for this discussion) using Ethertype
without incurring either an additional copy, or a page flip ?  The way I see
it is you will have to install a software "protocol demuxer" above the IB
layer to receive the IB frame, look at the first 4 bytes and then put it on
the appropriate protocols receive buffer.  I am most interested in listening
to peoples implementations that involve zero-copy going up the TCP stack.

Bill

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Koji takahara
Sent: Sunday, January 27, 2002 9:08 PM
To: ipoverib@ietf.org
Subject: RE: [ipoverib] resolving the Ethertype issue


I strongly agree with Roy.

I think the best method to identify the destination protocol is
the following.
(1) to specify a Ethertype as the first field of the payload
<Bill>
How do you propose handling the case where an Layer 3 (for IB that is)
protocol does not put a n-byte (I assume 4-byte) header in the message.  In
that case you CAN NOT determine if you are looking at a "EtherType" or the
first 4 bytes of a payload therefore...
</Bill>
If we do not use method (1), we have to use one of the following
methods.
(2) to use different QPNs for different Ethertypes, or
<Bill>
I claim you will have to do this anyway
</Bill>
(3) to peek into an upper-level protocol frame in link layer
<Bill>
You consider looking at the first 4 bytes of the ULP acceptable, yet here
you claim it is unclean... which is it ???
</Bill>
I also think both (2) and (3) are not clean.

And I think we should not assume Ethertypes are only ARP, IPv4,
IPv6.
That assumption restricts the expantion of ther upper-layer of
the ipoverib link layer driver.
<Bill>
Correct.  I claim option 2 is the way to go.  Then you don't have to worry
about forcing all other encapsulations to include a 4 byte header in their
framing, I do not see a way of enforcing this anyway..

What are peoples objections to using a small number of QPN to handle upper
layer protocols on top of Infiniband.  My assumptions are QPN are very
similar to Ethertypes except they can be dynamic...
</Bill>


Billhttps://www1.ietf.org/mailman/listinfo/ipoverib



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 28 04:15:42 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id EAA21103
	for <ipoverib-archive@odin.ietf.org>; Mon, 28 Jan 2002 04:15:42 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id EAA29170
	for ipoverib-archive@odin.ietf.org; Mon, 28 Jan 2002 04:15:44 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id EAA29139;
	Mon, 28 Jan 2002 04:13:29 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id EAA29111
	for <ipoverib@optimus.ietf.org>; Mon, 28 Jan 2002 04:13:27 -0500 (EST)
Received: from mxout1.netvision.net.il (mxout1.netvision.net.il [194.90.9.20])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id EAA21066
	for <ipoverib@ietf.org>; Mon, 28 Jan 2002 04:13:23 -0500 (EST)
Received: from vaio505he ([139.92.208.138]) by mxout1.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GQN0038H5LGFG@mxout1.netvision.net.il> for ipoverib@ietf.org;
 Mon, 28 Jan 2002 11:12:55 +0200 (IST)
Date: Mon, 28 Jan 2002 11:13:32 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: RE: [ipoverib] resolving the Ethertype issue
In-reply-to: <HHEDIJGOEFLPCPNILMPAIEIKCFAA.bill@strahm.net>
To: "'Bill Strahm'" <bill@strahm.net>, koji@cs.fujitsu.co.jp,
        ipoverib@ietf.org
Reply-to: yaronh@voltaire.com
Message-id: <001401c1a7dc$0f5be950$8dd05c8b@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7BIT

Bill,

No copy is/should be done between L2-L3, you just pass the buffer pointer
(sk_buff in linux, m_buff in BSD)
The mandatory copy is between the kernel and user space (TCP/IP private
buffers to Application receive buffers)
If an application can work with the packets when they are in the TCP/IP
private buffers chains (without doing a receive(buf) op for example) so it
could avoid the copy, usually it is not so simple, might work better for
kernel Apps
Another approach is to build TOE NIC's which place the data directly in the
application receive buffers (where the private buffers are on the NIC, no
magic)

The protocol demuxing relates to the execution process
The L2 look @ the protocol number, and calls the relevant protocol function
(or soft IRQ) based on the value, this could also be done from L3 to L4

By the way adding more QP's doesn't solve any problem, it just requires more
resources (I had a long email on that in the pass)

My take on the Ethertype issue is that who ever needs it when its not
provided could generate it for him self
Where who ever wants to ignore it when its available could also do that
This is why I don't think it is a critical decision if we do have it or we
don't, since the raw information is provided
It is up to the group to see if someone feels (and could also explain) that
the Ethertype provides him an Info he must have
And not just for future Ethernet over IB stuff since we deal with IP now

I personally could sleep well with the two decisions with/without, I just
want to resolve it quick

Yaron


-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Bill Strahm
Sent: Monday, January 28, 2002 10:24 AM
To: koji@cs.fujitsu.co.jp; ipoverib@ietf.org
Subject: RE: [ipoverib] resolving the Ethertype issue

Comments inline...

I guess my big question is why are people afraid of allocating QPN for a
given use of Infiniband.  I thought that was the purpose of QPN.

Can someone tell me how they can redirect packets to the correct Layer 3
(with IB UD being a Layer 2 protocol for this discussion) using Ethertype
without incurring either an additional copy, or a page flip ?  The way I see
it is you will have to install a software "protocol demuxer" above the IB
layer to receive the IB frame, look at the first 4 bytes and then put it on
the appropriate protocols receive buffer.  I am most interested in listening
to peoples implementations that involve zero-copy going up the TCP stack.

Bill

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Koji takahara
Sent: Sunday, January 27, 2002 9:08 PM
To: ipoverib@ietf.org
Subject: RE: [ipoverib] resolving the Ethertype issue


I strongly agree with Roy.

I think the best method to identify the destination protocol is
the following.
(1) to specify a Ethertype as the first field of the payload
<Bill>
How do you propose handling the case where an Layer 3 (for IB that is)
protocol does not put a n-byte (I assume 4-byte) header in the message.  In
that case you CAN NOT determine if you are looking at a "EtherType" or the
first 4 bytes of a payload therefore...
</Bill>
If we do not use method (1), we have to use one of the following
methods.
(2) to use different QPNs for different Ethertypes, or
<Bill>
I claim you will have to do this anyway
</Bill>
(3) to peek into an upper-level protocol frame in link layer
<Bill>
You consider looking at the first 4 bytes of the ULP acceptable, yet here
you claim it is unclean... which is it ???
</Bill>
I also think both (2) and (3) are not clean.

And I think we should not assume Ethertypes are only ARP, IPv4,
IPv6.
That assumption restricts the expantion of ther upper-layer of
the ipoverib link layer driver.
<Bill>
Correct.  I claim option 2 is the way to go.  Then you don't have to worry
about forcing all other encapsulations to include a 4 byte header in their
framing, I do not see a way of enforcing this anyway..

What are peoples objections to using a small number of QPN to handle upper
layer protocols on top of Infiniband.  My assumptions are QPN are very
similar to Ethertypes except they can be dynamic...
</Bill>


Billhttps://www1.ietf.org/mailman/listinfo/ipoverib



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 28 10:31:13 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA29288
	for <ipoverib-archive@odin.ietf.org>; Mon, 28 Jan 2002 10:31:13 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id KAA17249
	for ipoverib-archive@odin.ietf.org; Mon, 28 Jan 2002 10:31:17 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id KAA16821;
	Mon, 28 Jan 2002 10:29:48 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id KAA16763
	for <ipoverib@optimus.ietf.org>; Mon, 28 Jan 2002 10:29:45 -0500 (EST)
Received: from mail3.microsoft.com (mail3.microsoft.com [131.107.3.123])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA29154
	for <ipoverib@ietf.org>; Mon, 28 Jan 2002 10:29:40 -0500 (EST)
Received: from INET-VRS-03.redmond.corp.microsoft.com ([157.54.5.27]) by mail3.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Mon, 28 Jan 2002 07:29:11 -0800
Received: from 157.54.8.155 by INET-VRS-03.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Mon, 28 Jan 2002 07:29:11 -0800
Received: from red-msg-02.redmond.corp.microsoft.com ([157.54.12.70]) by inet-hub-04.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Mon, 28 Jan 2002 07:29:11 -0800
x-mimeole: Produced By Microsoft Exchange V6.0.5762.3
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Subject: RE: [ipoverib] resolving the Ethertype issue
Date: Mon, 28 Jan 2002 07:29:10 -0800
Message-ID: <24A715275661C8428C00432EFCA5CB7C055F9B2C@red-msg-02.redmond.corp.microsoft.com>
Thread-Topic: [ipoverib] resolving the Ethertype issue
Thread-Index: AcGn3BZqihp1QjpWQdmnBD8UXA3vSAAMVarg
From: "Jim Pinkerton" <jpink@microsoft.com>
To: <yaronh@voltaire.com>, "Bill Strahm" <bill@strahm.net>,
        <koji@cs.fujitsu.co.jp>, <ipoverib@ietf.org>
X-OriginalArrivalTime: 28 Jan 2002 15:29:11.0371 (UTC) FILETIME=[88F3D9B0:01C1A810]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id KAA16764
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit


I'm in the "resolve it quick" camp as well. I don't strongly believe one
way or the other, but would like to directly address Bill's question. As
Yaron states, the copy on current networking stacks occurs between the
transition from the protocol stack (SO layer), typically in the kernel,
to the application process, typically in user space. 

The transition from the device driver to the IP layer typically is done
by simply moving a pointer to the next header (i.e. no copy is
involved). For example, say the packet is:


+----------+   <- ptr handed to device driver
| LRH      |
+----------+
| GRH      |
+----------+
| BTH      |
+----------+
| DETH     |
+----------+   <- ptr handed to IP layer
| IP hdr   |
+----------+
| TCP hdr  |
+----------+   <- ptr handed to SO (sockets) layer
| payload  |
|          |
//        //
|          |
+----------+


So as the packet is handed up the stack, the same buffer is used and the
pointer into the buffer is simply advanced to the next header as the
packet is handed up the stack. Of course IB has some unique issues
because of the completion queue, so the DLL portion of the above is over
simplified, but it does not materially change the discussion. Then at
the final SO (sockets) layer, this is where either a page flip or a copy
occurs (in today's implementations) into the application buffer. Another
model which can avoid a copy is if the application api supports the
protocol layer giving the app a pointer to the buffer (rather then the
other way around, which is what sockets requires). In any of those cases
though, for the copy issue it is orthogonal as to whether the ethertype
is present or not. The ethertype simply adds another few bytes to
increment the pointer to get to the IP header (i.e. no big deal). 

So in conclusion I don't believe the copy avoidance algorithm is
materially effected by whether an ethertype is included or not.

On the issue of eating QP - if the goal is now to use one QP each for
IPv4, IPv6, and ARP, I am concerned. QP from our perspective are not
cheap. They should be used primarily if there is a security issue or
performance issue, and preferably all traffic should be able to co-exist
on the same QP otherwise. For the case of IPv4 mixing with IPv6 mixing
with ARP, it seems like any performance or security win is marginal,
thus making it seem reasonable to ensure that all traffic can go on a
single QP if possible, possibly while enabling implementations that want
to put them on different QP to do so. 

If that is the goal, it appears as though the cleanest implementation is
to have the ethertype. If it is not there though, I agree that we can
hack it up by looking at the first nibble and detecting whether it is an
ARP, IPv4, or IPv6 packet. I'm not sure this would make it through the
IESG though. 



Jim






-----Original Message-----
From: Yaron Haviv [mailto:yaronh@voltaire.com] 
Sent: Monday, January 28, 2002 1:14 AM
To: 'Bill Strahm'; koji@cs.fujitsu.co.jp; ipoverib@ietf.org
Subject: RE: [ipoverib] resolving the Ethertype issue

Bill,

No copy is/should be done between L2-L3, you just pass the buffer
pointer
(sk_buff in linux, m_buff in BSD)
The mandatory copy is between the kernel and user space (TCP/IP private
buffers to Application receive buffers)
If an application can work with the packets when they are in the TCP/IP
private buffers chains (without doing a receive(buf) op for example) so
it
could avoid the copy, usually it is not so simple, might work better for
kernel Apps
Another approach is to build TOE NIC's which place the data directly in
the
application receive buffers (where the private buffers are on the NIC,
no
magic)

The protocol demuxing relates to the execution process
The L2 look @ the protocol number, and calls the relevant protocol
function
(or soft IRQ) based on the value, this could also be done from L3 to L4

By the way adding more QP's doesn't solve any problem, it just requires
more
resources (I had a long email on that in the pass)

My take on the Ethertype issue is that who ever needs it when its not
provided could generate it for him self
Where who ever wants to ignore it when its available could also do that
This is why I don't think it is a critical decision if we do have it or
we
don't, since the raw information is provided
It is up to the group to see if someone feels (and could also explain)
that
the Ethertype provides him an Info he must have
And not just for future Ethernet over IB stuff since we deal with IP now

I personally could sleep well with the two decisions with/without, I
just
want to resolve it quick

Yaron


-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of
Bill Strahm
Sent: Monday, January 28, 2002 10:24 AM
To: koji@cs.fujitsu.co.jp; ipoverib@ietf.org
Subject: RE: [ipoverib] resolving the Ethertype issue

Comments inline...

I guess my big question is why are people afraid of allocating QPN for a
given use of Infiniband.  I thought that was the purpose of QPN.

Can someone tell me how they can redirect packets to the correct Layer 3
(with IB UD being a Layer 2 protocol for this discussion) using
Ethertype
without incurring either an additional copy, or a page flip ?  The way I
see
it is you will have to install a software "protocol demuxer" above the
IB
layer to receive the IB frame, look at the first 4 bytes and then put it
on
the appropriate protocols receive buffer.  I am most interested in
listening
to peoples implementations that involve zero-copy going up the TCP
stack.

Bill

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Koji takahara
Sent: Sunday, January 27, 2002 9:08 PM
To: ipoverib@ietf.org
Subject: RE: [ipoverib] resolving the Ethertype issue


I strongly agree with Roy.

I think the best method to identify the destination protocol is
the following.
(1) to specify a Ethertype as the first field of the payload
<Bill>
How do you propose handling the case where an Layer 3 (for IB that is)
protocol does not put a n-byte (I assume 4-byte) header in the message.
In
that case you CAN NOT determine if you are looking at a "EtherType" or
the
first 4 bytes of a payload therefore...
</Bill>
If we do not use method (1), we have to use one of the following
methods.
(2) to use different QPNs for different Ethertypes, or
<Bill>
I claim you will have to do this anyway
</Bill>
(3) to peek into an upper-level protocol frame in link layer
<Bill>
You consider looking at the first 4 bytes of the ULP acceptable, yet
here
you claim it is unclean... which is it ???
</Bill>
I also think both (2) and (3) are not clean.

And I think we should not assume Ethertypes are only ARP, IPv4,
IPv6.
That assumption restricts the expantion of ther upper-layer of
the ipoverib link layer driver.
<Bill>
Correct.  I claim option 2 is the way to go.  Then you don't have to
worry
about forcing all other encapsulations to include a 4 byte header in
their
framing, I do not see a way of enforcing this anyway..

What are peoples objections to using a small number of QPN to handle
upper
layer protocols on top of Infiniband.  My assumptions are QPN are very
similar to Ethertypes except they can be dynamic...
</Bill>


Billhttps://www1.ietf.org/mailman/listinfo/ipoverib



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 28 12:40:56 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA03644
	for <ipoverib-archive@odin.ietf.org>; Mon, 28 Jan 2002 12:40:55 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id MAA25860
	for ipoverib-archive@odin.ietf.org; Mon, 28 Jan 2002 12:40:57 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA25818;
	Mon, 28 Jan 2002 12:39:47 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA25784
	for <ipoverib@optimus.ietf.org>; Mon, 28 Jan 2002 12:39:46 -0500 (EST)
Received: from ntserver.voltaire.com (Volter-FW.ser.netvision.net.il [212.143.107.30])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA03617
	for <ipoverib@ietf.org>; Mon, 28 Jan 2002 12:39:43 -0500 (EST)
Received: by NTSERVER with Internet Mail Service (5.5.2653.19)
	id <CV7HBMM6>; Mon, 28 Jan 2002 19:35:11 +0200
Message-ID: <20083A3BAEF9D211BDB600805F8B14F301CBF18E@NTSERVER>
From: Yaron Haviv <yaronh@voltaire.com>
To: "'ipoverib@ietf.org'" <ipoverib@ietf.org>
Subject: RE: [ipoverib] moving forward with the open Issues - end node con
	figuration
Date: Mon, 28 Jan 2002 19:35:05 +0200
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Jerry,

I happened to check for you the FDDI<>Ethernet issue (through one of our
guys that built such a bridge in the pass)
The MTU problem I mentioned was exactly what happened there, those bridges
didn't really work well
You either manually configured the subnet for 1500, or you would implement
router farg/defrag functionality in those bridges
Eventually people mostly bought routers because of those reasons according
to him

It looks like history is going to repeat it self, since we do routers I
shouldn't really care, our competition should

Yaron

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
H.K. Jerry Chu
Sent: Sunday, January 27, 2002 10:57 PM
To: ipoverib@ietf.org; Yaron Haviv
Subject: RE: [ipoverib] moving forward with the open Issues - end node
configuration

Yaron,

>I guess you just missed my point; it was not to say that manual
>configuration is good, and we need to avoid the SM
>I just said that the process of agreeing on an automatic solution is not
>simple, and I would like to get an RFC first, than quarrel over this
>mechanism

Disagree. See my other mail. IETF already provides a number of mechanisms
for autoconfiguration that apply to IPoIB as well as any IPoverFoo. All
we need to solve here is to autoconfigure IPoIB links.

>Not to mention that it might involve additional standardization from IBTA
>(e.g. determining ServiceID's, etc') which wont make it faster
>This is the same reasoning like abandoning the capability bits issue today
>
>As for your proposal I'm not so sure that it is so efficient going over all
>the P_Key's defined, in some cases there could be more than a few
>I don't have strong feelings against it, just rather see something more
>efficient, if possible

Note first that this link configuration process only occur once per node
reboot. The efficiency concern IMHO is even less convincing here than the
one that argues for inlcuding LID in ARP. Second, there is nothing that
prevents implementations from supplementing additional configuration
files on an endnode that cache/specify IPoIB P_Keys, for example. These
local configuration practices don't present any interoperability problem,
therefore does not need to be standardized.

>On the same query to the SM we might also get more info on that IP subnet
>that will ease the configuration, and shorten the booting process
>
>Regarding the 1500 MTU, I know all about PMTU
>The 2K adds some load on the network, which also depends on the topology
and
>traffic characteristics
>It could be acceptable by many people
>I would just want to enable setting the values to a number lower than 2K,
if
>the admin requires so
>I'm not against setting 2K values for anyone that wants it, or even as the
>default magic number for IPoIB as long as smaller numbers are also valid
>And from your answer it looks like you agree to that too

Yup. I've been against some earlier proposals to designate a fixed MTU size
(e.g. 2K) for IPoIB because I don't think that is necessary. This is covered
in the link and multicast draft by using the MTU field in the MCGroupRecord.
But an issue just came up to me if one wants to support 1500 MTU. Contrary
to the "IP MTU" that allows one to specify an arbitrary size as long as
it's not greater than the link MTU, the "IB MTU" in the MCGroupRecord
field only allows fives choices from 256 to 4096.

We can either
1. punt it, since other IPoverFoo doesn't support the use of arbitrarily
configured MTU size either. (How does bridge vendors solve this problem when
bridging FDDI with Ethernet?)

or
2. find some place to store the IP MTU (ServiceRecord? I prefer not.)

Any suggestion?

Jerry

>
>Yaron
>
>-----Original Message-----
>From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
>H.K. Jerry Chu
>Sent: Sunday, January 27, 2002 1:25 AM
>To: ipoverib@ietf.org; Yaron Haviv
>Subject: RE: [ipoverib] moving forward with the open Issues - end node
>configuration
>
>Yaron,
>
>>I agree that some method should be specified in the future
>>However it is not so trivial to do it right now
>>a. There are issues between IBTA & IETF on things like service records
etc'
>>b. You could have few VLAN's on the same IB subnet, how would you
>>distinguish between them, etc' (maybe a service ID range)
>
>For each P_Key in a port's P_Key table, simply do a lookup to SA for
>an IPoIB MGID defined in the link and multicast draft. If the lookup
>succeeds, the IB partition defined by the P_Key will be used to host
>IPoIB, otherwise it won't. Isn't that pretty simple?
>
>>
>>I would tend to go for an RFC with a static configuration, with known
>>defaults like:
>>The default P_Key value, 1500 MTU
>>Work on a follow on proposal either in IBTA or IETF to address the
>>automation
>
>Static configuration is a thing of the past. Perhaps the motivation here
>is to bypass SM/SA again (like the LID proposal)? I can sympathize with
>the concern that SM/SA may be so complex that having a dependency on
>it is a bit risky. But let's see how feasible a bypass is. You can
>hardwire MTU, Q_Key, TClass... But you really need a broadcast mechanism
>(simulating broadcast by unicast is real messy) so you've got to have a
>MLID. Are you going to hardwire that too?
>
>If the concern is about the complexity of SM/SA, why not just dummy up
>a SM/SA implementation and hardwire all your favorite parameters there?
>This way we don't need to muck with IPoIB standard.
>
>>
>>You probably wonder why 1500, some of the reasons originally for
>determining
>>that IPoIB supporting devices will have minimum MTU of 2K was to maintain
>>similarity with Ethernet
>>Since in IB you have either 1K or 2K, 1500 falls in the 2K domain, so the
>>device must support 2K, not to say the MTU should by 2K
>>
>>Why 1500 again, lets say you have an Ethernet to IB bridge/router it will
>>have to fragment/reassemble the packets
>>This will create a lot of overhead on both the hosts and the routers that
>>will significantly affect performance
>
>There are two very different cases that you mentioned here, bridge and
>router. For the latter, no sweat, PMTU discovery takes care of it so
>you don't need to muck with individual link MTU. (That's the whole
>purpose of PMTU.) For bridges, no problem, just put your favorite
>MTU value in the MCGroupRecord when you set up an IPoIB link to bridge
>with an Ether segment.
>
>Jerry
>
>>
>>(it might be 1400 & something)
>>
>>Yaron
>>
>>-----Original Message-----
>>From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
>>Aditya Dube
>>Sent: Friday, January 25, 2002 9:17 PM
>>To: ipoverib@ietf.org
>>Subject: RE: [ipoverib] moving forward with the open Issues - end node
>>configuration
>>
>>
>>
>>The current IPOverIB spec requires that the P_Key and MTU be configured
>>on each node.
>>
>>The IPOverIB spec should provide a default method for an end-node to
>>come online with no IPoIB-specific configuration.
>>
>>During initialization, a node could query an IPOverIB specific
>>ServiceRecord (Table 164, pg 685). This query will return a P_Key and
>>ServiceSpecificFlags. The ServiceSpecificFlags can specify whether 2048
>>or 4096 is the link MTU.
>>
>>The ServiceRecord's ServiceID should have a default value specific to
>>IPOverIB. This number could be 0xFFFF:4000 in accordance with vol3
>>section on "ServiceID Namespace" (Table 128, pg 215).
>>
>>The ServiceRecord needs to be created just as the all-node or broadcast
>>MCGroup is created, i.e. by the first node on the network.
>>
>>If the Network Admin wishes to put the end-node on a different P_Key,
>>then the Admin may override the values returned by the ServiceRecord.
>>That is outside the scope of my proposal.
>>
>>
>>Aditya Dube
>>
>>
>>
>>_______________________________________________
>>IPoverIB mailing list
>>IPoverIB@ietf.org
>>https://www1.ietf.org/mailman/listinfo/ipoverib
>>
>>
>>_______________________________________________
>>IPoverIB mailing list
>>IPoverIB@ietf.org
>>https://www1.ietf.org/mailman/listinfo/ipoverib
>
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>https://www1.ietf.org/mailman/listinfo/ipoverib
>


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 28 13:04:01 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA04350
	for <ipoverib-archive@odin.ietf.org>; Mon, 28 Jan 2002 13:04:01 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id NAA27022
	for ipoverib-archive@odin.ietf.org; Mon, 28 Jan 2002 13:04:02 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA26985;
	Mon, 28 Jan 2002 13:02:32 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id LAA19153
	for <ipoverib@optimus.ietf.org>; Mon, 28 Jan 2002 11:02:40 -0500 (EST)
Received: from fgwmail5.fujitsu.co.jp (fgwmail5.fujitsu.co.jp [192.51.44.35])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA00716
	for <ipoverib@ietf.org>; Mon, 28 Jan 2002 11:02:34 -0500 (EST)
Received: from m4.gw.fujitsu.co.jp by fgwmail5.fujitsu.co.jp (8.9.3/3.7W-MX0201-Fujitsu Gateway)
	id BAA08871 for <ipoverib@ietf.org>; Tue, 29 Jan 2002 01:02:03 +0900 (JST)
	(envelope-from koji@lp.nm.fujitsu.co.jp)
Received: from lpdm.lp.nm.fujitsu.co.jp by m4.gw.fujitsu.co.jp (8.9.3/3.7W-0112-Fujitsu Domain Master)
	id BAA12460 for <ipoverib@ietf.org>; Tue, 29 Jan 2002 01:02:02 +0900 (JST)
	(envelope-from koji@lp.nm.fujitsu.co.jp)
Received: from ebara.lp.nm.fujitsu.co.jp (ebara [10.124.76.64])
	by lpdm.lp.nm.fujitsu.co.jp (8.9.3/3.6W) with SMTP id BAA27646
	for <ipoverib@ietf.org>; Tue, 29 Jan 2002 01:02:01 +0900 (JST)
Received: by ebara.lp.nm.fujitsu.co.jp (SMI-8.6/6.4J.6-lpslave03)
	id BAA19465; Tue, 29 Jan 2002 01:03:59 +0900
Date: Tue, 29 Jan 2002 01:03:59 +0900
From: Koji takahara <koji@lp.nm.fujitsu.co.jp>
Message-Id: <200201281603.BAA19465@ebara.lp.nm.fujitsu.co.jp>
To: ipoverib@ietf.org
Subject: RE: [ipoverib] resolving the Ethertype issue
In-Reply-To: <24A715275661C8428C00432EFCA5CB7C055F9B2C@red-msg-02.redmond.corp.microsoft.com>
References: <24A715275661C8428C00432EFCA5CB7C055F9B2C@red-msg-02.redmond.corp.microsoft.com>
Reply-To: koji@cs.fujitsu.co.jp
Mime-Version: 1.0 (generated by tm-edit 7.66)
Content-Type: text/plain; charset=US-ASCII
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Bill,

> I guess my big question is why are people afraid of allocating QPN for a
> given use of Infiniband.  I thought that was the purpose of QPN.

I think IPoverIB standard should not specify how to use 
multiple QPNs.

I am not afraid of using multiple QPNs.
I want to use multiple QPNs for the purpose of load balancing.
I do not want to use multiple QPNs for the purpose of 
distiguishsing ethertypes. 

> (2) to use different QPNs for different Ethertypes, or
> <Bill>
> I claim you will have to do this anyway
 
As Roy and Culley said, when using method (2), 
IPv4 and ARP must have special relation. 
I think it unclean.

As Yaron and Jim said, including ethertype at the head of 
payload does not have copy problem.

_______________________________________________________________
Koji Takahara  Email: koji@cs.fujitsu.co.jp
Development Dept.5,  Infrastructure Systems division,
Computer Group, Fujitsu Limitted


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 28 13:35:20 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA05349
	for <ipoverib-archive@odin.ietf.org>; Mon, 28 Jan 2002 13:35:20 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id NAA28524
	for ipoverib-archive@odin.ietf.org; Mon, 28 Jan 2002 13:35:22 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA28457;
	Mon, 28 Jan 2002 13:33:27 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA28429
	for <ipoverib@optimus.ietf.org>; Mon, 28 Jan 2002 13:33:26 -0500 (EST)
Received: from thalia.fm.intel.com (fmfdns02.fm.intel.com [132.233.247.11])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA05286
	for <ipoverib@ietf.org>; Mon, 28 Jan 2002 13:33:21 -0500 (EST)
Received: from fmsmsxvs043.fm.intel.com (fmsmsxv043-1.fm.intel.com [132.233.48.128])
	by thalia.fm.intel.com (8.9.1a+p1/8.9.1/d: relay.m4,v 1.49 2002/01/25 02:16:58 root Exp $) with SMTP id SAA19262
	for <ipoverib@ietf.org>; Mon, 28 Jan 2002 18:33:21 GMT
Received: from fmsmsx28.fm.intel.com ([132.233.42.28])
 by fmsmsxvs043.fm.intel.com (NAVGW 2.5.1.16) with SMTP id M2002012810323030911
 for <ipoverib@ietf.org>; Mon, 28 Jan 2002 10:32:30 -0800
Received: by fmsmsx28.fm.intel.com with Internet Mail Service (5.5.2653.19)
	id <DVG6VSBH>; Mon, 28 Jan 2002 10:33:20 -0800
Message-ID: <D9223EB959A5D511A98F00508B68C20C21A888@ORSMSX108>
From: "Larsen, Roy K" <roy.k.larsen@intel.com>
To: ipoverib@ietf.org
Subject: [ipoverib] Scope of QPN in ARP Request
Date: Mon, 28 Jan 2002 10:33:13 -0800
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="ISO-8859-1"
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

In the Ethertype thread, it was implied by some that the QPN could be used
to demultiplex all protocols carried by IPoIB.  For this to be true, nodes
receiving the ARP request broadcast can not assume that the source link
address is the destination link address for IPv4 frames.  That is, it is
only guaranteed to accept ARP replies.

The encapsulation draft must be clear on the scope of the QPN carried in an
ARP request.  If it only specifies the ARP reply queue, then a single ARP
request/reply transaction can not identify both IPv4 endpoints.

Since limiting the scope of the QPN in this way would double ARP traffic, I
would like to propose the encapsulation draft state that the QPN specified
in the ARP request source hardware address MUST be able to except all ARP
and IPv4 protocol frames.

Roy

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 28 13:57:49 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA05886
	for <ipoverib-archive@odin.ietf.org>; Mon, 28 Jan 2002 13:57:49 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id NAA29096
	for ipoverib-archive@odin.ietf.org; Mon, 28 Jan 2002 13:57:51 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA28889;
	Mon, 28 Jan 2002 13:52:47 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA28857
	for <ipoverib@optimus.ietf.org>; Mon, 28 Jan 2002 13:52:41 -0500 (EST)
Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA05764
	for <ipoverib@ietf.org>; Mon, 28 Jan 2002 13:52:38 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id MAA126022;
	Mon, 28 Jan 2002 12:48:10 -0600
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0SIqXF116496;
	Mon, 28 Jan 2002 13:52:34 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.6/8.11.2) with ESMTP id g0SIqXM09993;
	Mon, 28 Jan 2002 10:52:33 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0SIqWC04310;
	Mon, 28 Jan 2002 10:52:32 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201281852.g0SIqWC04310@eng4.beaverton.ibm.com>
Subject: Re: [ipoverib] resolving the Ethertype issue
To: koji@cs.fujitsu.co.jp
Date: Mon, 28 Jan 2002 10:52:31 -0800 (PST)
Cc: ipoverib@ietf.org
In-Reply-To: <200201281603.BAA19465@ebara.lp.nm.fujitsu.co.jp> from "Koji takahara" at Jan 29, 2002 12:03:59 AM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit


There have been a few notes referring to expanding IPoverIB to other
protocols. That is not correct. We are implementing 'IP over
InfiniBand' and that is it. Any other protocol that is/has to be
carried over IB will have to define its own encapsulation. We cannot
force them to use this unnecessary header ? So let us keep this
discussion limited to IP over InfiniBand and that implies IPv4/ARP and
IPv6. 

Those that prefer a type to distinguish between InfiniBand's
protocols, if they are IBTA members, can look at a proposal in LWG
that discusses including a type in the IB headers. This type will
allow all IB protocols to follow the same rules. Any format discussed
or specified in this forum(ipoverib) is limited, by definition, to
IPoverIB and is not binding on any other protocol. Thus there is no
way to distinguish the packets based on a header that is followed only
by 2 protocols (IP and ARP) if another protocol's packets come to the
same QP. The real solution of this type can come from IBTA only.

Basically, the desire is to receive IP and ARP on the same QP. To
distinguish these at the driver level (which many don't seem to
consider peeking into the next layer) you want a type. Why should it
matter what the size of the type is? If the driver looks at the first
4 bits it can easily distinguish between the three types
of(IPv4/ARP/IPv6). 

We should NOT be including the Ethertype for the following reasons:

      * there is no way to distinguish the ethertype from 'data' of
      another protocol since it is not bound by the rules of IPoverIB
      Since we limit the protocols by the QP then we can distinguish
      using the first 4 bits.

      * type in IB headers will make it unnecessary anyway

      * the receiver is limited to three protocols
            IPv4/ARP/IPv6

      The driver considers the first four bits as the type. It doesn't
      care about the rest of the payload and its format. The values
      are very specific: 4/6/0 respectively. The only difference from
      using a 4 byte ethertype is that the driver will not move the
      pointer forward to skip over the ethertype but just pass the
      received buffer as is to the relevant upper layer.

      * it doesn't seem right that to provide for some implementations
      wanting to use existing hardcoded values the solution should be
      pushed onto the wire. Instead it is very easy,for example, for
      the receiver to recognise a value of 4 in the first 4 bits and
      return it as 0x800 to the relevant function. Other
      implementations that merely pass the packet to the relevant
      protocol's handler and do not need a type can also work easily.


As far as I know the type is to distinguish the packets and pass them
to the relevant upper layer. The driver skips over the type and the
upper layer works with the rest of the packet. The same happens very
easily without the ethertype as described above.

Vivek
--
Vivek Kashyap
Linux Technology Center, IBM
vivk@us.ibm.com
kashyapv@us.ibm.com
Ph: 503 578 3422 T/L: 775 3422



                                                                                                                                       
                      Koji takahara                                                                                                    
                      <koji@lp.nm.fujit        To:       ipoverib@ietf.org                                                             
                      su.co.jp>                cc:                                                                                     
                      Sent by:                 Subject:  RE: [ipoverib] resolving the Ethertype issue                                  
                      ipoverib-admin@ie                                                                                                
                      tf.org                                                                                                           
                                                                                                                                       
                                                                                                                                       
                      01/27/2002 09:08                                                                                                 
                      PM                                                                                                               
                      Please respond to                                                                                                
                      koji                                                                                                             
                                                                                                                                       
                                                                                                                                       



I strongly agree with Roy.

I think the best method to identify the destination protocol is
the following.
(1) to specify a Ethertype as the first field of the payload

If we do not use method (1), we have to use one of the following
methods.
(2) to use different QPNs for different Ethertypes, or
(3) to peek into an upper-level protocol frame in link layer

I also think both (2) and (3) are not clean.

And I think we should not assume Ethertypes are only ARP, IPv4,
IPv6.
That assumption restricts the expantion of ther upper-layer of
the ipoverib link layer driver.

_______________________________________________________________
Koji Takahara  Email: koji@cs.fujitsu.co.jp
Development Dept.5,  Infrastructure Systems division,
Computer Group, Fujitsu Limitted


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib





-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 28 13:57:56 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA05899
	for <ipoverib-archive@odin.ietf.org>; Mon, 28 Jan 2002 13:57:52 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id NAA29078
	for ipoverib-archive@odin.ietf.org; Mon, 28 Jan 2002 13:57:42 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA29024;
	Mon, 28 Jan 2002 13:56:23 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA28992
	for <ipoverib@optimus.ietf.org>; Mon, 28 Jan 2002 13:56:21 -0500 (EST)
Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA05843
	for <ipoverib@ietf.org>; Mon, 28 Jan 2002 13:56:18 -0500 (EST)
Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210])
	by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id MAA43722;
	Mon, 28 Jan 2002 12:51:48 -0600
Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1])
	by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0SIuGF215578;
	Mon, 28 Jan 2002 13:56:16 -0500
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway.beaverton.ibm.com (8.10.0.Beta10/8.11.2) with ESMTP id g0SIuD926337;
	Mon, 28 Jan 2002 10:56:13 -0800 (PST)
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0SIuCL04805;
	Mon, 28 Jan 2002 10:56:12 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201281856.g0SIuCL04805@eng4.beaverton.ibm.com>
Subject: Re: [ipoverib] Scope of QPN in ARP Request
To: roy.k.larsen@intel.com (Larsen, Roy K)
Date: Mon, 28 Jan 2002 10:56:11 -0800 (PST)
Cc: ipoverib@ietf.org
In-Reply-To: <D9223EB959A5D511A98F00508B68C20C21A888@ORSMSX108> from "Larsen, Roy K" at Jan 28, 2002 09:33:13 AM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

> 
> In the Ethertype thread, it was implied by some that the QPN could be used
> to demultiplex all protocols carried by IPoIB.  For this to be true, nodes
> receiving the ARP request broadcast can not assume that the source link
> address is the destination link address for IPv4 frames.  That is, it is
> only guaranteed to accept ARP replies.
> 
> The encapsulation draft must be clear on the scope of the QPN carried in an
> ARP request.  If it only specifies the ARP reply queue, then a single ARP
> request/reply transaction can not identify both IPv4 endpoints.
> 
> Since limiting the scope of the QPN in this way would double ARP traffic, I
> would like to propose the encapsulation draft state that the QPN specified
> in the ARP request source hardware address MUST be able to except all ARP
> and IPv4 protocol frames.
> 

Yes, that is correct. The QPN included in the ARP request/reply MUST
accept both IPv4 and ARP packets in IBA's unreliable mode. Otherwise
we need to exchange two QPNs or exchange two messages.

Vivek

> Roy
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 28 14:10:24 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA06151
	for <ipoverib-archive@odin.ietf.org>; Mon, 28 Jan 2002 14:10:24 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id OAA00053
	for ipoverib-archive@odin.ietf.org; Mon, 28 Jan 2002 14:10:26 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA00019;
	Mon, 28 Jan 2002 14:09:07 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA29985
	for <ipoverib@optimus.ietf.org>; Mon, 28 Jan 2002 14:09:05 -0500 (EST)
Received: from thalia.fm.intel.com (fmfdns02.fm.intel.com [132.233.247.11])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA06102
	for <ipoverib@ietf.org>; Mon, 28 Jan 2002 14:08:56 -0500 (EST)
Received: from fmsmsxvs042.fm.intel.com (fmsmsxv042-1.fm.intel.com [132.233.48.110])
	by thalia.fm.intel.com (8.9.1a+p1/8.9.1/d: relay.m4,v 1.49 2002/01/25 02:16:58 root Exp $) with SMTP id TAA12567
	for <ipoverib@ietf.org>; Mon, 28 Jan 2002 19:08:55 GMT
Received: from fmsmsx019.fm.intel.com ([132.233.42.130])
 by fmsmsxvs042.fm.intel.com (NAVGW 2.5.1.16) with SMTP id M2002012811111611305
 ; Mon, 28 Jan 2002 11:11:16 -0800
Received: by fmsmsx019.fm.intel.com with Internet Mail Service (5.5.2653.19)
	id <DX9BWDZ9>; Mon, 28 Jan 2002 11:08:54 -0800
Message-ID: <D9223EB959A5D511A98F00508B68C20C21A889@ORSMSX108>
From: "Larsen, Roy K" <roy.k.larsen@intel.com>
To: "'Bill Strahm'" <bill@strahm.net>, koji@cs.fujitsu.co.jp,
        ipoverib@ietf.org
Subject: RE: [ipoverib] resolving the Ethertype issue
Date: Mon, 28 Jan 2002 11:08:49 -0800
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="iso-8859-1"
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

My comments embedded with <rkl>

-----Original Message-----
From: Bill Strahm [mailto:bill@strahm.net]
Sent: Monday, January 28, 2002 12:24 AM
To: koji@cs.fujitsu.co.jp; ipoverib@ietf.org
Subject: RE: [ipoverib] resolving the Ethertype issue


Comments inline...

I guess my big question is why are people afraid of allocating QPN for a
given use of Infiniband.  I thought that was the purpose of QPN.

Can someone tell me how they can redirect packets to the correct Layer 3
(with IB UD being a Layer 2 protocol for this discussion) using Ethertype
without incurring either an additional copy, or a page flip ?  The way I see
it is you will have to install a software "protocol demuxer" above the IB
layer to receive the IB frame, look at the first 4 bytes and then put it on
the appropriate protocols receive buffer.  I am most interested in listening
to peoples implementations that involve zero-copy going up the TCP stack.

Bill

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf
Of Koji takahara
Sent: Sunday, January 27, 2002 9:08 PM
To: ipoverib@ietf.org
Subject: RE: [ipoverib] resolving the Ethertype issue


I strongly agree with Roy.

I think the best method to identify the destination protocol is
the following.
(1) to specify a Ethertype as the first field of the payload
<Bill>
How do you propose handling the case where an Layer 3 (for IB that is)
protocol does not put a n-byte (I assume 4-byte) header in the message.  In
that case you CAN NOT determine if you are looking at a "EtherType" or the
first 4 bytes of a payload therefore...
</Bill>
<rkl>
I have seen this stated by others as well and I don't know what it means.
How would an arbitrary protocol choose the IPoIB QPN(s)?  I assume it would
be through their own address/neighbor resolution protocol which means it
would be under the control of the responding node and would presumably
followed a common link layer (from the OS point of view) protocol header.
So, if all Foo-over-IB encapsulation protocols used the same "link" header,
it would be very easy to integrate in to the interesting OSs network stack
with a common driver.  But that is not really the issue.  The issue is
protocol layering integrity. 
</rkl>

If we do not use method (1), we have to use one of the following
methods.
(2) to use different QPNs for different Ethertypes, or
<Bill>
I claim you will have to do this anyway
</Bill>
<rkl>
There is no need to do that at all. Our implementation will use a single QPN
for all encapsulation data because that is the most seamless way to
integrate into the demultiplexing interface of the OS.  But of course,
others are free to use more.
</rkl>
(3) to peek into an upper-level protocol frame in link layer
<Bill>
You consider looking at the first 4 bytes of the ULP acceptable, yet here
you claim it is unclean... which is it ???
</Bill>
I also think both (2) and (3) are not clean.

And I think we should not assume Ethertypes are only ARP, IPv4,
IPv6.
That assumption restricts the expantion of ther upper-layer of
the ipoverib link layer driver.
<Bill>
Correct.  I claim option 2 is the way to go.  Then you don't have to worry
about forcing all other encapsulations to include a 4 byte header in their
framing, I do not see a way of enforcing this anyway..

What are peoples objections to using a small number of QPN to handle upper
layer protocols on top of Infiniband.  My assumptions are QPN are very
similar to Ethertypes except they can be dynamic...
</Bill>
<rkl>
For ARP transaction efficiency, I believe APR and IPv4 must use the same
QPN.  I've started another thread to hopefully get consensus on this.
<rkl>


Billhttps://www1.ietf.org/mailman/listinfo/ipoverib



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 28 14:54:23 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA07223
	for <ipoverib-archive@odin.ietf.org>; Mon, 28 Jan 2002 14:54:18 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id OAA02044
	for ipoverib-archive@odin.ietf.org; Mon, 28 Jan 2002 14:54:21 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA02021;
	Mon, 28 Jan 2002 14:53:36 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA01989
	for <ipoverib@optimus.ietf.org>; Mon, 28 Jan 2002 14:53:35 -0500 (EST)
Received: from calliope1.fm.intel.com (fmfdns01.fm.intel.com [132.233.247.10])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA07203
	for <ipoverib@ietf.org>; Mon, 28 Jan 2002 14:53:31 -0500 (EST)
Received: from fmsmsxvs041.fm.intel.com (fmsmsxv041-1.fm.intel.com [132.233.48.109])
	by calliope1.fm.intel.com (8.9.1a+p1/8.9.1/d: relay.m4,v 1.49 2002/01/25 02:16:58 root Exp $) with SMTP id TAA27754
	for <ipoverib@ietf.org>; Mon, 28 Jan 2002 19:53:33 GMT
Received: from FMSMSX017.fm.intel.com ([132.233.42.196])
 by fmsmsxvs041.fm.intel.com (NAVGW 2.5.1.16) with SMTP id M2002012811542613107
 ; Mon, 28 Jan 2002 11:54:26 -0800
Received: by fmsmsx017.fm.intel.com with Internet Mail Service (5.5.2653.19)
	id <DX1PGP4X>; Mon, 28 Jan 2002 11:53:32 -0800
Message-ID: <D9223EB959A5D511A98F00508B68C20C21A88A@ORSMSX108>
From: "Larsen, Roy K" <roy.k.larsen@intel.com>
To: "'Vivek Kashyap'" <kashyapv@us.ibm.com>, koji@cs.fujitsu.co.jp
Cc: ipoverib@ietf.org
Subject: RE: [ipoverib] resolving the Ethertype issue
Date: Mon, 28 Jan 2002 11:53:31 -0800
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="iso-8859-1"
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

As I said in my first post, this issue is not worth dragging this into a
never ending rat hole for me.  So this will be my last public words on the
subject.

This is not an issue about enabling non-IPoverIB protocols.  The
encapsulation author has agreed that ARP and IPv4 MUST share the same QPN.
There are those of us who want to use a single link address for all IPoIB
traffic.  As a matter of protocol layering integrity, I support defining an
IPoIB encapsulation header that contains a destination protocol field.  I do
not know of a single widely used network protocol that does not include such
a destination specifier at every layer and, for the life of me, can't
understand resistance to one.  As Jim P. stated in a previous post, there
may be those in the standards community that will not indorse a protocol
that requires violating layering integrity.  Arguments that a proposed
change to the IBA will cover this are presumptuous and therefore irrelevant.

Yes, we can peek into the IP frames.  Yes we can peek into the ARP frame,
Yes, IP could peek into the TCP frame.  Yes, TCP could peek into user data.
It is just architecturally dishonest and without benefit.

Roy

-----Original Message-----
From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
Sent: Monday, January 28, 2002 10:53 AM
To: koji@cs.fujitsu.co.jp
Cc: ipoverib@ietf.org
Subject: Re: [ipoverib] resolving the Ethertype issue



There have been a few notes referring to expanding IPoverIB to other
protocols. That is not correct. We are implementing 'IP over
InfiniBand' and that is it. Any other protocol that is/has to be
carried over IB will have to define its own encapsulation. We cannot
force them to use this unnecessary header ? So let us keep this
discussion limited to IP over InfiniBand and that implies IPv4/ARP and
IPv6. 

Those that prefer a type to distinguish between InfiniBand's
protocols, if they are IBTA members, can look at a proposal in LWG
that discusses including a type in the IB headers. This type will
allow all IB protocols to follow the same rules. Any format discussed
or specified in this forum(ipoverib) is limited, by definition, to
IPoverIB and is not binding on any other protocol. Thus there is no
way to distinguish the packets based on a header that is followed only
by 2 protocols (IP and ARP) if another protocol's packets come to the
same QP. The real solution of this type can come from IBTA only.

Basically, the desire is to receive IP and ARP on the same QP. To
distinguish these at the driver level (which many don't seem to
consider peeking into the next layer) you want a type. Why should it
matter what the size of the type is? If the driver looks at the first
4 bits it can easily distinguish between the three types
of(IPv4/ARP/IPv6). 

We should NOT be including the Ethertype for the following reasons:

      * there is no way to distinguish the ethertype from 'data' of
      another protocol since it is not bound by the rules of IPoverIB
      Since we limit the protocols by the QP then we can distinguish
      using the first 4 bits.

      * type in IB headers will make it unnecessary anyway

      * the receiver is limited to three protocols
            IPv4/ARP/IPv6

      The driver considers the first four bits as the type. It doesn't
      care about the rest of the payload and its format. The values
      are very specific: 4/6/0 respectively. The only difference from
      using a 4 byte ethertype is that the driver will not move the
      pointer forward to skip over the ethertype but just pass the
      received buffer as is to the relevant upper layer.

      * it doesn't seem right that to provide for some implementations
      wanting to use existing hardcoded values the solution should be
      pushed onto the wire. Instead it is very easy,for example, for
      the receiver to recognise a value of 4 in the first 4 bits and
      return it as 0x800 to the relevant function. Other
      implementations that merely pass the packet to the relevant
      protocol's handler and do not need a type can also work easily.


As far as I know the type is to distinguish the packets and pass them
to the relevant upper layer. The driver skips over the type and the
upper layer works with the rest of the packet. The same happens very
easily without the ethertype as described above.

Vivek
--
Vivek Kashyap
Linux Technology Center, IBM
vivk@us.ibm.com
kashyapv@us.ibm.com
Ph: 503 578 3422 T/L: 775 3422



 

                      Koji takahara

                      <koji@lp.nm.fujit        To:       ipoverib@ietf.org

                      su.co.jp>                cc:

                      Sent by:                 Subject:  RE: [ipoverib]
resolving the Ethertype issue                                  
                      ipoverib-admin@ie

                      tf.org

 

 

                      01/27/2002 09:08

                      PM

                      Please respond to

                      koji

 

 




I strongly agree with Roy.

I think the best method to identify the destination protocol is
the following.
(1) to specify a Ethertype as the first field of the payload

If we do not use method (1), we have to use one of the following
methods.
(2) to use different QPNs for different Ethertypes, or
(3) to peek into an upper-level protocol frame in link layer

I also think both (2) and (3) are not clean.

And I think we should not assume Ethertypes are only ARP, IPv4,
IPv6.
That assumption restricts the expantion of ther upper-layer of
the ipoverib link layer driver.

_______________________________________________________________
Koji Takahara  Email: koji@cs.fujitsu.co.jp
Development Dept.5,  Infrastructure Systems division,
Computer Group, Fujitsu Limitted


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib





-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 28 20:24:03 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA12807
	for <ipoverib-archive@odin.ietf.org>; Mon, 28 Jan 2002 20:23:58 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id UAA15490
	for ipoverib-archive@odin.ietf.org; Mon, 28 Jan 2002 20:24:00 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA15420;
	Mon, 28 Jan 2002 20:22:27 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA15362
	for <ipoverib@optimus.ietf.org>; Mon, 28 Jan 2002 20:22:25 -0500 (EST)
Received: from thalia.fm.intel.com (fmfdns02.fm.intel.com [132.233.247.11])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA12776
	for <ipoverib@ietf.org>; Mon, 28 Jan 2002 20:22:22 -0500 (EST)
Received: from fmsmsxvs042.fm.intel.com (fmsmsxv042-1.fm.intel.com [132.233.48.110])
	by thalia.fm.intel.com (8.9.1a+p1/8.9.1/d: relay.m4,v 1.49 2002/01/25 02:16:58 root Exp $) with SMTP id BAA12635
	for <ipoverib@ietf.org>; Tue, 29 Jan 2002 01:22:23 GMT
Received: from FMSMSX018.fm.intel.com ([132.233.42.197])
 by fmsmsxvs042.fm.intel.com (NAVGW 2.5.1.16) with SMTP id M2002012817244501152
 for <ipoverib@ietf.org>; Mon, 28 Jan 2002 17:24:45 -0800
Received: by fmsmsx018.fm.intel.com with Internet Mail Service (5.5.2653.19)
	id <DZXBADPZ>; Mon, 28 Jan 2002 17:22:23 -0800
Message-ID: <D9223EB959A5D511A98F00508B68C20C06BC4E67@ORSMSX108>
From: "Larsen, Roy K" <roy.k.larsen@intel.com>
To: ipoverib@ietf.org
Subject: [ipoverib] SDP Capability Bit
Date: Mon, 28 Jan 2002 17:22:21 -0800
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="iso-8859-1"
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

I believe there is agreement for providing a mechanism in an IPoIB ARP to
indicate additional capabilities of the IPoIB infrastructure.  There has
been some discussion as to whether we should tackle the definition of these
bits at this time without the definition of the corresponding protocol.  I
agree with this position.

However, there is an IBA protocol definition that has a dependency on the
IPoIB address resolution protocol.  This is the Sockets Direct Protocol
(SDP) outlined in the Annex B draft.  The dependency is specified in section
B.4.

Given the interdependency of IPoIB and SDP, I propose that an SDP capability
bit be defined in the link address of an IPoIB ARP.  This will allow a node
to recognize that a target node supports SDP when it asks the IPoIB
implementation to resolve a target node address.

Comments?

Roy

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 28 20:35:36 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA13197
	for <ipoverib-archive@odin.ietf.org>; Mon, 28 Jan 2002 20:35:31 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id UAA16371
	for ipoverib-archive@odin.ietf.org; Mon, 28 Jan 2002 20:35:33 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA16328;
	Mon, 28 Jan 2002 20:34:47 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA16299
	for <ipoverib@optimus.ietf.org>; Mon, 28 Jan 2002 20:34:46 -0500 (EST)
Received: from mail6.microsoft.com (mail6.microsoft.com [131.107.3.126])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA13183
	for <ipoverib@ietf.org>; Mon, 28 Jan 2002 20:34:43 -0500 (EST)
Received: from inet-vrs-06.redmond.corp.microsoft.com ([157.54.6.201]) by mail6.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Mon, 28 Jan 2002 17:34:15 -0800
Received: from 157.54.5.25 by inet-vrs-06.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Mon, 28 Jan 2002 17:34:14 -0800
Received: from red-msg-02.redmond.corp.microsoft.com ([157.54.12.70]) by inet-hub-03.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Mon, 28 Jan 2002 17:34:14 -0800
x-mimeole: Produced By Microsoft Exchange V6.0.5762.3
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Subject: RE: [ipoverib] SDP Capability Bit
Date: Mon, 28 Jan 2002 17:34:14 -0800
Message-ID: <24A715275661C8428C00432EFCA5CB7C03FCAAA3@red-msg-02.redmond.corp.microsoft.com>
Thread-Topic: [ipoverib] SDP Capability Bit
Thread-Index: AcGoY3Pz7bn+98J6RhiVXJs/wUheeQAAVZEA
From: "Jim Pinkerton" <jpink@microsoft.com>
To: "Larsen, Roy K" <roy.k.larsen@intel.com>, <ipoverib@ietf.org>
X-OriginalArrivalTime: 29 Jan 2002 01:34:14.0316 (UTC) FILETIME=[0F31A6C0:01C1A865]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id UAA16300
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit


I personally can't see how to use this bit unfortunately. I don't think
it is worthwhile to define arbitrary bits unless we have a defined
mechanism for using them. I think that was the core of the argument for
simply calling them reserved bits - get the spec out now and then
afterwards we can always follow it up with an RFC for how some of the
reserved bits are consumed by additional functionality.


Jim



-----Original Message-----
From: Larsen, Roy K [mailto:roy.k.larsen@intel.com] 
Sent: Monday, January 28, 2002 5:22 PM
To: ipoverib@ietf.org
Subject: [ipoverib] SDP Capability Bit

I believe there is agreement for providing a mechanism in an IPoIB ARP
to
indicate additional capabilities of the IPoIB infrastructure.  There has
been some discussion as to whether we should tackle the definition of
these
bits at this time without the definition of the corresponding protocol.
I
agree with this position.

However, there is an IBA protocol definition that has a dependency on
the
IPoIB address resolution protocol.  This is the Sockets Direct Protocol
(SDP) outlined in the Annex B draft.  The dependency is specified in
section
B.4.

Given the interdependency of IPoIB and SDP, I propose that an SDP
capability
bit be defined in the link address of an IPoIB ARP.  This will allow a
node
to recognize that a target node supports SDP when it asks the IPoIB
implementation to resolve a target node address.

Comments?

Roy

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 28 22:11:51 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA15125
	for <ipoverib-archive@odin.ietf.org>; Mon, 28 Jan 2002 22:11:51 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id WAA19774
	for ipoverib-archive@odin.ietf.org; Mon, 28 Jan 2002 22:11:54 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA19753;
	Mon, 28 Jan 2002 22:11:27 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA19722
	for <ipoverib@optimus.ietf.org>; Mon, 28 Jan 2002 22:11:14 -0500 (EST)
Received: from dcmtechdom.dcmtech.co.in ([203.190.136.130])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA15115
	for <ipoverib@ietf.org>; Mon, 28 Jan 2002 22:10:59 -0500 (EST)
Received: by DCMTECHDOM with Internet Mail Service (5.5.2653.19)
	id <D59D3L80>; Tue, 29 Jan 2002 08:44:41 +0530
Message-ID: <7FADCB99FC82D41199F9000629A85D1A03069DCC@DCMTECHDOM>
From: Rakesh Khurana <Khurana.R@dcmtech.co.in>
To: ipoverib@ietf.org
Cc: infiniband_study@yahoogroups.com
Date: Tue, 29 Jan 2002 08:44:38 +0530
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="iso-8859-1"
Subject: [Ipoverib] InfiniBand Management Software-----DLID issue
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Hi All ,
We are develpoing the Management Software for the InfiniBand fabric. We are
stuck at one point.
Whenever the LID routed packet is receieved by the manaagement
entities(Agents) how the SLID and DLID is extracted as there is no such
field in LID routed Management Packet.

Rakesh Khurana
(Infini Group)
DCM Technologies 
Gurgaon , India

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 28 22:53:36 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA16578
	for <ipoverib-archive@odin.ietf.org>; Mon, 28 Jan 2002 22:53:36 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id WAA21279
	for ipoverib-archive@odin.ietf.org; Mon, 28 Jan 2002 22:53:39 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA21254;
	Mon, 28 Jan 2002 22:52:39 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA21219
	for <ipoverib@optimus.ietf.org>; Mon, 28 Jan 2002 22:52:37 -0500 (EST)
Received: from zcamail04.zca.compaq.com (zcamail04.zca.compaq.com [161.114.32.104])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA16556
	for <ipoverib@ietf.org>; Mon, 28 Jan 2002 22:52:33 -0500 (EST)
Received: by zcamail04.zca.compaq.com (Postfix, from userid 12345)
	id 9A1DB1552; Mon, 28 Jan 2002 19:55:40 -0800 (PST)
Received: from cacexg12.americas.cpqcorp.net (cacexg12.americas.cpqcorp.net [16.105.250.84])
	by zcamail04.zca.compaq.com (Postfix) with ESMTP
	id 8EE901402; Mon, 28 Jan 2002 19:55:40 -0800 (PST)
Received: from cacexc02.americas.cpqcorp.net ([16.105.250.92]) by cacexg12.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966);
	 Mon, 28 Jan 2002 22:51:31 -0500
X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Subject: RE: [Ipoverib] InfiniBand Management Software-----DLID issue
Date: Mon, 28 Jan 2002 19:51:30 -0800
Message-ID: <41A3F7855AE99D45B1BBB0C49C481AE72025A0@cacexc02.americas.cpqcorp.net>
Thread-Topic: [Ipoverib] InfiniBand Management Software-----DLID issue
Thread-Index: AcGoctV4UBJNhUaSQlqBBFNGrkaEzgABSc9Q
From: "Mehra, Pankaj" <pankaj.mehra@compaq.com>
To: "Rakesh Khurana" <Khurana.R@dcmtech.co.in>, <ipoverib@ietf.org>
Cc: <infiniband_study@yahoogroups.com>
X-OriginalArrivalTime: 29 Jan 2002 03:51:31.0270 (UTC) FILETIME=[3CCD1660:01C1A878]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id WAA21220
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

1.0.a version of Vol 1. Page 644. Table 155. SMP Directed Route format.

DrSLID and DrDLID.

The next such question will cost you money!

Pankaj


-----Original Message-----
From: Rakesh Khurana [mailto:Khurana.R@dcmtech.co.in]
Sent: Monday, January 28, 2002 7:15 PM
To: ipoverib@ietf.org
Cc: infiniband_study@yahoogroups.com
Subject: [Ipoverib] InfiniBand Management Software-----DLID issue


Hi All ,
We are develpoing the Management Software for the InfiniBand fabric. We
are
stuck at one point.
Whenever the LID routed packet is receieved by the manaagement
entities(Agents) how the SLID and DLID is extracted as there is no such
field in LID routed Management Packet.

Rakesh Khurana
(Infini Group)
DCM Technologies 
Gurgaon , India

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Mon Jan 28 22:56:58 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA16612
	for <ipoverib-archive@odin.ietf.org>; Mon, 28 Jan 2002 22:56:58 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id WAA21401
	for ipoverib-archive@odin.ietf.org; Mon, 28 Jan 2002 22:57:01 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA21391;
	Mon, 28 Jan 2002 22:56:33 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA21353
	for <ipoverib@optimus.ietf.org>; Mon, 28 Jan 2002 22:56:31 -0500 (EST)
Received: from dcmtechdom.dcmtech.co.in ([203.190.136.130])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA16605
	for <ipoverib@ietf.org>; Mon, 28 Jan 2002 22:56:24 -0500 (EST)
Received: by DCMTECHDOM with Internet Mail Service (5.5.2653.19)
	id <D59D3M1Z>; Tue, 29 Jan 2002 09:30:06 +0530
Message-ID: <7FADCB99FC82D41199F9000629A85D1A03069E63@DCMTECHDOM>
From: Rakesh Khurana <Khurana.R@dcmtech.co.in>
To: "Mehra, Pankaj" <pankaj.mehra@compaq.com>,
        Rakesh Khurana
	 <Khurana.R@dcmtech.co.in>, ipoverib@ietf.org
Cc: infiniband_study@yahoogroups.com
Subject: RE: [Ipoverib] InfiniBand Management Software-----DLID issue
Date: Tue, 29 Jan 2002 09:30:05 +0530
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="iso-8859-1"
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Hi Mr Pankaj 
I am not talking abt the Directed routed packet i am talking abt the LID
routed Management Packets.
Rakesh

-----Original Message-----
From: Mehra, Pankaj [mailto:pankaj.mehra@compaq.com]
Sent: Tuesday, January 29, 2002 9:22 AM
To: Rakesh Khurana; ipoverib@ietf.org
Cc: infiniband_study@yahoogroups.com
Subject: RE: [Ipoverib] InfiniBand Management Software-----DLID issue


1.0.a version of Vol 1. Page 644. Table 155. SMP Directed Route format.

DrSLID and DrDLID.

The next such question will cost you money!

Pankaj


-----Original Message-----
From: Rakesh Khurana [mailto:Khurana.R@dcmtech.co.in]
Sent: Monday, January 28, 2002 7:15 PM
To: ipoverib@ietf.org
Cc: infiniband_study@yahoogroups.com
Subject: [Ipoverib] InfiniBand Management Software-----DLID issue


Hi All ,
We are develpoing the Management Software for the InfiniBand fabric. We
are
stuck at one point.
Whenever the LID routed packet is receieved by the manaagement
entities(Agents) how the SLID and DLID is extracted as there is no such
field in LID routed Management Packet.

Rakesh Khurana
(Infini Group)
DCM Technologies 
Gurgaon , India

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 29 02:20:37 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA27298
	for <ipoverib-archive@odin.ietf.org>; Tue, 29 Jan 2002 02:20:33 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id CAA11630
	for ipoverib-archive@odin.ietf.org; Tue, 29 Jan 2002 02:20:34 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id CAA11372;
	Tue, 29 Jan 2002 02:14:56 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id CAA11337
	for <ipoverib@optimus.ietf.org>; Tue, 29 Jan 2002 02:14:53 -0500 (EST)
Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA27210
	for <ipoverib@ietf.org>; Tue, 29 Jan 2002 02:14:51 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e31.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id CAA55922;
	Tue, 29 Jan 2002 02:11:47 -0500
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0T7Eof50658;
	Tue, 29 Jan 2002 00:14:51 -0700
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.6/8.11.2) with ESMTP id g0T7EoM32703;
	Mon, 28 Jan 2002 23:14:50 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0T7En307923;
	Mon, 28 Jan 2002 23:14:49 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201290714.g0T7En307923@eng4.beaverton.ibm.com>
Subject: Re: [ipoverib] SDP Capability Bit
To: jpink@microsoft.com (Jim Pinkerton)
Date: Mon, 28 Jan 2002 23:14:48 -0800 (PST)
Cc: roy.k.larsen@intel.com (Larsen Roy K), ipoverib@ietf.org
In-Reply-To: <24A715275661C8428C00432EFCA5CB7C03FCAAA3@red-msg-02.redmond.corp.microsoft.com> from "Jim Pinkerton" at Jan 28, 2002 04:34:14 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Please refer to draft-kashyap-ipoib-advanced-capabilities-00.txt. It
defines an interoperable mechanism for such usage exactly on the lines
mentioned by Roy. The draft needs to be updated to drop the LID from
the link-descrtiption field. The draft, at present does not list the
SDP flag (it lists RC) but an SDP flag can be added to provide the
requested function.


Vivek

> 
> 
> I personally can't see how to use this bit unfortunately. I don't think
> it is worthwhile to define arbitrary bits unless we have a defined
> mechanism for using them. I think that was the core of the argument for
> simply calling them reserved bits - get the spec out now and then
> afterwards we can always follow it up with an RFC for how some of the
> reserved bits are consumed by additional functionality.
> 
> 
> Jim
> 
> 
> 
> -----Original Message-----
> From: Larsen, Roy K [mailto:roy.k.larsen@intel.com] 
> Sent: Monday, January 28, 2002 5:22 PM
> To: ipoverib@ietf.org
> Subject: [ipoverib] SDP Capability Bit
> 
> I believe there is agreement for providing a mechanism in an IPoIB ARP
> to
> indicate additional capabilities of the IPoIB infrastructure.  There has
> been some discussion as to whether we should tackle the definition of
> these
> bits at this time without the definition of the corresponding protocol.
> I
> agree with this position.
> 
> However, there is an IBA protocol definition that has a dependency on
> the
> IPoIB address resolution protocol.  This is the Sockets Direct Protocol
> (SDP) outlined in the Annex B draft.  The dependency is specified in
> section
> B.4.
> 
> Given the interdependency of IPoIB and SDP, I propose that an SDP
> capability
> bit be defined in the link address of an IPoIB ARP.  This will allow a
> node
> to recognize that a target node supports SDP when it asks the IPoIB
> implementation to resolve a target node address.
> 
> Comments?
> 
> Roy
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 29 03:35:12 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA28106
	for <ipoverib-archive@odin.ietf.org>; Tue, 29 Jan 2002 03:35:12 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id DAA15572
	for ipoverib-archive@odin.ietf.org; Tue, 29 Jan 2002 03:35:14 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id DAA15435;
	Tue, 29 Jan 2002 03:31:21 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id DAA15409
	for <ipoverib@optimus.ietf.org>; Tue, 29 Jan 2002 03:31:19 -0500 (EST)
Received: from ntserver.voltaire.com (Volter-FW.ser.netvision.net.il [212.143.107.30])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA28048
	for <ipoverib@ietf.org>; Tue, 29 Jan 2002 03:31:16 -0500 (EST)
Received: by NTSERVER with Internet Mail Service (5.5.2653.19)
	id <CV7HBMTD>; Tue, 29 Jan 2002 10:26:50 +0200
Message-ID: <20083A3BAEF9D211BDB600805F8B14F301CBF1A0@NTSERVER>
From: Yaron Haviv <yaronh@voltaire.com>
To: "'Larsen, Roy K'" <roy.k.larsen@intel.com>,
        "'ipoverib@ietf.org'"
	 <ipoverib@ietf.org>
Subject: RE: [ipoverib] SDP Capability Bit
Date: Tue, 29 Jan 2002 10:26:42 +0200
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="iso-8859-1"
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Roy,

As one of the advocates of SDP (Socket Direct) I would also like to see a
mechanism for it
However, I agree with Jim that first thing first lets finish this RFC

For the future it was mentioned several times that we could add a capability
bit for it
Or Jim could add his own proposal if it doesn't work for MS
I agree with you that we must provide a mechanism for it within IP over IB

I suggest postponing this issue for now, don't worry it wont be forgotten,
not by me

Yaron

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Larsen, Roy K
Sent: Tuesday, January 29, 2002 3:22 AM
To: ipoverib@ietf.org
Subject: [ipoverib] SDP Capability Bit

I believe there is agreement for providing a mechanism in an IPoIB ARP to
indicate additional capabilities of the IPoIB infrastructure.  There has
been some discussion as to whether we should tackle the definition of these
bits at this time without the definition of the corresponding protocol.  I
agree with this position.

However, there is an IBA protocol definition that has a dependency on the
IPoIB address resolution protocol.  This is the Sockets Direct Protocol
(SDP) outlined in the Annex B draft.  The dependency is specified in section
B.4.

Given the interdependency of IPoIB and SDP, I propose that an SDP capability
bit be defined in the link address of an IPoIB ARP.  This will allow a node
to recognize that a target node supports SDP when it asks the IPoIB
implementation to resolve a target node address.

Comments?

Roy

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 29 06:34:26 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA00107
	for <ipoverib-archive@odin.ietf.org>; Tue, 29 Jan 2002 06:34:26 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id GAA24433
	for ipoverib-archive@odin.ietf.org; Tue, 29 Jan 2002 06:34:28 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id GAA24415;
	Tue, 29 Jan 2002 06:33:35 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id GAA24384
	for <ipoverib@optimus.ietf.org>; Tue, 29 Jan 2002 06:33:29 -0500 (EST)
Received: from dcmtechdom.dcmtech.co.in ([203.190.136.130])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA00101
	for <ipoverib@ietf.org>; Tue, 29 Jan 2002 06:33:03 -0500 (EST)
Received: by DCMTECHDOM with Internet Mail Service (5.5.2653.19)
	id <D72STJA8>; Tue, 29 Jan 2002 17:06:39 +0530
Message-ID: <7FADCB99FC82D41199F9000629A85D1A0306A13A@DCMTECHDOM>
From: Rakesh Khurana <Khurana.R@dcmtech.co.in>
To: infiniband_study@yahoogroups.com
Cc: ipoverib@ietf.org
Date: Tue, 29 Jan 2002 17:06:38 +0530
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="iso-8859-1"
Subject: [Ipoverib] PortGUID field in NodeInfo
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Hi all,
On page 663 of IBA specs vol1a in table 122 of NodeInfo there is the field
for PortGUId and decscription of the field says 

"GUID of this port itself. One port within a node can return
the NodeGUID as its PortGUID if the port is an integral
part of the node and is not field-replaceable." 

and whats the significance of this field in NodeInfo since there is NodeGUID
for each Node.

Rakesh


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 29 09:03:51 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA03383
	for <ipoverib-archive@odin.ietf.org>; Tue, 29 Jan 2002 09:03:47 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id JAA02967
	for ipoverib-archive@odin.ietf.org; Tue, 29 Jan 2002 09:03:50 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id JAA02924;
	Tue, 29 Jan 2002 09:03:02 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id JAA02894
	for <ipoverib@optimus.ietf.org>; Tue, 29 Jan 2002 09:02:58 -0500 (EST)
Received: from ztxmail04.ztx.compaq.com (ztxmail04.ztx.compaq.com [161.114.1.208])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA03352
	for <ipoverib@ietf.org>; Tue, 29 Jan 2002 09:02:52 -0500 (EST)
Received: from cceexg13.americas.cpqcorp.net (cceexg13.americas.cpqcorp.net [16.110.250.119])
	by ztxmail04.ztx.compaq.com (Postfix) with ESMTP
	id 14977266D; Tue, 29 Jan 2002 08:02:25 -0600 (CST)
Received: from cceexc18.americas.cpqcorp.net ([16.110.250.64]) by cceexg13.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966);
	 Tue, 29 Jan 2002 08:02:24 -0600
X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Subject: RE: [Ipoverib] InfiniBand Management Software-----DLID issue
Date: Tue, 29 Jan 2002 08:02:24 -0600
Message-ID: <4D027986353D1341ADA4F2A4F8A170760E3180@cceexc18.americas.cpqcorp.net>
Thread-Topic: [Ipoverib] InfiniBand Management Software-----DLID issue
Thread-Index: AcGocuUN8nDz+crbRSWtePTQP7T0zAAWZ5/w
From: "Culley, Paul" <Paul.Culley@COMPAQ.com>
To: "Rakesh Khurana" <Khurana.R@dcmtech.co.in>, <ipoverib@ietf.org>
Cc: <infiniband_study@yahoogroups.com>
X-OriginalArrivalTime: 29 Jan 2002 14:02:24.0842 (UTC) FILETIME=[940852A0:01C1A8CD]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id JAA02895
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

DLID and SLID are in the packet header; LRH: DLID/SLID/SL.  See 1.0.a
page 128.  For HCAs these can be accessed for UD service in the Poll for
completion verb.  See page 534 table 84.  The DLID is actually only
returned as the low order "path" bits, to assemble the full DLID you
have to mix in the base DLID from the PortInfo Attribute, also
accessible for HCAs via the Query HCA verb.

Of course for TCAs, the designer is on his own for returning this
information.

Paul R. Culley
Compaq Sr. Fellow
281-514-5543


-----Original Message-----
From: Rakesh Khurana [mailto:Khurana.R@dcmtech.co.in]
Sent: Monday, January 28, 2002 9:15 PM
To: ipoverib@ietf.org
Cc: infiniband_study@yahoogroups.com
Subject: [Ipoverib] InfiniBand Management Software-----DLID issue


Hi All ,
We are develpoing the Management Software for the InfiniBand fabric. We
are
stuck at one point.
Whenever the LID routed packet is receieved by the manaagement
entities(Agents) how the SLID and DLID is extracted as there is no such
field in LID routed Management Packet.

Rakesh Khurana
(Infini Group)
DCM Technologies 
Gurgaon , India

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 29 11:47:55 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA08953
	for <ipoverib-archive@odin.ietf.org>; Tue, 29 Jan 2002 11:47:54 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id LAA12341
	for ipoverib-archive@odin.ietf.org; Tue, 29 Jan 2002 11:47:57 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id LAA12300;
	Tue, 29 Jan 2002 11:46:45 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id LAA12268
	for <ipoverib@optimus.ietf.org>; Tue, 29 Jan 2002 11:46:44 -0500 (EST)
Received: from thalia.fm.intel.com (fmfdns02.fm.intel.com [132.233.247.11])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA08901
	for <ipoverib@ietf.org>; Tue, 29 Jan 2002 11:46:39 -0500 (EST)
Received: from fmsmsxvs041.fm.intel.com (fmsmsxv041-1.fm.intel.com [132.233.48.109])
	by thalia.fm.intel.com (8.9.1a+p1/8.9.1/d: relay.m4,v 1.49 2002/01/25 02:16:58 root Exp $) with SMTP id QAA18778
	for <ipoverib@ietf.org>; Tue, 29 Jan 2002 16:46:30 GMT
Received: from FMSMSX017.fm.intel.com ([132.233.42.196])
 by fmsmsxvs041.fm.intel.com (NAVGW 2.5.1.16) with SMTP id M2002012908472313586
 for <ipoverib@ietf.org>; Tue, 29 Jan 2002 08:47:23 -0800
Received: by fmsmsx017.fm.intel.com with Internet Mail Service (5.5.2653.19)
	id <DX1PMX02>; Tue, 29 Jan 2002 08:46:29 -0800
Message-ID: <D9223EB959A5D511A98F00508B68C20C06BC4E69@ORSMSX108>
From: "Larsen, Roy K" <roy.k.larsen@intel.com>
To: ipoverib@ietf.org
Subject: RE: [ipoverib] SDP Capability Bit
Date: Tue, 29 Jan 2002 08:46:28 -0800
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="iso-8859-1"
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

It seems my proposal was a little premature.  I'd like to postpone its
consideration until after we finalize the encapsulation draft.

Roy

-----Original Message-----
From: Larsen, Roy K [mailto:roy.k.larsen@intel.com]
Sent: Monday, January 28, 2002 5:22 PM
To: ipoverib@ietf.org
Subject: [ipoverib] SDP Capability Bit


I believe there is agreement for providing a mechanism in an IPoIB ARP to
indicate additional capabilities of the IPoIB infrastructure.  There has
been some discussion as to whether we should tackle the definition of these
bits at this time without the definition of the corresponding protocol.  I
agree with this position.

However, there is an IBA protocol definition that has a dependency on the
IPoIB address resolution protocol.  This is the Sockets Direct Protocol
(SDP) outlined in the Annex B draft.  The dependency is specified in section
B.4.

Given the interdependency of IPoIB and SDP, I propose that an SDP capability
bit be defined in the link address of an IPoIB ARP.  This will allow a node
to recognize that a target node supports SDP when it asks the IPoIB
implementation to resolve a target node address.

Comments?

Roy

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 29 12:28:52 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA10375
	for <ipoverib-archive@odin.ietf.org>; Tue, 29 Jan 2002 12:28:52 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id MAA15887
	for ipoverib-archive@odin.ietf.org; Tue, 29 Jan 2002 12:28:55 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA15798;
	Tue, 29 Jan 2002 12:28:22 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA15731
	for <ipoverib@optimus.ietf.org>; Tue, 29 Jan 2002 12:28:19 -0500 (EST)
Received: from palrel11.hp.com (palrel11.hp.com [156.153.255.246])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA10315
	for <ipoverib@ietf.org>; Tue, 29 Jan 2002 12:27:59 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel11.hp.com (Postfix) with ESMTP id B62266003B6
	for <ipoverib@ietf.org>; Tue, 29 Jan 2002 09:27:32 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id JAA02162
	for <ipoverib@ietf.org>; Tue, 29 Jan 2002 09:27:32 -0800 (PST)
Message-Id: <5.1.0.14.2.20020127073528.00b1f640@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Sun, 27 Jan 2002 07:38:00 -0800
To: ipoverib@ietf.org
From: Michael Krause <krause@cup.hp.com>
Subject: RE: [ipoverib] moving forward with the open Issues - end node
  configuration
In-Reply-To: <OFBDD1AEE8.F7BE200D-ON88256B4C.006AF5B3@boulder.ibm.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

At 12:04 PM 1/25/2002 -0800, Vivek Kashyap wrote:

><VK> I am not sure what you gain by that. The P_Key is programmed into the
>port by the SM on startup. You already have it in the P_Key tables.  How
>you determine the P_Key to use is upto your administrator.

Concur with Vivek that P_Key, etc. management is outside the scope of the 
IP-over-IB spec as there are many policy decisions that need to be taken 
into account that are very IB specific and IP should not care about these 
policies.  There is precedence for such a stance as IP does not care about 
802.1Q (VLAN) and IB for this spec's purpose is a layer 2 which uses 
partitions conceptually similar to VLANs from the IP perspective and thus 
are transparent in terms of the management aspects.  The same can be said 
of service ID management.

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 29 12:28:55 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA10392
	for <ipoverib-archive@odin.ietf.org>; Tue, 29 Jan 2002 12:28:55 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id MAA15901
	for ipoverib-archive@odin.ietf.org; Tue, 29 Jan 2002 12:28:59 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA15809;
	Tue, 29 Jan 2002 12:28:23 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id MAA15733
	for <ipoverib@optimus.ietf.org>; Tue, 29 Jan 2002 12:28:19 -0500 (EST)
Received: from palrel11.hp.com (palrel11.hp.com [156.153.255.246])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA10316
	for <ipoverib@ietf.org>; Tue, 29 Jan 2002 12:27:59 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel11.hp.com (Postfix) with ESMTP id EA1606007D7
	for <ipoverib@ietf.org>; Tue, 29 Jan 2002 09:27:32 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id JAA02165
	for <ipoverib@ietf.org>; Tue, 29 Jan 2002 09:27:32 -0800 (PST)
Message-Id: <5.1.0.14.2.20020127081107.02c49258@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Sun, 27 Jan 2002 08:14:26 -0800
To: ipoverib@ietf.org
From: Michael Krause <krause@cup.hp.com>
Subject: Re: [ipoverib] reasons for not using LID ?
In-Reply-To: <200201242331.g0ONVCN04098@eng4.beaverton.ibm.com>
References: <IFEKJBDLPFOEGGPIIGMFEEGGCAAA.kamath@omegaband.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

At 03:31 PM 1/24/2002 -0800, Vivek Kashyap wrote:
>Ashwin,
>         That is a good point but connectivity is not guaranteed across
>         the IB fabric even if you exchange the GIDs. You may enquire
>         on a GID, received in a multicast (ARP/ND), but that does not
>         mean there is *any* path back to the sender.

 From a partition perspective, if one receives an ARP packet, the endnodes 
must be in the same partition.  Ideally, SAs / router protocols, etc. would 
be set up such that if the endnodes participate in such a multicast group 
that there indeed exists a unicast path between the endnodes else why link 
them together in a multicast group in the first place.  I see nothing wrong 
with IP assuming that at least one unicast path must exist between any two 
endnodes that can see each other from an ARP perspective.

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 29 13:06:21 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA11709
	for <ipoverib-archive@odin.ietf.org>; Tue, 29 Jan 2002 13:06:21 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id NAA18904
	for ipoverib-archive@odin.ietf.org; Tue, 29 Jan 2002 13:06:23 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA18772;
	Tue, 29 Jan 2002 13:05:10 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA18742
	for <ipoverib@optimus.ietf.org>; Tue, 29 Jan 2002 13:05:02 -0500 (EST)
Received: from mail.brocade.com (sj6-b.brocade.com [63.121.140.220])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA11674
	for <ipoverib@ietf.org>; Tue, 29 Jan 2002 13:05:00 -0500 (EST)
Received: from hq-ex-c2.brocade.com (hq-ex-c2 [192.168.126.35])
	by mail.brocade.com (8.8.8+Sun/8.8.8) with ESMTP id KAA28237;
	Tue, 29 Jan 2002 10:04:01 -0800 (PST)
Received: by hq-ex-c2.brocade.com with Internet Mail Service (5.5.2653.19)
	id <DTD6DW38>; Tue, 29 Jan 2002 10:04:00 -0800
Message-ID: <B6AB380934B5F0488974238446762E4A034E2E@hq-ex-5>
From: Brian Forbes <bforbes@brocade.com>
To: "'Rakesh Khurana'" <Khurana.R@dcmtech.co.in>, ipoverib@ietf.org
Cc: infiniband_study@yahoogroups.com
Subject: RE: [Ipoverib] InfiniBand Management Software-----DLID issue
Date: Tue, 29 Jan 2002 10:04:00 -0800
X-Mailer: Internet Mail Service (5.5.2653.19)
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

I'm always worried by questions that seem simple, but: are you referring to
something besides the existence of the DLID and SLID in the Local Route
Header of the Unreliable Datagram that carries the management packet? (e.g.
section 5.2.1 in Vol 1) Or are you referring to how these header fields get
passed to management software?

Brian Forbes
Brocade Communications 

-----Original Message-----
From: Rakesh Khurana [mailto:Khurana.R@dcmtech.co.in]
Sent: Monday, January 28, 2002 7:15 PM
To: ipoverib@ietf.org
Cc: infiniband_study@yahoogroups.com
Subject: [Ipoverib] InfiniBand Management Software-----DLID issue


Hi All ,
We are develpoing the Management Software for the InfiniBand fabric. We are
stuck at one point.
Whenever the LID routed packet is receieved by the manaagement
entities(Agents) how the SLID and DLID is extracted as there is no such
field in LID routed Management Packet.

Rakesh Khurana
(Infini Group)
DCM Technologies 
Gurgaon , India

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 29 13:12:14 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA11926
	for <ipoverib-archive@odin.ietf.org>; Tue, 29 Jan 2002 13:12:14 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id NAA19155
	for ipoverib-archive@odin.ietf.org; Tue, 29 Jan 2002 13:12:15 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA19115;
	Tue, 29 Jan 2002 13:11:47 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA19083
	for <ipoverib@optimus.ietf.org>; Tue, 29 Jan 2002 13:11:46 -0500 (EST)
Received: from mail.brocade.com (sj6-b.brocade.com [63.121.140.220])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA11910
	for <ipoverib@ietf.org>; Tue, 29 Jan 2002 13:11:43 -0500 (EST)
Received: from hq-ex-c2.brocade.com (hq-ex-c2 [192.168.126.35])
	by mail.brocade.com (8.8.8+Sun/8.8.8) with ESMTP id KAA28852;
	Tue, 29 Jan 2002 10:11:01 -0800 (PST)
Received: by hq-ex-c2.brocade.com with Internet Mail Service (5.5.2653.19)
	id <DTD6DWW2>; Tue, 29 Jan 2002 10:11:01 -0800
Message-ID: <B6AB380934B5F0488974238446762E4A034E2F@hq-ex-5>
From: Brian Forbes <bforbes@brocade.com>
To: "'Rakesh Khurana'" <Khurana.R@dcmtech.co.in>,
        infiniband_study@yahoogroups.com
Cc: ipoverib@ietf.org
Subject: RE: [Ipoverib] PortGUID field in NodeInfo
Date: Tue, 29 Jan 2002 10:11:00 -0800
X-Mailer: Internet Mail Service (5.5.2653.19)
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Hi, Rakesh, the Port GUID forms the basis for the port's global address(es)
(GIDs). Refer to 4.1.1 in
Vol. 1. Node GUIDs are used for other purposes, e.g. to convey topology
information. 

Brian Forbes
Brocade Communications 

-----Original Message-----
From: Rakesh Khurana [mailto:Khurana.R@dcmtech.co.in]
Sent: Tuesday, January 29, 2002 3:37 AM
To: infiniband_study@yahoogroups.com
Cc: ipoverib@ietf.org
Subject: [Ipoverib] PortGUID field in NodeInfo


Hi all,
On page 663 of IBA specs vol1a in table 122 of NodeInfo there is the field
for PortGUId and decscription of the field says 

"GUID of this port itself. One port within a node can return
the NodeGUID as its PortGUID if the port is an integral
part of the node and is not field-replaceable." 

and whats the significance of this field in NodeInfo since there is NodeGUID
for each Node.

Rakesh


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Tue Jan 29 13:16:16 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA12072
	for <ipoverib-archive@odin.ietf.org>; Tue, 29 Jan 2002 13:16:16 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id NAA19399
	for ipoverib-archive@odin.ietf.org; Tue, 29 Jan 2002 13:16:17 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA19345;
	Tue, 29 Jan 2002 13:15:19 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA20936
	for <ipoverib@optimus.ietf.org>; Mon, 28 Jan 2002 22:40:27 -0500 (EST)
Received: from dakiya.controlnet.co.in (hidden-user@dakiya.controlnet.co.in.116.54.202.in-addr.arpa [202.54.116.69] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA16411
	for <ipoverib@ietf.org>; Mon, 28 Jan 2002 22:40:21 -0500 (EST)
Received: from lloyd ([192.168.4.1]) by dakiya.controlnet.co.in
          (Netscape Messaging Server 4.15) with ESMTP id GQOL8S00.RL3;
          Tue, 29 Jan 2002 09:18:28 +0530 
Message-ID: <000b01c1a877$9b195db0$bd01a8c0@lloyd>
From: "Lloyd D'cruz" <lloyd@controlnet.co.in>
To: <infiniband_study@yahoogroups.com>, <ipoverib@ietf.org>
Cc: <infiniband_study@yahoogroups.com>
References: <7FADCB99FC82D41199F9000629A85D1A03069DCC@DCMTECHDOM>
Date: Tue, 29 Jan 2002 09:16:56 +0530
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2600.0000
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
Content-Transfer-Encoding: 7bit
Subject: [Ipoverib] Re: [infiniband_study] InfiniBand Management Software-----DLID issue
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

hi All,
I guess the SLID and DLID could be recovered from the Completion Queue
Entity for that Particular received Management Packet.

Regards
Lloyd
----- Original Message -----
From: "Rakesh Khurana" <Khurana.R@dcmtech.co.in>
To: <ipoverib@ietf.org>
Cc: <infiniband_study@yahoogroups.com>
Sent: Tuesday, January 29, 2002 8:44 AM
Subject: [infiniband_study] InfiniBand Management Software-----DLID issue


> Hi All ,
> We are develpoing the Management Software for the InfiniBand fabric. We
are
> stuck at one point.
> Whenever the LID routed packet is receieved by the manaagement
> entities(Agents) how the SLID and DLID is extracted as there is no such
> field in LID routed Management Packet.
>
> Rakesh Khurana
> (Infini Group)
> DCM Technologies
> Gurgaon , India
>
> ------------------------ Yahoo! Groups Sponsor ---------------------~-->
> Sponsored by VeriSign - The Value of Trust
> When building an e-commerce site, you want to start with a
> secure foundation. Learn how with VeriSign's FREE Guide.
> http://us.click.yahoo.com/kWSNbC/XdiDAA/yigFAA/26EolB/TM
> ---------------------------------------------------------------------~->
>
> To unsubscribe from this group, send an email to:
> infiniband_study-unsubscribe@yahoogroups.com
>
>
>
> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
>
>



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Tue Jan 29 18:04:38 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA23778
	for <ipoverib-archive@odin.ietf.org>; Tue, 29 Jan 2002 18:04:38 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id SAA06329
	for ipoverib-archive@odin.ietf.org; Tue, 29 Jan 2002 18:04:41 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA05517;
	Tue, 29 Jan 2002 17:54:01 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id RAA05487
	for <ipoverib@ns.ietf.org>; Tue, 29 Jan 2002 17:53:59 -0500 (EST)
Received: from mxout1.netvision.net.il (mxout1.netvision.net.il [194.90.9.20])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA23644
	for <ipoverib@ietf.org>; Tue, 29 Jan 2002 17:53:55 -0500 (EST)
Received: from vaio505he ([139.92.208.192]) by mxout1.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GQQ00F2L290FY@mxout1.netvision.net.il> for ipoverib@ietf.org;
 Wed, 30 Jan 2002 00:53:27 +0200 (IST)
Date: Wed, 30 Jan 2002 00:54:05 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: RE: [ipoverib] moving forward with the open Issues - end node
 configuration
In-reply-to: <126C80DA5F8FD111B4D200805F8B14F30AC91092@NTSERVER>
To: "'H.K. Jerry Chu'" <Jerry.Chu@eng.sun.com>, ipoverib@ietf.org
Reply-to: yaronh@voltaire.com
Message-id: <000b01c1a917$db481200$a3d05c8b@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: text/plain; charset=US-ASCII
Content-transfer-encoding: 7BIT
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7BIT

Jerry,

I agree that we could start with your proposal using the MCGroupRecord,
maybe in the future we could add a service record or something, like Aditya
Dube suggested

The only problem is that you cannot define an MTU of 1500 this way
I would like to have this option, in a way that could be configured somehow
It doesn't have to be the default choice, just an optional choice

Yaron

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
H.K. Jerry Chu
Sent: Sunday, January 27, 2002 10:57 PM
To: ipoverib@ietf.org; Yaron Haviv
Subject: RE: [ipoverib] moving forward with the open Issues - end node
configuration

Yaron,

>I guess you just missed my point; it was not to say that manual
>configuration is good, and we need to avoid the SM
>I just said that the process of agreeing on an automatic solution is not
>simple, and I would like to get an RFC first, than quarrel over this
>mechanism

Disagree. See my other mail. IETF already provides a number of mechanisms
for autoconfiguration that apply to IPoIB as well as any IPoverFoo. All
we need to solve here is to autoconfigure IPoIB links.

>Not to mention that it might involve additional standardization from IBTA
>(e.g. determining ServiceID's, etc') which wont make it faster
>This is the same reasoning like abandoning the capability bits issue today
>
>As for your proposal I'm not so sure that it is so efficient going over all
>the P_Key's defined, in some cases there could be more than a few
>I don't have strong feelings against it, just rather see something more
>efficient, if possible

Note first that this link configuration process only occur once per node
reboot. The efficiency concern IMHO is even less convincing here than the
one that argues for inlcuding LID in ARP. Second, there is nothing that
prevents implementations from supplementing additional configuration
files on an endnode that cache/specify IPoIB P_Keys, for example. These
local configuration practices don't present any interoperability problem,
therefore does not need to be standardized.

>On the same query to the SM we might also get more info on that IP subnet
>that will ease the configuration, and shorten the booting process
>
>Regarding the 1500 MTU, I know all about PMTU
>The 2K adds some load on the network, which also depends on the topology
and
>traffic characteristics
>It could be acceptable by many people
>I would just want to enable setting the values to a number lower than 2K,
if
>the admin requires so
>I'm not against setting 2K values for anyone that wants it, or even as the
>default magic number for IPoIB as long as smaller numbers are also valid
>And from your answer it looks like you agree to that too

Yup. I've been against some earlier proposals to designate a fixed MTU size
(e.g. 2K) for IPoIB because I don't think that is necessary. This is covered
in the link and multicast draft by using the MTU field in the MCGroupRecord.
But an issue just came up to me if one wants to support 1500 MTU. Contrary
to the "IP MTU" that allows one to specify an arbitrary size as long as
it's not greater than the link MTU, the "IB MTU" in the MCGroupRecord
field only allows fives choices from 256 to 4096.

We can either
1. punt it, since other IPoverFoo doesn't support the use of arbitrarily
configured MTU size either. (How does bridge vendors solve this problem when
bridging FDDI with Ethernet?)

or
2. find some place to store the IP MTU (ServiceRecord? I prefer not.)

Any suggestion?

Jerry

>
>Yaron
>
>-----Original Message-----
>From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
>H.K. Jerry Chu
>Sent: Sunday, January 27, 2002 1:25 AM
>To: ipoverib@ietf.org; Yaron Haviv
>Subject: RE: [ipoverib] moving forward with the open Issues - end node
>configuration
>
>Yaron,
>
>>I agree that some method should be specified in the future
>>However it is not so trivial to do it right now
>>a. There are issues between IBTA & IETF on things like service records
etc'
>>b. You could have few VLAN's on the same IB subnet, how would you
>>distinguish between them, etc' (maybe a service ID range)
>
>For each P_Key in a port's P_Key table, simply do a lookup to SA for
>an IPoIB MGID defined in the link and multicast draft. If the lookup
>succeeds, the IB partition defined by the P_Key will be used to host
>IPoIB, otherwise it won't. Isn't that pretty simple?
>
>>
>>I would tend to go for an RFC with a static configuration, with known
>>defaults like:
>>The default P_Key value, 1500 MTU
>>Work on a follow on proposal either in IBTA or IETF to address the
>>automation
>
>Static configuration is a thing of the past. Perhaps the motivation here
>is to bypass SM/SA again (like the LID proposal)? I can sympathize with
>the concern that SM/SA may be so complex that having a dependency on
>it is a bit risky. But let's see how feasible a bypass is. You can
>hardwire MTU, Q_Key, TClass... But you really need a broadcast mechanism
>(simulating broadcast by unicast is real messy) so you've got to have a
>MLID. Are you going to hardwire that too?
>
>If the concern is about the complexity of SM/SA, why not just dummy up
>a SM/SA implementation and hardwire all your favorite parameters there?
>This way we don't need to muck with IPoIB standard.
>
>>
>>You probably wonder why 1500, some of the reasons originally for
>determining
>>that IPoIB supporting devices will have minimum MTU of 2K was to maintain
>>similarity with Ethernet
>>Since in IB you have either 1K or 2K, 1500 falls in the 2K domain, so the
>>device must support 2K, not to say the MTU should by 2K
>>
>>Why 1500 again, lets say you have an Ethernet to IB bridge/router it will
>>have to fragment/reassemble the packets
>>This will create a lot of overhead on both the hosts and the routers that
>>will significantly affect performance
>
>There are two very different cases that you mentioned here, bridge and
>router. For the latter, no sweat, PMTU discovery takes care of it so
>you don't need to muck with individual link MTU. (That's the whole
>purpose of PMTU.) For bridges, no problem, just put your favorite
>MTU value in the MCGroupRecord when you set up an IPoIB link to bridge
>with an Ether segment.
>
>Jerry
>
>>
>>(it might be 1400 & something)
>>
>>Yaron
>>
>>-----Original Message-----
>>From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
>>Aditya Dube
>>Sent: Friday, January 25, 2002 9:17 PM
>>To: ipoverib@ietf.org
>>Subject: RE: [ipoverib] moving forward with the open Issues - end node
>>configuration
>>
>>
>>
>>The current IPOverIB spec requires that the P_Key and MTU be configured
>>on each node.
>>
>>The IPOverIB spec should provide a default method for an end-node to
>>come online with no IPoIB-specific configuration.
>>
>>During initialization, a node could query an IPOverIB specific
>>ServiceRecord (Table 164, pg 685). This query will return a P_Key and
>>ServiceSpecificFlags. The ServiceSpecificFlags can specify whether 2048
>>or 4096 is the link MTU.
>>
>>The ServiceRecord's ServiceID should have a default value specific to
>>IPOverIB. This number could be 0xFFFF:4000 in accordance with vol3
>>section on "ServiceID Namespace" (Table 128, pg 215).
>>
>>The ServiceRecord needs to be created just as the all-node or broadcast
>>MCGroup is created, i.e. by the first node on the network.
>>
>>If the Network Admin wishes to put the end-node on a different P_Key,
>>then the Admin may override the values returned by the ServiceRecord.
>>That is outside the scope of my proposal.
>>
>>
>>Aditya Dube
>>
>>
>>
>>_______________________________________________
>>IPoverIB mailing list
>>IPoverIB@ietf.org
>>https://www1.ietf.org/mailman/listinfo/ipoverib
>>
>>
>>_______________________________________________
>>IPoverIB mailing list
>>IPoverIB@ietf.org
>>https://www1.ietf.org/mailman/listinfo/ipoverib
>
>
>_______________________________________________
>IPoverIB mailing list
>IPoverIB@ietf.org
>https://www1.ietf.org/mailman/listinfo/ipoverib
>


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Wed Jan 30 01:41:31 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA02965
	for <ipoverib-archive@odin.ietf.org>; Wed, 30 Jan 2002 01:41:31 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id BAA04741
	for ipoverib-archive@odin.ietf.org; Wed, 30 Jan 2002 01:41:31 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id BAA04714;
	Wed, 30 Jan 2002 01:40:21 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id BAA04683
	for <ipoverib@ns.ietf.org>; Wed, 30 Jan 2002 01:40:18 -0500 (EST)
Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA02946
	for <ipoverib@ietf.org>; Wed, 30 Jan 2002 01:40:17 -0500 (EST)
Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24])
	by e31.co.us.ibm.com (8.9.3/8.9.3) with ESMTP id BAA29794;
	Wed, 30 Jan 2002 01:37:11 -0500
Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2])
	by westrelay03.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id g0U6eGq72612;
	Tue, 29 Jan 2002 23:40:16 -0700
Received: from eng4.beaverton.ibm.com (eng4.beaverton.ibm.com [138.95.7.64])
	by gateway1.beaverton.ibm.com (8.11.6/8.11.2) with ESMTP id g0U6eGM01022;
	Tue, 29 Jan 2002 22:40:16 -0800
Received: (from viv@localhost)
	by eng4.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) id g0U6eFc14537;
	Tue, 29 Jan 2002 22:40:15 -0800 (PST)
From: Vivek Kashyap <kashyapv@us.ibm.com>
Message-Id: <200201300640.g0U6eFc14537@eng4.beaverton.ibm.com>
Subject: Re: [ipoverib] moving forward with the open Issues - end node
To: yaronh@voltaire.com
Date: Tue, 29 Jan 2002 22:40:14 -0800 (PST)
Cc: Jerry.Chu@eng.sun.com ('H.K. Jerry Chu'), ipoverib@ietf.org
In-Reply-To: <000b01c1a917$db481200$a3d05c8b@vaio505he> from "Yaron Haviv" at Jan 29, 2002 11:54:05 PM PST
X-Mailer: ELM [version 2.5 PL3]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

Yaron,
        The MCGroupRecord provides the IPoIB link MTU not IP MTU. The
        IP MTU is set at the interface - by default implmentations
        would set it to the link MTU itself.

        If one wishes to set the IP MTU to 1500 one may do so. It
        would be wise to ensure that the IPoIB link supports 2K(set in
        MCGroupRecord) in this case otherwise every interface driver
        on the link must provide fragmentation/reassembly of the
        packets. 

        In IPv6 one can use router advertisement link option to set
        the IP MTU. I don't think there is a method for advertising a
        common MTU (though path MTU discovery will work within a
        subnet too) in a similar manner to IPv6 when using IPv4.

        Anyway why should this be any different from IP over any other
        media? In my opinion if some other solution is needed then
        that should be dealt with in terms of IP itself and not
        limited to IPoIB.

Vivek


> 
> Jerry,
> 
> I agree that we could start with your proposal using the MCGroupRecord,
> maybe in the future we could add a service record or something, like Aditya
> Dube suggested
> 
> The only problem is that you cannot define an MTU of 1500 this way
> I would like to have this option, in a way that could be configured somehow
> It doesn't have to be the default choice, just an optional choice
> 
> Yaron
> 
> -----Original Message-----
> From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
> H.K. Jerry Chu
> Sent: Sunday, January 27, 2002 10:57 PM
> To: ipoverib@ietf.org; Yaron Haviv
> Subject: RE: [ipoverib] moving forward with the open Issues - end node
> configuration
> 
> Yaron,
> 
> >I guess you just missed my point; it was not to say that manual
> >configuration is good, and we need to avoid the SM
> >I just said that the process of agreeing on an automatic solution is not
> >simple, and I would like to get an RFC first, than quarrel over this
> >mechanism
> 
> Disagree. See my other mail. IETF already provides a number of mechanisms
> for autoconfiguration that apply to IPoIB as well as any IPoverFoo. All
> we need to solve here is to autoconfigure IPoIB links.
> 
> >Not to mention that it might involve additional standardization from IBTA
> >(e.g. determining ServiceID's, etc') which wont make it faster
> >This is the same reasoning like abandoning the capability bits issue today
> >
> >As for your proposal I'm not so sure that it is so efficient going over all
> >the P_Key's defined, in some cases there could be more than a few
> >I don't have strong feelings against it, just rather see something more
> >efficient, if possible
> 
> Note first that this link configuration process only occur once per node
> reboot. The efficiency concern IMHO is even less convincing here than the
> one that argues for inlcuding LID in ARP. Second, there is nothing that
> prevents implementations from supplementing additional configuration
> files on an endnode that cache/specify IPoIB P_Keys, for example. These
> local configuration practices don't present any interoperability problem,
> therefore does not need to be standardized.
> 
> >On the same query to the SM we might also get more info on that IP subnet
> >that will ease the configuration, and shorten the booting process
> >
> >Regarding the 1500 MTU, I know all about PMTU
> >The 2K adds some load on the network, which also depends on the topology
> and
> >traffic characteristics
> >It could be acceptable by many people
> >I would just want to enable setting the values to a number lower than 2K,
> if
> >the admin requires so
> >I'm not against setting 2K values for anyone that wants it, or even as the
> >default magic number for IPoIB as long as smaller numbers are also valid
> >And from your answer it looks like you agree to that too
> 
> Yup. I've been against some earlier proposals to designate a fixed MTU size
> (e.g. 2K) for IPoIB because I don't think that is necessary. This is covered
> in the link and multicast draft by using the MTU field in the MCGroupRecord.
> But an issue just came up to me if one wants to support 1500 MTU. Contrary
> to the "IP MTU" that allows one to specify an arbitrary size as long as
> it's not greater than the link MTU, the "IB MTU" in the MCGroupRecord
> field only allows fives choices from 256 to 4096.
> 
> We can either
> 1. punt it, since other IPoverFoo doesn't support the use of arbitrarily
> configured MTU size either. (How does bridge vendors solve this problem when
> bridging FDDI with Ethernet?)
> 
> or
> 2. find some place to store the IP MTU (ServiceRecord? I prefer not.)
> 
> Any suggestion?
> 
> Jerry
> 
> >
> >Yaron
> >
> >-----Original Message-----
> >From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
> >H.K. Jerry Chu
> >Sent: Sunday, January 27, 2002 1:25 AM
> >To: ipoverib@ietf.org; Yaron Haviv
> >Subject: RE: [ipoverib] moving forward with the open Issues - end node
> >configuration
> >
> >Yaron,
> >
> >>I agree that some method should be specified in the future
> >>However it is not so trivial to do it right now
> >>a. There are issues between IBTA & IETF on things like service records
> etc'
> >>b. You could have few VLAN's on the same IB subnet, how would you
> >>distinguish between them, etc' (maybe a service ID range)
> >
> >For each P_Key in a port's P_Key table, simply do a lookup to SA for
> >an IPoIB MGID defined in the link and multicast draft. If the lookup
> >succeeds, the IB partition defined by the P_Key will be used to host
> >IPoIB, otherwise it won't. Isn't that pretty simple?
> >
> >>
> >>I would tend to go for an RFC with a static configuration, with known
> >>defaults like:
> >>The default P_Key value, 1500 MTU
> >>Work on a follow on proposal either in IBTA or IETF to address the
> >>automation
> >
> >Static configuration is a thing of the past. Perhaps the motivation here
> >is to bypass SM/SA again (like the LID proposal)? I can sympathize with
> >the concern that SM/SA may be so complex that having a dependency on
> >it is a bit risky. But let's see how feasible a bypass is. You can
> >hardwire MTU, Q_Key, TClass... But you really need a broadcast mechanism
> >(simulating broadcast by unicast is real messy) so you've got to have a
> >MLID. Are you going to hardwire that too?
> >
> >If the concern is about the complexity of SM/SA, why not just dummy up
> >a SM/SA implementation and hardwire all your favorite parameters there?
> >This way we don't need to muck with IPoIB standard.
> >
> >>
> >>You probably wonder why 1500, some of the reasons originally for
> >determining
> >>that IPoIB supporting devices will have minimum MTU of 2K was to maintain
> >>similarity with Ethernet
> >>Since in IB you have either 1K or 2K, 1500 falls in the 2K domain, so the
> >>device must support 2K, not to say the MTU should by 2K
> >>
> >>Why 1500 again, lets say you have an Ethernet to IB bridge/router it will
> >>have to fragment/reassemble the packets
> >>This will create a lot of overhead on both the hosts and the routers that
> >>will significantly affect performance
> >
> >There are two very different cases that you mentioned here, bridge and
> >router. For the latter, no sweat, PMTU discovery takes care of it so
> >you don't need to muck with individual link MTU. (That's the whole
> >purpose of PMTU.) For bridges, no problem, just put your favorite
> >MTU value in the MCGroupRecord when you set up an IPoIB link to bridge
> >with an Ether segment.
> >
> >Jerry
> >
> >>
> >>(it might be 1400 & something)
> >>
> >>Yaron
> >>
> >>-----Original Message-----
> >>From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
> >>Aditya Dube
> >>Sent: Friday, January 25, 2002 9:17 PM
> >>To: ipoverib@ietf.org
> >>Subject: RE: [ipoverib] moving forward with the open Issues - end node
> >>configuration
> >>
> >>
> >>
> >>The current IPOverIB spec requires that the P_Key and MTU be configured
> >>on each node.
> >>
> >>The IPOverIB spec should provide a default method for an end-node to
> >>come online with no IPoIB-specific configuration.
> >>
> >>During initialization, a node could query an IPOverIB specific
> >>ServiceRecord (Table 164, pg 685). This query will return a P_Key and
> >>ServiceSpecificFlags. The ServiceSpecificFlags can specify whether 2048
> >>or 4096 is the link MTU.
> >>
> >>The ServiceRecord's ServiceID should have a default value specific to
> >>IPOverIB. This number could be 0xFFFF:4000 in accordance with vol3
> >>section on "ServiceID Namespace" (Table 128, pg 215).
> >>
> >>The ServiceRecord needs to be created just as the all-node or broadcast
> >>MCGroup is created, i.e. by the first node on the network.
> >>
> >>If the Network Admin wishes to put the end-node on a different P_Key,
> >>then the Admin may override the values returned by the ServiceRecord.
> >>That is outside the scope of my proposal.
> >>
> >>
> >>Aditya Dube
> >>
> >>
> >>
> >>_______________________________________________
> >>IPoverIB mailing list
> >>IPoverIB@ietf.org
> >>https://www1.ietf.org/mailman/listinfo/ipoverib
> >>
> >>
> >>_______________________________________________
> >>IPoverIB mailing list
> >>IPoverIB@ietf.org
> >>https://www1.ietf.org/mailman/listinfo/ipoverib
> >
> >
> >_______________________________________________
> >IPoverIB mailing list
> >IPoverIB@ietf.org
> >https://www1.ietf.org/mailman/listinfo/ipoverib
> >
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 


-- 
Vivek Kashyap
Linux Technology Center, IBM
kashyapv@us.ibm.com 
vivk@us.ibm.com 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Wed Jan 30 01:45:46 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA03002
	for <ipoverib-archive@odin.ietf.org>; Wed, 30 Jan 2002 01:45:46 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id BAA04987
	for ipoverib-archive@odin.ietf.org; Wed, 30 Jan 2002 01:45:46 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id BAA04971;
	Wed, 30 Jan 2002 01:45:17 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id BAA04943
	for <ipoverib@ns.ietf.org>; Wed, 30 Jan 2002 01:45:16 -0500 (EST)
Received: from mail2.microsoft.com (mail2.microsoft.com [131.107.3.124])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA02996
	for <ipoverib@ietf.org>; Wed, 30 Jan 2002 01:45:14 -0500 (EST)
Received: from INET-VRS-02.redmond.corp.microsoft.com ([157.54.8.110]) by mail2.microsoft.com with Microsoft SMTPSVC(5.0.2195.4617);
	 Tue, 29 Jan 2002 22:44:42 -0800
Received: from 157.54.5.25 by INET-VRS-02.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Tue, 29 Jan 2002 22:44:42 -0800
Received: from red-imc-04.redmond.corp.microsoft.com ([157.54.2.168]) by inet-hub-03.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Tue, 29 Jan 2002 22:44:41 -0800
Received: from win-imc-02.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.84]) by red-imc-04.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966);
	 Tue, 29 Jan 2002 22:44:41 -0800
Received: from win-msg-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.52]) by win-imc-02.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3588.0);
	 Tue, 29 Jan 2002 22:42:24 -0800
X-MimeOLE: Produced By Microsoft Exchange V6.0.6132.0
Content-Class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Subject: RE: [ipoverib] moving forward with the open Issues - end node configuration
Date: Tue, 29 Jan 2002 22:42:23 -0800
Message-ID: <2E33960095B58E40A4D3345AB9F65EC102450CE3@win-msg-01.wingroup.windeploy.ntdev.microsoft.com>
Thread-Topic: [ipoverib] moving forward with the open Issues - end node configuration
Thread-Index: AcGoInhHqSaikP+ATRmo1fq08fF8SwBNKcEg
From: "Aditya Dube" <adube@windows.microsoft.com>
To: <ipoverib@ietf.org>
X-OriginalArrivalTime: 30 Jan 2002 06:42:24.0731 (UTC) FILETIME=[46C0D2B0:01C1A959]
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id BAA04944
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit


I would like to withdraw my proposal for zero config as we will need the
help of the IBTA to reserve special values for the Service Record.

Configuration of the P_Key and the MTU should be an end-node issue in
the IpOverIB spec. 


Aditya Dube


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 30 08:22:09 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA17429
	for <ipoverib-archive@odin.ietf.org>; Wed, 30 Jan 2002 08:22:09 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id IAA24183
	for ipoverib-archive@odin.ietf.org; Wed, 30 Jan 2002 08:22:10 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id IAA24033;
	Wed, 30 Jan 2002 08:18:17 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id IAA24001
	for <ipoverib@optimus.ietf.org>; Wed, 30 Jan 2002 08:18:14 -0500 (EST)
Received: from ntserver.voltaire.com (Volter-FW.ser.netvision.net.il [212.143.107.30])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA17323
	for <ipoverib@ietf.org>; Wed, 30 Jan 2002 08:18:10 -0500 (EST)
Received: by NTSERVER with Internet Mail Service (5.5.2653.19)
	id <CV7HBNCT>; Wed, 30 Jan 2002 15:13:53 +0200
Message-ID: <20083A3BAEF9D211BDB600805F8B14F301CBF1F0@NTSERVER>
From: Yaron Haviv <yaronh@voltaire.com>
To: "'Bill Strahm'" <bill@strahm.net>
Cc: "'ipoverib@ietf.org'" <ipoverib@ietf.org>
Subject: [ipoverib] summerising the last week & open issues
Date: Wed, 30 Jan 2002 15:13:52 +0200
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="windows-1255"
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id IAA24002
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 8bit

It looks to me like we had a lot of progress done in the last 2 weeks
I will try and summaries the issues, and status, to enable people to respond
on the last issues  

1. LID in ARP (Closed)
it was agreed to remove the LID from the ARP body, who ever needs it will
get it from the LRH

2. Capabilities bits (Closed)
no one rejected the idea of postponing the capability discussion to a
separate RFC
where another related issue is the SDP (Socket Direct) capability (also
postponed)

3. end node configuration (almost closed)
for now MS (Dube) has withdrawn from its proposal to do something automatic
with the end node configuration in the context of this RFC
we might target it in the future also with the help of IBTA (ServiceID,
etc’)
it might also solve my concern around the 1500MTU size, so I don’t mind
dropping the 1500MTU issue for now too

4. Ethertype field in data (need to close)
many people has mentioned that this issue is not critical, and doesn’t harm
anyone’s ability to implement a solution
it is more of a philosophical debate around strict layering vs. efficiency 
(and maybe also possibility to enable Ethernet over IB one day)
since it is not a critical issue I would tend to go with the Author (Vivek)
and leave it as it is specified already 

this one to my understanding is the only real open issue right now 
I hope we finalize it and get a consensus quick 

Thanks, Yaron



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Wed Jan 30 20:31:32 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA06012
	for <ipoverib-archive@odin.ietf.org>; Wed, 30 Jan 2002 20:31:31 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id UAA04798
	for ipoverib-archive@odin.ietf.org; Wed, 30 Jan 2002 20:31:33 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA04682;
	Wed, 30 Jan 2002 20:30:44 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA04635
	for <ipoverib@optimus.ietf.org>; Wed, 30 Jan 2002 20:30:42 -0500 (EST)
Received: from fgwmail6.fujitsu.co.jp (fgwmail6.fujitsu.co.jp [192.51.44.36])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA05993
	for <ipoverib@ietf.org>; Wed, 30 Jan 2002 20:30:39 -0500 (EST)
Received: from m3.gw.fujitsu.co.jp by fgwmail6.fujitsu.co.jp (8.9.3/3.7W-MX0201-Fujitsu Gateway)
	id KAA14292; Thu, 31 Jan 2002 10:30:08 +0900 (JST)
	(envelope-from imai@giulia.ds.oaks.cs.fujitsu.co.jp)
Received: from dm2.oaks.cs.fujitsu.co.jp by m3.gw.fujitsu.co.jp (8.9.3/3.7W-0112-Fujitsu Domain Master)
	id KAA03896; Thu, 31 Jan 2002 10:30:07 +0900 (JST)
	(envelope-from imai@giulia.ds.oaks.cs.fujitsu.co.jp)
Received: from lovewars.ds.oaks.cs.fujitsu.co.jp by dm2.oaks.cs.fujitsu.co.jp (8.7.4+2.6Wbeta6/3.4W4-MXM960524)
	id JAA20235; Thu, 31 Jan 2002 09:21:34 +0900 (JST)
Received: from giulia.ds.oaks.cs.fujitsu.co.jp by lovewars.ds.oaks.cs.fujitsu.co.jp (5.0/6.4J.6)
	id AA23437; Thu, 31 Jan 2002 10:51:02 --900
Received: from giulia (imai@localhost)
	by giulia.ds.oaks.cs.fujitsu.co.jp (8.10.2+Sun/8.10.2) with ESMTP id g0V1UIx02878;
	Thu, 31 Jan 2002 10:30:19 +0900 (JST)
Message-Id: <200201310130.g0V1UIx02878@giulia.ds.oaks.cs.fujitsu.co.jp>
To: Yaron Haviv <yaronh@voltaire.com>
Cc: "'Bill Strahm'" <bill@strahm.net>,
        "'ipoverib@ietf.org'" <ipoverib@ietf.org>
From: akihiko_imai@cs.fujitsu.co.jp
Subject: Re: [ipoverib] summerising the last week & open issues 
In-Reply-To: Your message of "Wed, 30 Jan 2002 15:13:52 JST."
             <20083A3BAEF9D211BDB600805F8B14F301CBF1F0@NTSERVER> 
Date: Thu, 31 Jan 2002 10:30:18 +0900
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

All,

>Message-ID: <20083A3BAEF9D211BDB600805F8B14F301CBF1F0@NTSERVER>
>From: Yaron Haviv <yaronh@voltaire.com>
>To: "'Bill Strahm'" <bill@strahm.net>
>Cc: "'ipoverib@ietf.org'" <ipoverib@ietf.org>
>Subject: [ipoverib] summerising the last week & open issues
>Date: Wed, 30 Jan 2002 15:13:52 +0200
>
>It looks to me like we had a lot of progress done in the last 2 weeks
>I will try and summaries the issues, and status, to enable people to respond
>on the last issues  

>4. Ethertype field in data (need to close)
>many people has mentioned that this issue is not critical, and doesn’t harm
>anyone’s ability to implement a solution
>it is more of a philosophical debate around strict layering vs. efficiency 
>(and maybe also possibility to enable Ethernet over IB one day)
>since it is not a critical issue I would tend to go with the Author (Vivek)
>and leave it as it is specified already 

If "ethertype' arguments was summarized so,
we would have some critical problems. 

We don't think these debate/argument as philosophical.
It effects our InfiniBand product development.
Without ethertype in payload, probably other vendors can't implement other
 protcols (than IPv4/v6/ARP) on IPoIB neither.

Of course, as Vivek wrote, this mailing-list focuses ipv4/ipv6/arp.
But I think IPoIB is infrastructure for network applications/protocols
 over InfiniBand environment.
If Ethertype exists in payload, many other developers could also
 use this IPoIB protocol to implement many applications to realize many market
 requirements. If not, vendors can't do so.

So we absolutely need ethertype field in payload (by using IBTA 1.0a. not 2.0).

For example, we implement some "protocols" over IPoIB framework, such as
 'multiplexing software (between arp/ipv4/ipv6 and ipoib driver) which uses
  multiple IPoIB interfaces',
 'cluster nodes patrol software which uses multiple IPoIB interfaces.'
 'rarp',etc.

They all are implemented by IPoIB driver's using ethertype in payload and 
IPoIB driver's dispatching protocols by ethertype value.
If info is only 4bit in arp/ipv4/ipv6 packet itself, we cannot implement
any other protocols/applications over IPoIB at all.

>this one to my understanding is the only real open issue right now 
>I hope we finalize it and get a consensus quick 
>
>Thanks, Yaron

Thanks,
------------------------------------------------------------------------------
Akihiko Imai/Development Dept.5, Infrastructure Systems Division, Fujitsu Ltd.
  e-mail: akihiko_imai@cs.fujitsu.co.jp  TEL: +81-044-754-3456 (ext)7113-8706
  address: 4-4-1 Kamiodanaka Nakahara-ku, Kawasaki 211-8588, Japan

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 31 11:52:56 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA03051
	for <ipoverib-archive@odin.ietf.org>; Thu, 31 Jan 2002 11:52:56 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id LAA04202
	for ipoverib-archive@odin.ietf.org; Thu, 31 Jan 2002 11:52:59 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id LAA04001;
	Thu, 31 Jan 2002 11:48:47 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id LAA03967
	for <ipoverib@optimus.ietf.org>; Thu, 31 Jan 2002 11:48:46 -0500 (EST)
Received: from palrel12.hp.com (palrel12.hp.com [156.153.255.237])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA02909
	for <ipoverib@ietf.org>; Thu, 31 Jan 2002 11:48:41 -0500 (EST)
Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230])
	by palrel12.hp.com (Postfix) with ESMTP id D9F01600C8D
	for <ipoverib@ietf.org>; Thu, 31 Jan 2002 08:48:14 -0800 (PST)
Received: from mkrause.cup.hp.com (mk731912.cup.hp.com [15.8.80.111])
	by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id IAA06446
	for <ipoverib@ietf.org>; Thu, 31 Jan 2002 08:48:14 -0800 (PST)
Message-Id: <5.1.0.14.2.20020131082019.03001d00@esalpha2.cup.hp.com>
X-Sender: krause@esalpha2.cup.hp.com
X-Mailer: QUALCOMM Windows Eudora Version 5.1
Date: Thu, 31 Jan 2002 08:23:08 -0800
To: ipoverib@ietf.org
From: Michael Krause <krause@cup.hp.com>
Subject: Re: [ipoverib] moving forward with the open Issues - end node
In-Reply-To: <200201300640.g0U6eFc14537@eng4.beaverton.ibm.com>
References: <000b01c1a917$db481200$a3d05c8b@vaio505he>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

At 10:40 PM 1/29/2002 -0800, Vivek Kashyap wrote:
>Yaron,
>         The MCGroupRecord provides the IPoIB link MTU not IP MTU. The
>         IP MTU is set at the interface - by default implmentations
>         would set it to the link MTU itself.

Assumption is that the multicast PMTU while at least on one or more IB 
subnets is set to 2KB or 4KB (function of what the underlying fabric 
supports).  For unicast, the normal IP PMTU discovery process is used and 
nothing new is required to be specified.

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 31 13:29:31 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA07038
	for <ipoverib-archive@odin.ietf.org>; Thu, 31 Jan 2002 13:29:31 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id NAA11851
	for ipoverib-archive@odin.ietf.org; Thu, 31 Jan 2002 13:29:32 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA11607;
	Thu, 31 Jan 2002 13:22:12 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA11574
	for <ipoverib@optimus.ietf.org>; Thu, 31 Jan 2002 13:22:09 -0500 (EST)
Received: from mxout1.netvision.net.il (mxout1.netvision.net.il [194.90.9.20])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA06855
	for <ipoverib@ietf.org>; Thu, 31 Jan 2002 13:22:05 -0500 (EST)
Received: from vaio505he ([139.92.208.50]) by mxout1.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GQT004DSEZKNW@mxout1.netvision.net.il> for ipoverib@ietf.org;
 Thu, 31 Jan 2002 20:21:26 +0200 (IST)
Date: Thu, 31 Jan 2002 20:19:17 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: RE: [ipoverib] summerising the last week & open issues
In-reply-to: <20083A3BAEF9D211BDB600805F8B14F301F24097@NTSERVER>
To: akihiko_imai@cs.fujitsu.co.jp, ipoverib@ietf.org
Reply-to: yaronh@voltaire.com
Message-id: <000201c1aa84$281d2c10$0100007f@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: text/plain; charset=windows-1255
Content-transfer-encoding: 7BIT
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7BIT

Lets review your arguments:
You want to use Ethertype to do non IPoIB stuff (even without the ethertype
you could support future IP stuff like maybe IP v 8)
This is not the focus of this group, even if you do that than it might not
work with other implementations that work strictly with IP

Regarding the second argument you could use different QP's in order to have
multiple IPoIB networks
And even then having Ethertype field doesn't help you multiplexing things
beyond layer 2

The reason for saying it's a philosophical question is that for the things
defined in the IPoIB charter (IP only)
You don't need the Ethertype

Yaron

 -----Original Message-----
From: 	ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]  On Behalf
Of akihiko_imai@cs.fujitsu.co.jp
Sent:	Thursday, January 31, 2002 3:30 AM
To:	Yaron Haviv
Cc:	'Bill Strahm'; 'ipoverib@ietf.org'
Subject:	Re: [ipoverib] summerising the last week & open issues

All,

>Message-ID: <20083A3BAEF9D211BDB600805F8B14F301CBF1F0@NTSERVER>
>From: Yaron Haviv <yaronh@voltaire.com>
>To: "'Bill Strahm'" <bill@strahm.net>
>Cc: "'ipoverib@ietf.org'" <ipoverib@ietf.org>
>Subject: [ipoverib] summerising the last week & open issues
>Date: Wed, 30 Jan 2002 15:13:52 +0200
>
>It looks to me like we had a lot of progress done in the last 2 weeks
>I will try and summaries the issues, and status, to enable people to
respond
>on the last issues

>4. Ethertype field in data (need to close)
>many people has mentioned that this issue is not critical, and doesn't harm
>anyone's ability to implement a solution
>it is more of a philosophical debate around strict layering vs. efficiency
>(and maybe also possibility to enable Ethernet over IB one day)
>since it is not a critical issue I would tend to go with the Author (Vivek)
>and leave it as it is specified already

If "ethertype' arguments was summarized so,
we would have some critical problems.

We don't think these debate/argument as philosophical.
It effects our InfiniBand product development.
Without ethertype in payload, probably other vendors can't implement other
 protcols (than IPv4/v6/ARP) on IPoIB neither.

Of course, as Vivek wrote, this mailing-list focuses ipv4/ipv6/arp.
But I think IPoIB is infrastructure for network applications/protocols
 over InfiniBand environment.
If Ethertype exists in payload, many other developers could also
 use this IPoIB protocol to implement many applications to realize many
market
 requirements. If not, vendors can't do so.

So we absolutely need ethertype field in payload (by using IBTA 1.0a. not
2.0).

For example, we implement some "protocols" over IPoIB framework, such as
 'multiplexing software (between arp/ipv4/ipv6 and ipoib driver) which uses
  multiple IPoIB interfaces',
 'cluster nodes patrol software which uses multiple IPoIB interfaces.'
 'rarp',etc.

They all are implemented by IPoIB driver's using ethertype in payload and
IPoIB driver's dispatching protocols by ethertype value.
If info is only 4bit in arp/ipv4/ipv6 packet itself, we cannot implement
any other protocols/applications over IPoIB at all.

>this one to my understanding is the only real open issue right now
>I hope we finalize it and get a consensus quick
>
>Thanks, Yaron

Thanks,
----------------------------------------------------------------------------
--
Akihiko Imai/Development Dept.5, Infrastructure Systems Division, Fujitsu
Ltd.
  e-mail: akihiko_imai@cs.fujitsu.co.jp  TEL: +81-044-754-3456
(ext)7113-8706
  address: 4-4-1 Kamiodanaka Nakahara-ku, Kawasaki 211-8588, Japan

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@optimus.ietf.org  Thu Jan 31 13:41:13 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA07454
	for <ipoverib-archive@odin.ietf.org>; Thu, 31 Jan 2002 13:41:08 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id NAA12773
	for ipoverib-archive@odin.ietf.org; Thu, 31 Jan 2002 13:41:11 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA12735;
	Thu, 31 Jan 2002 13:39:06 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id NAA12701
	for <ipoverib@optimus.ietf.org>; Thu, 31 Jan 2002 13:39:04 -0500 (EST)
Received: from mxout2.netvision.net.il (mxout2.netvision.net.il [194.90.9.21])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA07325
	for <ipoverib@ietf.org>; Thu, 31 Jan 2002 13:38:59 -0500 (EST)
Received: from vaio505he ([139.92.208.50]) by mxout2.netvision.net.il
 (iPlanet Messaging Server 5.1 (built Sep  5 2001))
 with ESMTPA id <0GQT00MC4FS4M0@mxout2.netvision.net.il> for ipoverib@ietf.org;
 Thu, 31 Jan 2002 20:38:31 +0200 (IST)
Date: Thu, 31 Jan 2002 20:36:34 +0200
From: Yaron Haviv <yaronh@voltaire.com>
Subject: RE: [ipoverib] moving forward with the open Issues - end node
In-reply-to: <5.1.0.14.2.20020131082019.03001d00@esalpha2.cup.hp.com>
To: "'Michael Krause'" <krause@cup.hp.com>
Cc: ipoverib@ietf.org
Reply-to: yaronh@voltaire.com
Message-id: <000f01c1aa86$93db31c0$0100007f@vaio505he>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7BIT

Mike,

If you look at my last email from yesterday
I acknowledge that, and say that we need to look into in the future and not
in this RFC, so this issue isn't open anymore
The only open issue currently is the Ethertype field

Yaron

-----Original Message-----
From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
Michael Krause
Sent: Thursday, January 31, 2002 6:23 PM
To: ipoverib@ietf.org
Subject: Re: [ipoverib] moving forward with the open Issues - end node

At 10:40 PM 1/29/2002 -0800, Vivek Kashyap wrote:
>Yaron,
>         The MCGroupRecord provides the IPoIB link MTU not IP MTU. The
>         IP MTU is set at the interface - by default implmentations
>         would set it to the link MTU itself.

Assumption is that the multicast PMTU while at least on one or more IB
subnets is set to 2KB or 4KB (function of what the underlying fabric
supports).  For unicast, the normal IP PMTU discovery process is used and
nothing new is required to be specified.

Mike


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Thu Jan 31 14:52:43 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA10078
	for <ipoverib-archive@odin.ietf.org>; Thu, 31 Jan 2002 14:52:43 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id OAA19365
	for ipoverib-archive@odin.ietf.org; Thu, 31 Jan 2002 14:52:46 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA19047;
	Thu, 31 Jan 2002 14:51:49 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA19005
	for <ipoverib@ns.ietf.org>; Thu, 31 Jan 2002 14:51:47 -0500 (EST)
Received: from vieo.com (vieo.com [216.30.79.131])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA10005
	for <ipoverib@ietf.org>; Thu, 31 Jan 2002 14:51:42 -0500 (EST)
Received: (from root@localhost)
	by vieo.com (8.11.2/8.11.2) id g0VJpEI42652
	for ipoverib@ietf.org; Thu, 31 Jan 2002 13:51:14 -0600 (CST)
	(envelope-from rronald@VIEO.com)
Received: from Fishhead (dhcp-10-2-15-164.vieo.com [10.2.15.164])
	by vieo.com (8.11.2/8.11.2) with SMTP id g0VJp9c42170;
	Thu, 31 Jan 2002 13:51:09 -0600 (CST)
	(envelope-from rronald@VIEO.com)
Message-ID: <01f901c1aa90$99ea0d70$a40f020a@Fishhead>
From: "Roger Ronald" <rronald@VIEO.com>
To: <yaronh@voltaire.com>, <akihiko_imai@cs.fujitsu.co.jp>,
        <ipoverib@ietf.org>
References: <000201c1aa84$281d2c10$0100007f@vaio505he>
Subject: Re: [ipoverib] summerising the last week & open issues
Date: Thu, 31 Jan 2002 13:50:53 -0600
MIME-Version: 1.0
Content-Type: text/plain;
	charset="windows-1255"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 5.50.4133.2400
X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400
X-scanner: scanned by Inflex 0.1.5c+ on vieo.com
Content-Transfer-Encoding: 7bit
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org
Content-Transfer-Encoding: 7bit

I'm convinced that the Ethertype is not required for IP over IB.

But if there are folks who want to take advantage of a traditional
differentiator that's always been available, it seems reasonable
to accomodate them. Making up for the lack of the LLC/SNAP
header field that does this is the only way to retain the same
demuxing paradigm that's been used since the 70s.

Some comments are also embedded below:

RR
----- Original Message ----- 
From: "Yaron Haviv" <yaronh@voltaire.com>
To: <akihiko_imai@cs.fujitsu.co.jp>; <ipoverib@ietf.org>
Sent: Thursday, January 31, 2002 12:19 PM
Subject: RE: [ipoverib] summerising the last week & open issues


> Lets review your arguments:

Their "argument" is that their desire to mux protocols in the fashion
that has been traditionally supported and that they would like this support
to continue in IP over IB. I believe that this is not an argument, but a
statement of fact.

> You want to use Ethertype to do non IPoIB stuff (even without the ethertype
> you could support future IP stuff like maybe IP v 8)
> This is not the focus of this group, even if you do that than it might not
> work with other implementations that work strictly with IP

It's a no-brainer that non-IP protocols won't get handled properly 
where there are not implemented. I doubt that anyone expects or 
wants different. They do want the "option" to support those protocols.

> Regarding the second argument you could use different QP's in order to have
> multiple IPoIB networks
> And even then having Ethertype field doesn't help you multiplexing things
> beyond layer 2
 
I doubt that anyone wants any help in demuxing beyond layer 2. They do
not "want" to uses different QPNs since that is a different demuxing method
than they use today.

> The reason for saying it's a philosophical question is that for the things
> defined in the IPoIB charter (IP only)
> You don't need the Ethertype

They want the Ethertype. It provides a value for them. 

> Yaron
> 
>  -----Original Message-----
> From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]  On Behalf
> Of akihiko_imai@cs.fujitsu.co.jp
> Sent: Thursday, January 31, 2002 3:30 AM
> To: Yaron Haviv
> Cc: 'Bill Strahm'; 'ipoverib@ietf.org'
> Subject: Re: [ipoverib] summerising the last week & open issues
> 
> All,
> 
> >Message-ID: <20083A3BAEF9D211BDB600805F8B14F301CBF1F0@NTSERVER>
> >From: Yaron Haviv <yaronh@voltaire.com>
> >To: "'Bill Strahm'" <bill@strahm.net>
> >Cc: "'ipoverib@ietf.org'" <ipoverib@ietf.org>
> >Subject: [ipoverib] summerising the last week & open issues
> >Date: Wed, 30 Jan 2002 15:13:52 +0200
> >
> >It looks to me like we had a lot of progress done in the last 2 weeks
> >I will try and summaries the issues, and status, to enable people to
> respond
> >on the last issues
> 
> >4. Ethertype field in data (need to close)
> >many people has mentioned that this issue is not critical, and doesn't harm
> >anyone's ability to implement a solution
> >it is more of a philosophical debate around strict layering vs. efficiency
> >(and maybe also possibility to enable Ethernet over IB one day)
> >since it is not a critical issue I would tend to go with the Author (Vivek)
> >and leave it as it is specified already
> 
> If "ethertype' arguments was summarized so,
> we would have some critical problems.
> 
> We don't think these debate/argument as philosophical.
> It effects our InfiniBand product development.
> Without ethertype in payload, probably other vendors can't implement other
>  protcols (than IPv4/v6/ARP) on IPoIB neither.
> 
> Of course, as Vivek wrote, this mailing-list focuses ipv4/ipv6/arp.
> But I think IPoIB is infrastructure for network applications/protocols
>  over InfiniBand environment.
> If Ethertype exists in payload, many other developers could also
>  use this IPoIB protocol to implement many applications to realize many
> market
>  requirements. If not, vendors can't do so.
> 
> So we absolutely need ethertype field in payload (by using IBTA 1.0a. not
> 2.0).
> 
> For example, we implement some "protocols" over IPoIB framework, such as
>  'multiplexing software (between arp/ipv4/ipv6 and ipoib driver) which uses
>   multiple IPoIB interfaces',
>  'cluster nodes patrol software which uses multiple IPoIB interfaces.'
>  'rarp',etc.
> 
> They all are implemented by IPoIB driver's using ethertype in payload and
> IPoIB driver's dispatching protocols by ethertype value.
> If info is only 4bit in arp/ipv4/ipv6 packet itself, we cannot implement
> any other protocols/applications over IPoIB at all.
> 
> >this one to my understanding is the only real open issue right now
> >I hope we finalize it and get a consensus quick
> >
> >Thanks, Yaron
> 
> Thanks,
> ----------------------------------------------------------------------------
> --
> Akihiko Imai/Development Dept.5, Infrastructure Systems Division, Fujitsu
> Ltd.
>   e-mail: akihiko_imai@cs.fujitsu.co.jp  TEL: +81-044-754-3456
> (ext)7113-8706
>   address: 4-4-1 Kamiodanaka Nakahara-ku, Kawasaki 211-8588, Japan
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 
> 
> _______________________________________________
> IPoverIB mailing list
> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 
> 


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Thu Jan 31 22:00:26 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA19237
	for <ipoverib-archive@odin.ietf.org>; Thu, 31 Jan 2002 22:00:26 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id WAA09150
	for ipoverib-archive@odin.ietf.org; Thu, 31 Jan 2002 22:00:30 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA08891;
	Thu, 31 Jan 2002 21:59:25 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id VAA08859
	for <ipoverib@ns.ietf.org>; Thu, 31 Jan 2002 21:59:22 -0500 (EST)
Received: from fgwmail5.fujitsu.co.jp (fgwmail5.fujitsu.co.jp [192.51.44.35])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA19214
	for <ipoverib@ietf.org>; Thu, 31 Jan 2002 21:59:17 -0500 (EST)
Received: from m6.gw.fujitsu.co.jp by fgwmail5.fujitsu.co.jp (8.9.3/3.7W-MX0201-Fujitsu Gateway)
	id LAA02080; Fri, 1 Feb 2002 11:58:43 +0900 (JST)
	(envelope-from imai@giulia.ds.oaks.cs.fujitsu.co.jp)
Received: from dm2.oaks.cs.fujitsu.co.jp by m6.gw.fujitsu.co.jp (8.9.3/3.7W-0112-Fujitsu Domain Master)
	id LAA16914; Fri, 1 Feb 2002 11:58:42 +0900
	(envelope-from imai@giulia.ds.oaks.cs.fujitsu.co.jp)
Received: from lovewars.ds.oaks.cs.fujitsu.co.jp by dm2.oaks.cs.fujitsu.co.jp (8.7.4+2.6Wbeta6/3.4W4-MXM960524)
	id KAA01631; Fri, 1 Feb 2002 10:50:04 +0900 (JST)
Received: from giulia.ds.oaks.cs.fujitsu.co.jp by lovewars.ds.oaks.cs.fujitsu.co.jp (5.0/6.4J.6)
	id AA29652; Fri, 1 Feb 2002 12:19:38 --900
Received: from giulia (imai@localhost)
	by giulia.ds.oaks.cs.fujitsu.co.jp (8.10.2+Sun/8.10.2) with ESMTP id g112wtc11153;
	Fri, 1 Feb 2002 11:58:55 +0900 (JST)
Message-Id: <200202010258.g112wtc11153@giulia.ds.oaks.cs.fujitsu.co.jp>
To: yaronh@voltaire.com
Cc: ipoverib@ietf.org
From: akihiko_imai@cs.fujitsu.co.jp
Subject: Re: [ipoverib] summerising the last week & open issues 
In-Reply-To: Your message of "Thu, 31 Jan 2002 20:19:17 JST."
             <000201c1aa84$281d2c10$0100007f@vaio505he> 
Date: Fri, 01 Feb 2002 11:58:55 +0900
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

Yaron,

>Lets review your arguments:
>You want to use Ethertype to do non IPoIB stuff (even without the ethertype
>you could support future IP stuff like maybe IP v 8)

Yes.
 I want to use Ethertype. Ethertype should be in ipoib payload, I think.
And I implemnt a QP per ipoib-interface, which handles multiple protocols
such as (a)ipv4,arp,ipv6, (b) rarp and other protocol modules multiplexing
 multiple ipoib-interfaces (between ip/arp/ipv6 and ipoib driver).
They all have its protocol identifier, i.e. Ethertype, sap.

>Regarding the second argument you could use different QP's in order to have
>multiple IPoIB networks
>And even then having Ethertype field doesn't help you multiplexing things
>beyond layer 2
>
>The reason for saying it's a philosophical question is that for the things
>defined in the IPoIB charter (IP only)
>You don't need the Ethertype
>
>Yaron

If Ethertype does not exist in payload, I cannot implement (b)type software.
(Even standard "rarp". I'd let me say Not "don't want to" but "cannot".)

Even if I use multiple QPs per ipoib-interface,
relation between ipv4 and arp is a exception.
If so, they would need to use the same QP as Paul,Roy,and Koji already wrote.
or we would need to make 'unclean' interfaces between ipv4 and arp.
(IMO, using multiple QPs is very suitable for load-balancing, e.g. for multiple
 IPv4 interfaces, multiple IPv6 interfaces.)
 on the other hand, using multiple QPs only to demultiplex received packet
 is too rich to me, I think. Probably many of you think/have some
 good/different ideas/merits for using different QPs per ipoib-interfaces.
 So I don't deny multiple QPs per ipoib-interface.
 But in my case, Multiple QPs are used for load-balancing by using multiple
 ipoib-interces (defining mutiple ip subnets on the same wire.)

As roy wrote and probably some other people think or said,
datalink driver(=ipoib driver)'s snooping upper layer's packet content is
 not right. I think ipoib sould not specify snooping upper layers packets.
I think Ethertype in payload solves these problem and it's .

>This is not the focus of this group, even if you do that than it might not
>work with other implementations that work strictly with IP

Yes. I know this group focuses on ipv4/arp/ipv6. (How about rarp ?)
Of course, I never say "Let's argue about other protocols itself".
I say IPoIB should specify other protocols CAN be carried on the same ipoib
 packet format(with Ethertype) and leave the other protocols specify their
 own protcols/reliability details,etc themselves over ipoib.

If Snooping upper layer packets and Being without Ethertype, they would limit
 some possibility of ipoib itself and its applied range, I think.

Thanks,
  Akihiko

> -----Original Message-----
>From: 	ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]  On Behalf
>Of akihiko_imai@cs.fujitsu.co.jp
>Sent:	Thursday, January 31, 2002 3:30 AM
>To:	Yaron Haviv
>Cc:	'Bill Strahm'; 'ipoverib@ietf.org'
>Subject:	Re: [ipoverib] summerising the last week & open issues
>
>All,
>
>>Message-ID: <20083A3BAEF9D211BDB600805F8B14F301CBF1F0@NTSERVER>
>>From: Yaron Haviv <yaronh@voltaire.com>
>>To: "'Bill Strahm'" <bill@strahm.net>
>>Cc: "'ipoverib@ietf.org'" <ipoverib@ietf.org>
>>Subject: [ipoverib] summerising the last week & open issues
>>Date: Wed, 30 Jan 2002 15:13:52 +0200
>>
>>It looks to me like we had a lot of progress done in the last 2 weeks
>>I will try and summaries the issues, and status, to enable people to
>respond
>>on the last issues
>
>>4. Ethertype field in data (need to close)
>>many people has mentioned that this issue is not critical, and doesn't harm
>>anyone's ability to implement a solution
>>it is more of a philosophical debate around strict layering vs. efficiency
>>(and maybe also possibility to enable Ethernet over IB one day)
>>since it is not a critical issue I would tend to go with the Author (Vivek)
>>and leave it as it is specified already
>
>If "ethertype' arguments was summarized so,
>we would have some critical problems.
>
>We don't think these debate/argument as philosophical.
>It effects our InfiniBand product development.
>Without ethertype in payload, probably other vendors can't implement other
> protcols (than IPv4/v6/ARP) on IPoIB neither.
>
>Of course, as Vivek wrote, this mailing-list focuses ipv4/ipv6/arp.
>But I think IPoIB is infrastructure for network applications/protocols
> over InfiniBand environment.
>If Ethertype exists in payload, many other developers could also
> use this IPoIB protocol to implement many applications to realize many
>market
> requirements. If not, vendors can't do so.
>
>So we absolutely need ethertype field in payload (by using IBTA 1.0a. not
>2.0).
>
>For example, we implement some "protocols" over IPoIB framework, such as
> 'multiplexing software (between arp/ipv4/ipv6 and ipoib driver) which uses
>  multiple IPoIB interfaces',
> 'cluster nodes patrol software which uses multiple IPoIB interfaces.'
> 'rarp',etc.
>
>They all are implemented by IPoIB driver's using ethertype in payload and
>IPoIB driver's dispatching protocols by ethertype value.
>If info is only 4bit in arp/ipv4/ipv6 packet itself, we cannot implement
>any other protocols/applications over IPoIB at all.
>
>>this one to my understanding is the only real open issue right now
>>I hope we finalize it and get a consensus quick
>>
>>Thanks, Yaron
------------------------------------------------------------------------------
Akihiko Imai/Development Dept.5, Infrastructure Systems Division, Fujitsu Ltd.
  e-mail: akihiko_imai@cs.fujitsu.co.jp  TEL: +81-044-754-3456 (ext)7113-8706
  address: 4-4-1 Kamiodanaka Nakahara-ku, Kawasaki 211-8588, Japan

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From daemon@ns.ietf.org  Thu Jan 31 22:21:27 2002
Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA19502
	for <ipoverib-archive@odin.ietf.org>; Thu, 31 Jan 2002 22:21:27 -0500 (EST)
Received: (from daemon@localhost)
	by optimus.ietf.org (8.9.1a/8.9.1) id WAA09907
	for ipoverib-archive@odin.ietf.org; Thu, 31 Jan 2002 22:21:31 -0500 (EST)
Received: from optimus.ietf.org (localhost [127.0.0.1])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA09849;
	Thu, 31 Jan 2002 22:20:09 -0500 (EST)
Received: from ietf.org (odin [132.151.1.176])
	by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id WAA09815
	for <ipoverib@ns.ietf.org>; Thu, 31 Jan 2002 22:20:07 -0500 (EST)
Received: from fgwmail7.fujitsu.co.jp (fgwmail7.fujitsu.co.jp [192.51.44.37])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA19481
	for <ipoverib@ietf.org>; Thu, 31 Jan 2002 22:20:01 -0500 (EST)
Received: from m2.gw.fujitsu.co.jp by fgwmail7.fujitsu.co.jp (8.9.3/3.7W-MX0201-Fujitsu Gateway)
	id MAA24424; Fri, 1 Feb 2002 12:19:33 +0900 (JST)
	(envelope-from imai@giulia.ds.oaks.cs.fujitsu.co.jp)
Received: from dm2.oaks.cs.fujitsu.co.jp by m2.gw.fujitsu.co.jp (8.9.3/3.7W-0112-Fujitsu Domain Master)
	id MAA08230; Fri, 1 Feb 2002 12:19:32 +0900 (JST)
	(envelope-from imai@giulia.ds.oaks.cs.fujitsu.co.jp)
Received: from lovewars.ds.oaks.cs.fujitsu.co.jp by dm2.oaks.cs.fujitsu.co.jp (8.7.4+2.6Wbeta6/3.4W4-MXM960524)
	id LAA01865; Fri, 1 Feb 2002 11:10:53 +0900 (JST)
Received: from giulia.ds.oaks.cs.fujitsu.co.jp by lovewars.ds.oaks.cs.fujitsu.co.jp (5.0/6.4J.6)
	id AA29839; Fri, 1 Feb 2002 12:40:28 --900
Received: from giulia (imai@localhost)
	by giulia.ds.oaks.cs.fujitsu.co.jp (8.10.2+Sun/8.10.2) with ESMTP id g113Jha11344;
	Fri, 1 Feb 2002 12:19:43 +0900 (JST)
Message-Id: <200202010319.g113Jha11344@giulia.ds.oaks.cs.fujitsu.co.jp>
To: yaronh@voltaire.com
Cc: "'Michael Krause'" <krause@cup.hp.com>, ipoverib@ietf.org
From: akihiko_imai@cs.fujitsu.co.jp
Subject: Re: [ipoverib] moving forward with the open Issues - end node 
In-Reply-To: Your message of "Thu, 31 Jan 2002 20:36:34 JST."
             <000f01c1aa86$93db31c0$0100007f@vaio505he> 
Date: Fri, 01 Feb 2002 12:19:43 +0900
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
X-BeenThere: ipoverib@ietf.org

>Date: Thu, 31 Jan 2002 20:36:34 +0200
>From: Yaron Haviv <yaronh@voltaire.com>
>Subject: RE: [ipoverib] moving forward with the open Issues - end node
>Mike,
>
>If you look at my last email from yesterday
>I acknowledge that, and say that we need to look into in the future and not
>in this RFC, so this issue isn't open anymore
>The only open issue currently is the Ethertype field
>
>Yaron

Please let me confirm this default MTU issue.

If Ethertype is not in payload, default IP-MTU is 2048.
Otherwise (=if Ethertype is in payload),
 default IP-MTU is not 2048.
It's 2048 - EterhType field length. (4? probably not 2)

>-----Original Message-----
>From: ipoverib-admin@ietf.org [mailto:ipoverib-admin@ietf.org]On Behalf Of
>Michael Krause
>Sent: Thursday, January 31, 2002 6:23 PM
>To: ipoverib@ietf.org
>Subject: Re: [ipoverib] moving forward with the open Issues - end node
>
>At 10:40 PM 1/29/2002 -0800, Vivek Kashyap wrote:
>>Yaron,
>>         The MCGroupRecord provides the IPoIB link MTU not IP MTU. The
>>         IP MTU is set at the interface - by default implmentations
>>         would set it to the link MTU itself.
>
>Assumption is that the multicast PMTU while at least on one or more IB
>subnets is set to 2KB or 4KB (function of what the underlying fabric
>supports).  For unicast, the normal IP PMTU discovery process is used and
>nothing new is required to be specified.
>
>Mike

Thanks,
------------------------------------------------------------------------------
Akihiko Imai/Development Dept.5, Infrastructure Systems Division, Fujitsu Ltd.
  e-mail: akihiko_imai@cs.fujitsu.co.jp  TEL: +81-044-754-3456 (ext)7113-8706
  address: 4-4-1 Kamiodanaka Nakahara-ku, Kawasaki 211-8588, Japan

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



