From ipoverib-admin@ietf.org  Mon Mar  3 19:21:13 2003
Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA19145
	for <ipoverib-archive@lists.ietf.org>; Mon, 3 Mar 2003 19:21:13 -0500 (EST)
Received: from www1.ietf.org (localhost.localdomain [127.0.0.1])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h240UQp07518;
	Mon, 3 Mar 2003 19:30:26 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h240Tnp07485
	for <ipoverib@optimus.ietf.org>; Mon, 3 Mar 2003 19:29:49 -0500
Received: from pheriche.sun.com (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA19099
	for <ipoverib@ietf.org>; Mon, 3 Mar 2003 19:19:03 -0500 (EST)
Received: from jurassic.eng.sun.com ([129.146.17.55])
	by pheriche.sun.com (8.9.3+Sun/8.9.3) with ESMTP id RAA14054
	for <ipoverib@ietf.org>; Mon, 3 Mar 2003 17:21:04 -0700 (MST)
Received: from taipei (taipei.Eng.Sun.COM [129.146.85.178])
	by jurassic.eng.sun.com (8.12.8+Sun/8.12.8) with SMTP id h240L3im307684
	for <ipoverib@ietf.org>; Mon, 3 Mar 2003 16:21:03 -0800 (PST)
Message-Id: <200303040021.h240L3im307684@jurassic.eng.sun.com>
Date: Mon, 3 Mar 2003 16:21:01 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
To: ipoverib@ietf.org
MIME-Version: 1.0
Content-Type: MULTIPART/mixed; BOUNDARY=Grist_of_Bees_910_000
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.6_06 SunOS 5.8 sun4u sparc 
Subject: [Ipoverib] draft-ietf-ipoib-link-multicast-03.txt
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-BeenThere: ipoverib@ietf.org
X-Mailman-Version: 2.0.12
Precedence: bulk
List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=unsubscribe>
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
List-Post: <mailto:ipoverib@ietf.org>
List-Help: <mailto:ipoverib-request@ietf.org?subject=help>
List-Subscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=subscribe>

--Grist_of_Bees_910_000
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: KG6RlEkcowMVQpsCzsLESA==

The link and multicast draft has been revised according to the IBA
1.1 specification. Most of the changes are in section 9 and a
restored section 10 on IP multicast routing. The draft revison
also incorporates various comments from the past, and a number of
minor corrections throughput the draft.

The draft was submitted to IETF last week, and is currently waiting
to be published. A copy of the draft is enclosed below. Please
review and comment so that the draft can go back to the IESG review
queue asap.

Thanks,

Jerry


--Grist_of_Bees_910_000
Content-Type: TEXT/plain; name="draft-ietf-ipoib-link-multicast-03.txt"; charset=us-ascii; x-unix-mode=0664
Content-Description: draft-ietf-ipoib-link-multicast-03.txt
Content-MD5: D2fWDLKs2m53Jkfe78hm5A==







INTERNET-DRAFT                                            H.K. Jerry Chu
<draft-ietf-ipoib-link-multicast-03.txt>                Sun Microsystems
                                                           Vivek Kashyap
                                                                     IBM
Expires: September, 2003                                     March, 2003


             IP link and multicast over InfiniBand networks


Status of this Memo

   This document is an Internet-Draft and is in full conformance with
   all provisions of Section 10 of RFC2026.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups. Note that other
   groups may also distribute working documents as Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time. It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

   Copyright (C) The Internet Society (2003).  All Rights Reserved.


Abstract

   This document specifies a method for setting up IP subnets and
   multicast services over InfiniBand(TM) networks. Discussions in this
   document are applicable to both IPv4 and IPv6, unless explicitly
   specified. A separate document will cover unicast and encapsulation
   of IP datagrams over InfiniBand networks.


Table of Contents
   1.0     Introduction
   2.0     Terminology
   3.0     Basic IPoIB Transport - Unreliable Datagram
   4.0     IB Multicast Architecture
   5.0     IB Links vs. IPoIB Links



Chu & Kashyap                                                   [Page 1]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   6.0     Setting up an IPoIB Link
   6.1     Maximum Transmission Unit
   6.2     IPoIB Link Q_Key
   6.3     Other Link Attributes
   7.0     The IPoIB Broadcast Group
   8.0     Mapping for other Multicast Groups
   9.0     Sending and Receiving IP Multicast Packets
   10.0    IP Multicast Routing
   11.0    Security Considerations
   12.0    Acknowledgments
   13.0    References
   14.0    Author's Address
   15.0    Full Copyright Statement


1.0 Introduction

   InfiniBand Architecture (IBA) defines four layers of network services
   corresponding to layer one through layer four of the OSI reference
   model.  For the purpose of running IP over an InfiniBand (IB)
   network, the IB link, network, and transport layers collectively
   constitute the data link layer to the IP stack. One can find a
   general overview of IB architecture related to IP networks in
   [IPoIB_ARCH].

   This document will focus on the necessary steps in order to lay out
   an IP network on top of an IB network. It will describe all the
   elements of an IP over InfiniBand (IPoIB) link, how to configure its
   associated attributes, and how to set up basic broadcast and
   multicast services for it. IPoIB links are the building blocks upon
   which an IP network consisting of many IP subnets connected by
   routers can be built.  Subnetting allows the containment of broadcast
   traffic within a single link. It also provides certain degree of
   isolation for the administration purpose between nodes on different
   subnets.

2.0 Terminology

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119].

3.0 Basic IPoIB Transport - Unreliable Datagram

   InfiniBand defines four types of transport services [IBTA]. They are
   reliable connection, unreliable connection, reliable datagram,
   unreliable datagram. IBA also defines a special raw datagram service
   for encapsulation purpose. Both unreliable datagram and raw datagram



Chu & Kashyap                                                   [Page 2]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   define support for multicast. They provide the basic transport
   mechanism that best matches the IP datagram paradigm.

   IB unreliable datagram provides many additional features such as the
   partition key (P_Key) protection, multiple queue pairs (QPs), and
   Q_Key protection. Moreover, it defines a 32-bit invariant CRC
   checksum, which provides a much stronger protection against data
   corruption, compared with the 16-bit CRC that a raw datagram carries.

   For these reasons, IB unreliable datagram is considered to be a much
   better choice as the basic IPoIB transport than the raw datagram, and
   is chosen as the default IPoIB transport mechanism ([IPoIB_ARCH],
   [IPoIB_ENCAP]).

4.0 IB Multicast Architecture

   The following discussion gives a short overview of the multicast
   architecture in InfiniBand. For a complete specification, the reader
   is referred to [IBTA].

   IBA defines two layers of multicast services. Its link layer uses
   multicast LIDs (MLIDs) in the Local Route Header (LRH). LIDs are
   allocated by the Subnet Manager (SM) and fall in the range between
   0xC0000 to 0xFFFE (approximately 16k). MLIDs are used by IB switches
   to program their multicast forwarding tables. An IB switch
   implementation may support much fewer MLIDs in its forwarding table
   though.

   The IB network layer uses multicast GIDs (MGIDs) in the Global Route
   Header (GRH). MGIDs closely resemble IPv6 multicast addresses [AARCH]
   shown below.

   |   8    |  4 |  4 |                  112 bits                   |
   +------ -+----+----+---------------------------------------------+
   |11111111|flgs|scop|                  group ID                   |
   +--------+----+----+---------------------------------------------+

                                 Figure 1

   [IPoIB_ARCH] describes each field in more details.

   Since every IB multicast packet is required to carry a LRH and a GRH,
   both a valid MGID and a valid MLID are needed before an IB multicast
   packet can be constructed.

   An IB multicast group is uniquely identified by a valid MGID. Before
   a MGID can be used within an IB subnet, either as a destination
   address of a multicast packet, or to represent a multicast group that



Chu & Kashyap                                                   [Page 3]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   an IB node can join, an IB multicast group corresponding to the MGID
   must be created through the Subnet Administrator (SA). Besides the
   the MGID, the creator of an IB multicast group must supply values of
   path MTU, P_Key, Q_Key, Service Level (SL), FlowLabel, TClass that
   are appropriate for all the potential clients of the multicast group
   to use. In return, SA will allocate a MLID to be used by switches in
   the local IB subnet.

   Unreliable multicast is defined by IBA as an optional functionality
   for channel adaptors (CAs) and switches. In today's IP technology,
   link multicast has become an indispensable function for better
   supporting a modern IP network. For this reason, it is required that
   an IPoIB fabric supports multicast. This includes all the CAs and
   switches that are part of an IP network.

5.0 IB Links vs. IPoIB Links

   A link segment on top of which an IP subnet can be configured is
   defined in [IPV6] as a communication facility or medium over which
   nodes can communicate at the "link" layer.  For most types of
   communication media, the boundary between different data link
   segments closely follows the physical topology of the network. For
   instance, an Ethernet network connected by switches, hubs, or bridges
   usually forms a single link segment and broadcast/multicast domain.
   Different Ethernet segments can be connected by IP routers at the
   network layer to form an IP network.

   InfiniBand defines its own link-layer and subnets consisting of nodes
   connected by IB switches and routers. However, the IPoIB link
   boundary need not follow the IB link boundary. Nodes residing on
   different IB subnets can still communicate directly with one another
   through IB routers at the InfiniBand network layer. This
   communication at the network layer applies to unicast as well as
   multicast.

   The ultimate requirement for two nodes in the same IB fabric to
   communicate at the IB level, besides physical connectivity, is a
   common P_Key.

   Partitioning in IB provides an isolation mechanism among nodes in an
   IB fabric, much like VLANs in the Ethernet network.  Each port of an
   HCA (Host Channel Adaptor) contains a P_Key table holding all the
   valid P_Keys the port is allowed to use. The P_Key table is set up by
   the SM of the local IB subnet. Each QP is programmed with a P_Key
   from the local P_Key table. This P_Key is carried in all the outgoing
   packets from the QP, and is used to compare against the P_Key of all
   incoming packets to the QP. Any packet with an invalid P_Key will be
   discarded by the QP and a P_Key violation trap will be generated.  IB



Chu & Kashyap                                                   [Page 4]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   switches may optionally enforce partition checking too.

   Following the above, IB partitions are the natural choice for
   defining IPoIB link boundary. It also provides much needed
   flexibility for a network administrator to group nodes logically into
   different subnets in a large network.

6.0 Setting up an IPoIB Link

   A network administrator defines an IPoIB link by setting up an IB
   partition and assigning it a unique P_Key. Since a full-duplex
   communication is required among IP nodes, full-membership P_Keys,
   that is, those with the high-order bit set to 1 shall be used. An IB
   partition may or may not span multiple IB subnets; and whether it
   does or not is mostly transparent to IPoIB.

   Each node attached to an IB partition MUST have one of its HCAs
   assigned the P_Key to use. Note that the P_key table of an HCA port
   may contain many P_Keys. It is up to the implementation to define the
   method by which the P_Key relevant to a particular IPoIB subnet is
   determined and conveyed to the IPoIB stack. For instance,
   implementations may resort to a manual configuration when choosing
   the P_key or a set of P_Keys for IPoIB, and rely on DHCP [DHCP] to
   assign an IP subnet number to each IPoIB link.

   Once an IB partition is established for IPoIB use, the link MTU and
   Q_Key are two other attributes that must be chosen before an IPoIB
   link can be configured.

6.1 Maximum Transmission Unit

   IB defines five permissible maximum payload sizes (MTUs). They are
   256, 512, 1024, 2048 and 4096 bytes. [IPV6] requires a link MTU of
   1280 bytes or greater. To be better compatible with Ethernet, the
   dominant network media in both the LAN and WAN environment, the IPoIB
   link MTU should be 1500 bytes or greater. This leaves only 2048 and
   4096 bytes as the two acceptable MTUs for IPoIB. Channel adaptors
   supporting a MTU less than the minimal requirement can still expose
   an acceptable MTU to IP through an adaptation layer that fragments
   larger messages into smaller IB packets, and reassembles them on the
   receiving end. But this must be done in a way that is transparent to
   the IP stack.

   It is up to the network administrator to select a link MTU to use
   when configuring an IPoIB link. The link MTU SHALL not be greater
   than the MTU of any IB devices on the IPoIB link. Here the IB devices
   include IB switches, CAs, or routers.




Chu & Kashyap                                                   [Page 5]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   In general, a maximum link MTU should be employed whenever possible
   to attain a better throughput performance. One caveat is that once a
   link MTU is chosen for a given IPoIB link, nodes connected by CAs of
   a smaller MTU won't be able to join the link unless the whole link
   and all the devices attached to it are reconfigured to use the
   smaller MTU.

   It may be desirable in some case to use a smaller link MTU than the
   full size. For example, bridging an IPoIB link with an Ethernet link
   could be made much easier if the IPoIB link MTU is reduced to 1500
   bytes. For IPv4, this may require a manual configuration of a
   different link MTU than the maximum that all the nodes support.  For
   IPv6, one can use the MTU option of the router advertisement [DISC]
   to announce a smaller MTU to all the nodes.

   In case an IPoIB link spans more than one IB subnet, the IPoIB link
   MTU MUST not exceed the path MTU of any path connecting two nodes in
   the same IB partition. It is up to the network administrator to
   determine the appropriate path MTU value that will work for any node
   in the same IPoIB link.

6.2 IPoIB Link Q_Key

   A Q_Key is programmed by the source QP in every IB datagram, and is
   compared against the Q_Key of the destination QP.  A Q_Key violation
   will cause the offending datagram to be dropped, and a Q_Key
   violation counter to be incremented on the receiving port. A trap is
   also generated if the feature is supported on that port.

   A single Q_Key must be selected for all the QPs attached to an IPoIB
   link to use. It is recommended that a controlled Q_Key be used with
   the high order bit set. This is to prevent non-privileged software
   from fabricating and sending out bogus IP datagrams. All QPs
   configured for a given IPoIB link SHALL be assigned the same per-link
   Q_Key.

6.3 Other Link Attributes

   TClass, FlowLabel, HopLimit, and SL are four other attributes that
   are required if an IPoIB link covers more than a single IB subnet.
   The selection of these values are implementation dependent.
   Implementations must take into account the topology of IB subnets
   comprising the IPoIB link to ensure a successful communication
   between any two nodes in the same IPoIB link.

7.0 The IPoIB Broadcast Group

   Once an IB partition is created with link attributes identified for



Chu & Kashyap                                                   [Page 6]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   an IPoIB link, the network administrator must create a special IB
   all-node multicast group (henceforth referred to as the broadcast
   group) with these link attributes for every node on the IPoIB link to
   join.  The creation of an IB multicast group is through the use of
   the "MCMemberRecord" SA attribute as described in the IBA
   specification.

   The MGID of an IPoIB broadcast group will embed in it the P_Key of
   the IB partition that defines the IPoIB link. A special signature is
   also embedded to identify all the MGIDs for IPoIB use only. For IPv4
   over IB, the signature will be "0x401B". For IPv6 over IB, the
   signature will be "0x601B".

   For an IPv4 subnet, the MGID for this special IB multicast group
   SHALL have the following format:

   |   8    |  4 |  4 |     16 bits    | 16 bits | 48 bits  | 32 bits |
   +--------+----+----+----------------+---------+----------+---------+
   |11111111|0001|scop|0100000000011011|< P_Key >|00.......0|<all 1's>|
   +--------+----+----+----------------+---------+----------+---------+

                                 Figure 2


   For an IPv6 subnet, the format of the MGID SHALL look like this:

   |   8    |  4 |  4 |     16 bits    | 16 bits |       80 bits      |
   +--------+----+----+----------------+---------+--------------------+
   |11111111|0001|scop|0110000000011011|< P_Key >|000.............0001|
   +--------+----+----+----------------+---------+--------------------+

                                 Figure 3

   As for the scop bits, if the IPoIB link is fully contained within a
   single IB subnet, the scop bits SHALL be set to 2 (link-local).
   Otherwise the scope will be set higher.

   The broadcast group for IPv4 will serve to provide a broadcast
   service for protocols like ARP to use.

   When a node is first brought up on an IPoIB link identified by a
   P_Key, it must look for the right broadcast group to join. This is
   done by querying the SA MCMemberRecord database for a multicast group
   with a MGID matching the one constructed from the link P_Key and the
   IPoIB signature. The node SHOULD always look for a MGID of a link-
   local scope first before attempting one with a greater scope.

   Once the right MGID and broadcast group are identified, the local



Chu & Kashyap                                                   [Page 7]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   node SHOULD use the MTU associated with the broadcast group.  In case
   the MTU of the broadcast group is greater than what the local HCA can
   support, the node can not join the IPoIB link and operate as an IP
   node. Otherwise the local node must join the broadcast group as a
   "full member" and use the rest of link attributes associated with the
   group for all future communication to the link.

   In addition to the special all-node multicast group for broadcast
   purpose, an all-router multicast group SHOULD be created at link
   configuration time if an IP router will be attached to the link. This
   is to facilitate IP multicast operations described later. An IB
   multicast group for the all-router MGID must cover every IB subnet
   that the IPoIB link encompasses.  The format of the all-router MGID
   will be covered in the next section.

8.0 Mapping for other Multicast Groups

   The general IP multicast [IPMULT] support over IB is similar to the
   case of the special broadcast group discussed above. An algorithmic
   mapping is used so that given an IP multicast address, individual
   host can compute the corresponding IB multicast address (MGID) all by
   itself without having to consult an external entity. This also
   removes the need for an externally maintained IP to IB multicast
   mapping table.

   The IPoIB multicast mapping is depicted in Figure 4. The same mapping
   function is used for both IPv4 and IPv6 except the IPoIB signature
   field.

   |   8    |  4 |  4 |     16 bits     | 16 bits |      80 bits       |
   +------ -+----+----+-----------------+---------+--------------------+
   |11111111|0001|scop|<IPoIB signature>|< P_Key >|      group ID      |
   +--------+----+----+-----------------+---------+--------------------+

                                 Figure 4

   Since a MGID allocated for transporting IP multicast datagrams is
   considered only a transient link-layer multicast address, all IB
   MGIDs allocated for IPoIB purpose SHOULD have T = 1. The scope bits
   SHALL be the same as that of the all-node MGID for the same IPoIB
   link.

   An IP multicast address is used together with a given IPoIB link
   P_Key to form the MGID of the IB multicast group. For IPv6 the lower
   80-bit of the group ID is used directly in the lower 80-bit of the
   MGID. For IPv4, the group ID is only 28-bit long and the rest of the
   80 bits are filled with 0.




Chu & Kashyap                                                   [Page 8]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   The rest of the bits are the same as those of the broadcast MGID.
   For example, on an IPoIB link that is fully contained within a single
   IB subnet with a P_Key of 0x8006, the MGIDs for the all-router
   multicast group with group ID 2 [AARCH, IGMP2] are:

   FF12:401B:8006:0:0:0:0:2

   or

   FF12:401B:8006::2

   for IPv4 in a compressed format, and

   FF12:601B:8006:0:0:0:0:2

   or

   FF12:601B:8006::2

   for IPv6 in a compressed format.

   A special case exists for the IPv4 limited broadcast address
   "255.255.255.255" [HOSTS]. The address SHALL be mapped to the
   broadcast MGID for IPv4 networks as described in section 7 above.
   Also the IPv6 all-node multicast address "FF0X::1" [AARCH] maps
   naturally to the the special broadcast MGID for IPv6 networks.

9.0 Sending and Receiving IP Multicast Packets

   Multicast in InfiniBand differs in a number of ways from multicast in
   Ethernet. This adds some complexity to an IPoIB implementation when
   supporting IP multicast over IB.

   A) An IB multicast group must be explicitly created through the SA
   before it can be used.

   This implies that in order to send a packet destined for an IP
   multicast address, the IPoIB implementation must check with the SA on
   the outbound link first for a "MCMemberRecord" that matches the MGID.
   If one does exist, the MLID associated with the multicast group is
   used as the DLID for the packet. Otherwise, it implies no member
   exists on the local link.  The packet SHOULD be forwarded to locally
   connected routers. This is to allow local routers to forward the
   packet to multicast listeners on remote networks. The specific
   mechanism for a sender to forward packets to routers are left to
   implementations. One can use, for example, the broadcast group, or
   the all-router multicast group for this purpose.




Chu & Kashyap                                                   [Page 9]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   B) A multicast sender must join the target multicast group as a
   "SendOnlyNonMember" before outgoing multicast messages from it can be
   successfully routed. The "SendOnlyNonMember" join is different from
   the regular "FullMember" join in two aspects. First, both types of
   joins enable multicast packets to be routed FROM the local port, but
   only the "FullMember" join causes multicast packets to be routed TO
   the port.  Second, the sender port of a "SendOnlyNonMember" join will
   not be counted as a member of the multicast group for purposes of
   group creation and deletion.

   The following code snippet demonstrates the steps in a typical
   implementation when processing an egress multicast packet.

   if the egress port is already a "SendOnlyNonMember", or a
   "FullMember"
           => send the packet

   else if the target multicast group exists
           => do "SendOnlyNonMember" join
           => send the packet

   else if the all-router multicast group exists
           => send the packet to all routers
   else
           => drop the packet

   Implementations should cache the information about the existence of
   an IB multicast group, its MLID and other attributes. This is to
   avoid expensive SA calls on every outgoing multicast packet. The
   cache may need to be validated periodically. Senders should also
   subscribe to the multicast group create and delete traps in order to
   monitor the status of specific IB multicast groups. Multicast packets
   directed to the all-router multicast group due to a lack of listener
   on the local subnet must be forwarded to the right multicast group if
   the group is created later.  This happens when a listener shows up on
   the local subnet.

   A node joining an IP multicast group must first construct a MGID
   according to the rule described in section 8 above. Once the correct
   MGID is calculated, the node must call the SA of the outbound link to
   attempt a "FullMember" join of the IB multicast group corresponding
   to the MGID.  If the IB multicast group doesn't already exist, one
   must be created first with the IPoIB link MTU. For the rest of
   attributes, it is recommended the same values from the all-node
   multicast/broadcast group be used.

   The join request will cause the local port to be added to the
   multicast group. It also enables the SM to program IB switches and



Chu & Kashyap                                                  [Page 10]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   routers with the new multicast information to ensure the correct
   forwarding of multicast packets for the group.

   When a node leaves an IP multicast group, it SHOULD make a
   "FullMember" leave request to the SA. This gives SM an opportunity to
   update relevant forwarding information, to delete an IB multicast
   group if the local port is the last FullMember to leave, and free up
   the MLID allocated for it. The specific algorithm is implementation-
   dependent, and is out of the scope of this document.

   Note that for an IPoIB link that spans more than one IB subnet
   connected by IB routers, an adequate multicast forwarding support at
   the IB level is required for multicast packets to reach listeners on
   a remote IB subnet. The specific mechanism for this will be covered
   in [IBTA], and is beyond the scope of IPoIB.

10.0 IP Multicast Routing

   IP multicast routing requires multicast routers to receive a copy of
   every link multicast packet on a locally connected link [IPMULT,
   IP6MLD].  For Ethernet this is usually achieved by turning on the
   promiscuous multicast mode on a locally connected Ethernet interface.

   IBA does not provide any hardware support for promiscuous multicast
   mode.  Fortunately a promiscuous multicast mode can be emulated in
   the software running on a router through the following steps.

   A) Obtain a list of all active IB multicast groups from the local SA.

   B) Make a "NonMember" join request to the SA for every group that has
   a signature in its MGID matching the one for either IPv4 or IPv6.

   C) Subscribe to the IB multicast group creation events using a
   wildcarded MGID so that the router can "NonMember" join all IB
   multicast groups created subsequently for IPv4 or IPv6.

   The "NonMember" join has the same effect as a "FullMember" join
   except that the former will not be counted as a member of the
   multicast group for purposes of group creation or deletion. That is,
   when the last "FullMember" leaves a multicast group, the group can be
   safely deleted by the SA without concerning any "NonMember" routers.

11.0 Security Considerations

   All the operations for creating and configuring an IPoIB link
   described in this document, including assigning P_Keys to CAs,
   creating IB multicast groups in SA, creating and attaching QPs to IB
   multicast groups,... etc, are privileged operations, and MUST be



Chu & Kashyap                                                  [Page 11]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   protected by the underlying operating system. This is to prevent
   malicious, non- privileged software from hijacking important
   resources and configurations.  For example, A bogus IPoIB broadcast
   group may prevent a proper one from being created when the network
   administrator tries to set up a link.

   Controlled Q_Keys SHOULD be used in IPoIB links. This is to prevent
   non-privileged software from fabricating IP datagrams to send, as
   mentioned in section 6.2.

12.0 Acknowledgments

   The authors would like to thank Bruce Beukema, David Brean, Dan
   Cassiday, Aditya Dube, Yaron Haviv, Michael Krause, Thomas Narten,
   Erik Nordmark, Greg Pfister, Renato Recio, Satya Sharma, and David L.
   Stevens for their suggestions and many clarifications on the IBA
   specification.

13.0 References

   [AARCH]   Hinden, R. and S. Deering "IP Version 6 Addressing
             Architecture", RFC 2373, July 1998.

   [DHCP]    R. Droms "Dynamic Host Configuration Protocol", RFC 2131,
             March 1997.

   [DISC]    Narten, T., Nordmark, E. and W. Simpson, "Neighbor
             Discovery for IP Version 6 (IPv6)", RFC 2461, December
             1998.

   [HOSTS]   Braden R., "Requirements for Internet Hosts --
             Communication Layers", RFC 1122, October 1989

   [IBTA]    InfiniBand Architecture Specification, Release 1.0.a by
             InfiniBand Trade Association at www.infinibandta.org

   [IGMP2]   Fenner W., "Internet Group Management Protocol, Version 2",
             RFC 2236, November 1997.

   [IPMULT]  Deering S., "Host Extensions for IP Multicasting", RFC
             1112, August 1989.

   [IPoIB_ARCH]  draft-ietf-ipoib-architecture-01.txt

   [IPoIB_ENCAP] draft-ietf-ipoib-ip-over-infiniband-01.txt

   [IPV6]    Deering, S. and R. Hinden, "Internet Protocol, Version 6
             (IPv6) Specification", RFC 2460, December 1998.



Chu & Kashyap                                                  [Page 12]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   [IP6MLD]  Deering S., Fenner W., Haberman B., "Multicast Listener
             Discovery (MLD) for IPv6", RFC 2710, October 1999.

   [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
             Requirement Levels", BCP 14, RFC 2119, March 1997.


14.0 Author's Address

   H.K. Jerry Chu
   17 Network Circle, UMPK17-201
   Menlo Park, CA 94025
   USA

   Phone: +1 650 786-5146
   EMail: jerry.chu@sun.com


   Vivek Kashyap
   IBM
   15450, SW Koll Parkway
   Beaverton, OR 97006

   Phone: 503 578 3422
   EMail: vivk@us.ibm.com



15.0 Full Copyright Statement

   Copyright (C) The Internet Society (2003>.  All Rights Reserved.

   This document and translations of it may be copied and furnished to
   others, and derivative works that comment on or otherwise explain it
   or assist in its implementation may be prepared, copied, published
   and distributed, in whole or in part, without restriction of any
   kind, provided that the above copyright notice and this paragraph are
   included on all such copies and derivative works.  However, this
   document itself may not be modified in any way, such as by removing
   the copyright notice or references to the Internet Society or other
   Internet organizations, except as needed for the purpose of
   developing Internet standards in which case the procedures for
   copyrights defined in the Internet Standards process must be
   followed, or as required to translate it into languages other than
   English.

   The limited permissions granted above are perpetual and will not be
   revoked by the Internet Society or its successors or assigns.



Chu & Kashyap                                                  [Page 13]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   This document and the information contained herein is provided on an
   "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
   TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
   BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
   HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
   MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.













































Chu & Kashyap                                                  [Page 14]


--Grist_of_Bees_910_000--
_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


From mailnull@www1.ietf.org  Mon Mar  3 19:21:14 2003
Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA19155
	for <ipoverib-archive@odin.ietf.org>; Mon, 3 Mar 2003 19:21:14 -0500 (EST)
Received: (from mailnull@localhost)
	by www1.ietf.org (8.11.6/8.11.6) id h240VQk07564
	for ipoverib-archive@odin.ietf.org; Mon, 3 Mar 2003 19:31:26 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h240VQp07561
	for <ipoverib-web-archive@optimus.ietf.org>; Mon, 3 Mar 2003 19:31:26 -0500
Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA19123
	for <ipoverib-web-archive@ietf.org>; Mon, 3 Mar 2003 19:20:39 -0500 (EST)
Received: from www1.ietf.org (localhost.localdomain [127.0.0.1])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h240UQp07518;
	Mon, 3 Mar 2003 19:30:26 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h240Tnp07485
	for <ipoverib@optimus.ietf.org>; Mon, 3 Mar 2003 19:29:49 -0500
Received: from pheriche.sun.com (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA19099
	for <ipoverib@ietf.org>; Mon, 3 Mar 2003 19:19:03 -0500 (EST)
Received: from jurassic.eng.sun.com ([129.146.17.55])
	by pheriche.sun.com (8.9.3+Sun/8.9.3) with ESMTP id RAA14054
	for <ipoverib@ietf.org>; Mon, 3 Mar 2003 17:21:04 -0700 (MST)
Received: from taipei (taipei.Eng.Sun.COM [129.146.85.178])
	by jurassic.eng.sun.com (8.12.8+Sun/8.12.8) with SMTP id h240L3im307684
	for <ipoverib@ietf.org>; Mon, 3 Mar 2003 16:21:03 -0800 (PST)
Message-Id: <200303040021.h240L3im307684@jurassic.eng.sun.com>
Date: Mon, 3 Mar 2003 16:21:01 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
To: ipoverib@ietf.org
MIME-Version: 1.0
Content-Type: MULTIPART/mixed; BOUNDARY=Grist_of_Bees_910_000
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.6_06 SunOS 5.8 sun4u sparc 
Subject: [Ipoverib] draft-ietf-ipoib-link-multicast-03.txt
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-BeenThere: ipoverib@ietf.org
X-Mailman-Version: 2.0.12
Precedence: bulk
List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=unsubscribe>
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
List-Post: <mailto:ipoverib@ietf.org>
List-Help: <mailto:ipoverib-request@ietf.org?subject=help>
List-Subscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=subscribe>

--Grist_of_Bees_910_000
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: KG6RlEkcowMVQpsCzsLESA==

The link and multicast draft has been revised according to the IBA
1.1 specification. Most of the changes are in section 9 and a
restored section 10 on IP multicast routing. The draft revison
also incorporates various comments from the past, and a number of
minor corrections throughput the draft.

The draft was submitted to IETF last week, and is currently waiting
to be published. A copy of the draft is enclosed below. Please
review and comment so that the draft can go back to the IESG review
queue asap.

Thanks,

Jerry


--Grist_of_Bees_910_000
Content-Type: TEXT/plain; name="draft-ietf-ipoib-link-multicast-03.txt"; charset=us-ascii; x-unix-mode=0664
Content-Description: draft-ietf-ipoib-link-multicast-03.txt
Content-MD5: D2fWDLKs2m53Jkfe78hm5A==







INTERNET-DRAFT                                            H.K. Jerry Chu
<draft-ietf-ipoib-link-multicast-03.txt>                Sun Microsystems
                                                           Vivek Kashyap
                                                                     IBM
Expires: September, 2003                                     March, 2003


             IP link and multicast over InfiniBand networks


Status of this Memo

   This document is an Internet-Draft and is in full conformance with
   all provisions of Section 10 of RFC2026.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups. Note that other
   groups may also distribute working documents as Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time. It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

   Copyright (C) The Internet Society (2003).  All Rights Reserved.


Abstract

   This document specifies a method for setting up IP subnets and
   multicast services over InfiniBand(TM) networks. Discussions in this
   document are applicable to both IPv4 and IPv6, unless explicitly
   specified. A separate document will cover unicast and encapsulation
   of IP datagrams over InfiniBand networks.


Table of Contents
   1.0     Introduction
   2.0     Terminology
   3.0     Basic IPoIB Transport - Unreliable Datagram
   4.0     IB Multicast Architecture
   5.0     IB Links vs. IPoIB Links



Chu & Kashyap                                                   [Page 1]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   6.0     Setting up an IPoIB Link
   6.1     Maximum Transmission Unit
   6.2     IPoIB Link Q_Key
   6.3     Other Link Attributes
   7.0     The IPoIB Broadcast Group
   8.0     Mapping for other Multicast Groups
   9.0     Sending and Receiving IP Multicast Packets
   10.0    IP Multicast Routing
   11.0    Security Considerations
   12.0    Acknowledgments
   13.0    References
   14.0    Author's Address
   15.0    Full Copyright Statement


1.0 Introduction

   InfiniBand Architecture (IBA) defines four layers of network services
   corresponding to layer one through layer four of the OSI reference
   model.  For the purpose of running IP over an InfiniBand (IB)
   network, the IB link, network, and transport layers collectively
   constitute the data link layer to the IP stack. One can find a
   general overview of IB architecture related to IP networks in
   [IPoIB_ARCH].

   This document will focus on the necessary steps in order to lay out
   an IP network on top of an IB network. It will describe all the
   elements of an IP over InfiniBand (IPoIB) link, how to configure its
   associated attributes, and how to set up basic broadcast and
   multicast services for it. IPoIB links are the building blocks upon
   which an IP network consisting of many IP subnets connected by
   routers can be built.  Subnetting allows the containment of broadcast
   traffic within a single link. It also provides certain degree of
   isolation for the administration purpose between nodes on different
   subnets.

2.0 Terminology

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119].

3.0 Basic IPoIB Transport - Unreliable Datagram

   InfiniBand defines four types of transport services [IBTA]. They are
   reliable connection, unreliable connection, reliable datagram,
   unreliable datagram. IBA also defines a special raw datagram service
   for encapsulation purpose. Both unreliable datagram and raw datagram



Chu & Kashyap                                                   [Page 2]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   define support for multicast. They provide the basic transport
   mechanism that best matches the IP datagram paradigm.

   IB unreliable datagram provides many additional features such as the
   partition key (P_Key) protection, multiple queue pairs (QPs), and
   Q_Key protection. Moreover, it defines a 32-bit invariant CRC
   checksum, which provides a much stronger protection against data
   corruption, compared with the 16-bit CRC that a raw datagram carries.

   For these reasons, IB unreliable datagram is considered to be a much
   better choice as the basic IPoIB transport than the raw datagram, and
   is chosen as the default IPoIB transport mechanism ([IPoIB_ARCH],
   [IPoIB_ENCAP]).

4.0 IB Multicast Architecture

   The following discussion gives a short overview of the multicast
   architecture in InfiniBand. For a complete specification, the reader
   is referred to [IBTA].

   IBA defines two layers of multicast services. Its link layer uses
   multicast LIDs (MLIDs) in the Local Route Header (LRH). LIDs are
   allocated by the Subnet Manager (SM) and fall in the range between
   0xC0000 to 0xFFFE (approximately 16k). MLIDs are used by IB switches
   to program their multicast forwarding tables. An IB switch
   implementation may support much fewer MLIDs in its forwarding table
   though.

   The IB network layer uses multicast GIDs (MGIDs) in the Global Route
   Header (GRH). MGIDs closely resemble IPv6 multicast addresses [AARCH]
   shown below.

   |   8    |  4 |  4 |                  112 bits                   |
   +------ -+----+----+---------------------------------------------+
   |11111111|flgs|scop|                  group ID                   |
   +--------+----+----+---------------------------------------------+

                                 Figure 1

   [IPoIB_ARCH] describes each field in more details.

   Since every IB multicast packet is required to carry a LRH and a GRH,
   both a valid MGID and a valid MLID are needed before an IB multicast
   packet can be constructed.

   An IB multicast group is uniquely identified by a valid MGID. Before
   a MGID can be used within an IB subnet, either as a destination
   address of a multicast packet, or to represent a multicast group that



Chu & Kashyap                                                   [Page 3]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   an IB node can join, an IB multicast group corresponding to the MGID
   must be created through the Subnet Administrator (SA). Besides the
   the MGID, the creator of an IB multicast group must supply values of
   path MTU, P_Key, Q_Key, Service Level (SL), FlowLabel, TClass that
   are appropriate for all the potential clients of the multicast group
   to use. In return, SA will allocate a MLID to be used by switches in
   the local IB subnet.

   Unreliable multicast is defined by IBA as an optional functionality
   for channel adaptors (CAs) and switches. In today's IP technology,
   link multicast has become an indispensable function for better
   supporting a modern IP network. For this reason, it is required that
   an IPoIB fabric supports multicast. This includes all the CAs and
   switches that are part of an IP network.

5.0 IB Links vs. IPoIB Links

   A link segment on top of which an IP subnet can be configured is
   defined in [IPV6] as a communication facility or medium over which
   nodes can communicate at the "link" layer.  For most types of
   communication media, the boundary between different data link
   segments closely follows the physical topology of the network. For
   instance, an Ethernet network connected by switches, hubs, or bridges
   usually forms a single link segment and broadcast/multicast domain.
   Different Ethernet segments can be connected by IP routers at the
   network layer to form an IP network.

   InfiniBand defines its own link-layer and subnets consisting of nodes
   connected by IB switches and routers. However, the IPoIB link
   boundary need not follow the IB link boundary. Nodes residing on
   different IB subnets can still communicate directly with one another
   through IB routers at the InfiniBand network layer. This
   communication at the network layer applies to unicast as well as
   multicast.

   The ultimate requirement for two nodes in the same IB fabric to
   communicate at the IB level, besides physical connectivity, is a
   common P_Key.

   Partitioning in IB provides an isolation mechanism among nodes in an
   IB fabric, much like VLANs in the Ethernet network.  Each port of an
   HCA (Host Channel Adaptor) contains a P_Key table holding all the
   valid P_Keys the port is allowed to use. The P_Key table is set up by
   the SM of the local IB subnet. Each QP is programmed with a P_Key
   from the local P_Key table. This P_Key is carried in all the outgoing
   packets from the QP, and is used to compare against the P_Key of all
   incoming packets to the QP. Any packet with an invalid P_Key will be
   discarded by the QP and a P_Key violation trap will be generated.  IB



Chu & Kashyap                                                   [Page 4]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   switches may optionally enforce partition checking too.

   Following the above, IB partitions are the natural choice for
   defining IPoIB link boundary. It also provides much needed
   flexibility for a network administrator to group nodes logically into
   different subnets in a large network.

6.0 Setting up an IPoIB Link

   A network administrator defines an IPoIB link by setting up an IB
   partition and assigning it a unique P_Key. Since a full-duplex
   communication is required among IP nodes, full-membership P_Keys,
   that is, those with the high-order bit set to 1 shall be used. An IB
   partition may or may not span multiple IB subnets; and whether it
   does or not is mostly transparent to IPoIB.

   Each node attached to an IB partition MUST have one of its HCAs
   assigned the P_Key to use. Note that the P_key table of an HCA port
   may contain many P_Keys. It is up to the implementation to define the
   method by which the P_Key relevant to a particular IPoIB subnet is
   determined and conveyed to the IPoIB stack. For instance,
   implementations may resort to a manual configuration when choosing
   the P_key or a set of P_Keys for IPoIB, and rely on DHCP [DHCP] to
   assign an IP subnet number to each IPoIB link.

   Once an IB partition is established for IPoIB use, the link MTU and
   Q_Key are two other attributes that must be chosen before an IPoIB
   link can be configured.

6.1 Maximum Transmission Unit

   IB defines five permissible maximum payload sizes (MTUs). They are
   256, 512, 1024, 2048 and 4096 bytes. [IPV6] requires a link MTU of
   1280 bytes or greater. To be better compatible with Ethernet, the
   dominant network media in both the LAN and WAN environment, the IPoIB
   link MTU should be 1500 bytes or greater. This leaves only 2048 and
   4096 bytes as the two acceptable MTUs for IPoIB. Channel adaptors
   supporting a MTU less than the minimal requirement can still expose
   an acceptable MTU to IP through an adaptation layer that fragments
   larger messages into smaller IB packets, and reassembles them on the
   receiving end. But this must be done in a way that is transparent to
   the IP stack.

   It is up to the network administrator to select a link MTU to use
   when configuring an IPoIB link. The link MTU SHALL not be greater
   than the MTU of any IB devices on the IPoIB link. Here the IB devices
   include IB switches, CAs, or routers.




Chu & Kashyap                                                   [Page 5]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   In general, a maximum link MTU should be employed whenever possible
   to attain a better throughput performance. One caveat is that once a
   link MTU is chosen for a given IPoIB link, nodes connected by CAs of
   a smaller MTU won't be able to join the link unless the whole link
   and all the devices attached to it are reconfigured to use the
   smaller MTU.

   It may be desirable in some case to use a smaller link MTU than the
   full size. For example, bridging an IPoIB link with an Ethernet link
   could be made much easier if the IPoIB link MTU is reduced to 1500
   bytes. For IPv4, this may require a manual configuration of a
   different link MTU than the maximum that all the nodes support.  For
   IPv6, one can use the MTU option of the router advertisement [DISC]
   to announce a smaller MTU to all the nodes.

   In case an IPoIB link spans more than one IB subnet, the IPoIB link
   MTU MUST not exceed the path MTU of any path connecting two nodes in
   the same IB partition. It is up to the network administrator to
   determine the appropriate path MTU value that will work for any node
   in the same IPoIB link.

6.2 IPoIB Link Q_Key

   A Q_Key is programmed by the source QP in every IB datagram, and is
   compared against the Q_Key of the destination QP.  A Q_Key violation
   will cause the offending datagram to be dropped, and a Q_Key
   violation counter to be incremented on the receiving port. A trap is
   also generated if the feature is supported on that port.

   A single Q_Key must be selected for all the QPs attached to an IPoIB
   link to use. It is recommended that a controlled Q_Key be used with
   the high order bit set. This is to prevent non-privileged software
   from fabricating and sending out bogus IP datagrams. All QPs
   configured for a given IPoIB link SHALL be assigned the same per-link
   Q_Key.

6.3 Other Link Attributes

   TClass, FlowLabel, HopLimit, and SL are four other attributes that
   are required if an IPoIB link covers more than a single IB subnet.
   The selection of these values are implementation dependent.
   Implementations must take into account the topology of IB subnets
   comprising the IPoIB link to ensure a successful communication
   between any two nodes in the same IPoIB link.

7.0 The IPoIB Broadcast Group

   Once an IB partition is created with link attributes identified for



Chu & Kashyap                                                   [Page 6]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   an IPoIB link, the network administrator must create a special IB
   all-node multicast group (henceforth referred to as the broadcast
   group) with these link attributes for every node on the IPoIB link to
   join.  The creation of an IB multicast group is through the use of
   the "MCMemberRecord" SA attribute as described in the IBA
   specification.

   The MGID of an IPoIB broadcast group will embed in it the P_Key of
   the IB partition that defines the IPoIB link. A special signature is
   also embedded to identify all the MGIDs for IPoIB use only. For IPv4
   over IB, the signature will be "0x401B". For IPv6 over IB, the
   signature will be "0x601B".

   For an IPv4 subnet, the MGID for this special IB multicast group
   SHALL have the following format:

   |   8    |  4 |  4 |     16 bits    | 16 bits | 48 bits  | 32 bits |
   +--------+----+----+----------------+---------+----------+---------+
   |11111111|0001|scop|0100000000011011|< P_Key >|00.......0|<all 1's>|
   +--------+----+----+----------------+---------+----------+---------+

                                 Figure 2


   For an IPv6 subnet, the format of the MGID SHALL look like this:

   |   8    |  4 |  4 |     16 bits    | 16 bits |       80 bits      |
   +--------+----+----+----------------+---------+--------------------+
   |11111111|0001|scop|0110000000011011|< P_Key >|000.............0001|
   +--------+----+----+----------------+---------+--------------------+

                                 Figure 3

   As for the scop bits, if the IPoIB link is fully contained within a
   single IB subnet, the scop bits SHALL be set to 2 (link-local).
   Otherwise the scope will be set higher.

   The broadcast group for IPv4 will serve to provide a broadcast
   service for protocols like ARP to use.

   When a node is first brought up on an IPoIB link identified by a
   P_Key, it must look for the right broadcast group to join. This is
   done by querying the SA MCMemberRecord database for a multicast group
   with a MGID matching the one constructed from the link P_Key and the
   IPoIB signature. The node SHOULD always look for a MGID of a link-
   local scope first before attempting one with a greater scope.

   Once the right MGID and broadcast group are identified, the local



Chu & Kashyap                                                   [Page 7]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   node SHOULD use the MTU associated with the broadcast group.  In case
   the MTU of the broadcast group is greater than what the local HCA can
   support, the node can not join the IPoIB link and operate as an IP
   node. Otherwise the local node must join the broadcast group as a
   "full member" and use the rest of link attributes associated with the
   group for all future communication to the link.

   In addition to the special all-node multicast group for broadcast
   purpose, an all-router multicast group SHOULD be created at link
   configuration time if an IP router will be attached to the link. This
   is to facilitate IP multicast operations described later. An IB
   multicast group for the all-router MGID must cover every IB subnet
   that the IPoIB link encompasses.  The format of the all-router MGID
   will be covered in the next section.

8.0 Mapping for other Multicast Groups

   The general IP multicast [IPMULT] support over IB is similar to the
   case of the special broadcast group discussed above. An algorithmic
   mapping is used so that given an IP multicast address, individual
   host can compute the corresponding IB multicast address (MGID) all by
   itself without having to consult an external entity. This also
   removes the need for an externally maintained IP to IB multicast
   mapping table.

   The IPoIB multicast mapping is depicted in Figure 4. The same mapping
   function is used for both IPv4 and IPv6 except the IPoIB signature
   field.

   |   8    |  4 |  4 |     16 bits     | 16 bits |      80 bits       |
   +------ -+----+----+-----------------+---------+--------------------+
   |11111111|0001|scop|<IPoIB signature>|< P_Key >|      group ID      |
   +--------+----+----+-----------------+---------+--------------------+

                                 Figure 4

   Since a MGID allocated for transporting IP multicast datagrams is
   considered only a transient link-layer multicast address, all IB
   MGIDs allocated for IPoIB purpose SHOULD have T = 1. The scope bits
   SHALL be the same as that of the all-node MGID for the same IPoIB
   link.

   An IP multicast address is used together with a given IPoIB link
   P_Key to form the MGID of the IB multicast group. For IPv6 the lower
   80-bit of the group ID is used directly in the lower 80-bit of the
   MGID. For IPv4, the group ID is only 28-bit long and the rest of the
   80 bits are filled with 0.




Chu & Kashyap                                                   [Page 8]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   The rest of the bits are the same as those of the broadcast MGID.
   For example, on an IPoIB link that is fully contained within a single
   IB subnet with a P_Key of 0x8006, the MGIDs for the all-router
   multicast group with group ID 2 [AARCH, IGMP2] are:

   FF12:401B:8006:0:0:0:0:2

   or

   FF12:401B:8006::2

   for IPv4 in a compressed format, and

   FF12:601B:8006:0:0:0:0:2

   or

   FF12:601B:8006::2

   for IPv6 in a compressed format.

   A special case exists for the IPv4 limited broadcast address
   "255.255.255.255" [HOSTS]. The address SHALL be mapped to the
   broadcast MGID for IPv4 networks as described in section 7 above.
   Also the IPv6 all-node multicast address "FF0X::1" [AARCH] maps
   naturally to the the special broadcast MGID for IPv6 networks.

9.0 Sending and Receiving IP Multicast Packets

   Multicast in InfiniBand differs in a number of ways from multicast in
   Ethernet. This adds some complexity to an IPoIB implementation when
   supporting IP multicast over IB.

   A) An IB multicast group must be explicitly created through the SA
   before it can be used.

   This implies that in order to send a packet destined for an IP
   multicast address, the IPoIB implementation must check with the SA on
   the outbound link first for a "MCMemberRecord" that matches the MGID.
   If one does exist, the MLID associated with the multicast group is
   used as the DLID for the packet. Otherwise, it implies no member
   exists on the local link.  The packet SHOULD be forwarded to locally
   connected routers. This is to allow local routers to forward the
   packet to multicast listeners on remote networks. The specific
   mechanism for a sender to forward packets to routers are left to
   implementations. One can use, for example, the broadcast group, or
   the all-router multicast group for this purpose.




Chu & Kashyap                                                   [Page 9]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   B) A multicast sender must join the target multicast group as a
   "SendOnlyNonMember" before outgoing multicast messages from it can be
   successfully routed. The "SendOnlyNonMember" join is different from
   the regular "FullMember" join in two aspects. First, both types of
   joins enable multicast packets to be routed FROM the local port, but
   only the "FullMember" join causes multicast packets to be routed TO
   the port.  Second, the sender port of a "SendOnlyNonMember" join will
   not be counted as a member of the multicast group for purposes of
   group creation and deletion.

   The following code snippet demonstrates the steps in a typical
   implementation when processing an egress multicast packet.

   if the egress port is already a "SendOnlyNonMember", or a
   "FullMember"
           => send the packet

   else if the target multicast group exists
           => do "SendOnlyNonMember" join
           => send the packet

   else if the all-router multicast group exists
           => send the packet to all routers
   else
           => drop the packet

   Implementations should cache the information about the existence of
   an IB multicast group, its MLID and other attributes. This is to
   avoid expensive SA calls on every outgoing multicast packet. The
   cache may need to be validated periodically. Senders should also
   subscribe to the multicast group create and delete traps in order to
   monitor the status of specific IB multicast groups. Multicast packets
   directed to the all-router multicast group due to a lack of listener
   on the local subnet must be forwarded to the right multicast group if
   the group is created later.  This happens when a listener shows up on
   the local subnet.

   A node joining an IP multicast group must first construct a MGID
   according to the rule described in section 8 above. Once the correct
   MGID is calculated, the node must call the SA of the outbound link to
   attempt a "FullMember" join of the IB multicast group corresponding
   to the MGID.  If the IB multicast group doesn't already exist, one
   must be created first with the IPoIB link MTU. For the rest of
   attributes, it is recommended the same values from the all-node
   multicast/broadcast group be used.

   The join request will cause the local port to be added to the
   multicast group. It also enables the SM to program IB switches and



Chu & Kashyap                                                  [Page 10]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   routers with the new multicast information to ensure the correct
   forwarding of multicast packets for the group.

   When a node leaves an IP multicast group, it SHOULD make a
   "FullMember" leave request to the SA. This gives SM an opportunity to
   update relevant forwarding information, to delete an IB multicast
   group if the local port is the last FullMember to leave, and free up
   the MLID allocated for it. The specific algorithm is implementation-
   dependent, and is out of the scope of this document.

   Note that for an IPoIB link that spans more than one IB subnet
   connected by IB routers, an adequate multicast forwarding support at
   the IB level is required for multicast packets to reach listeners on
   a remote IB subnet. The specific mechanism for this will be covered
   in [IBTA], and is beyond the scope of IPoIB.

10.0 IP Multicast Routing

   IP multicast routing requires multicast routers to receive a copy of
   every link multicast packet on a locally connected link [IPMULT,
   IP6MLD].  For Ethernet this is usually achieved by turning on the
   promiscuous multicast mode on a locally connected Ethernet interface.

   IBA does not provide any hardware support for promiscuous multicast
   mode.  Fortunately a promiscuous multicast mode can be emulated in
   the software running on a router through the following steps.

   A) Obtain a list of all active IB multicast groups from the local SA.

   B) Make a "NonMember" join request to the SA for every group that has
   a signature in its MGID matching the one for either IPv4 or IPv6.

   C) Subscribe to the IB multicast group creation events using a
   wildcarded MGID so that the router can "NonMember" join all IB
   multicast groups created subsequently for IPv4 or IPv6.

   The "NonMember" join has the same effect as a "FullMember" join
   except that the former will not be counted as a member of the
   multicast group for purposes of group creation or deletion. That is,
   when the last "FullMember" leaves a multicast group, the group can be
   safely deleted by the SA without concerning any "NonMember" routers.

11.0 Security Considerations

   All the operations for creating and configuring an IPoIB link
   described in this document, including assigning P_Keys to CAs,
   creating IB multicast groups in SA, creating and attaching QPs to IB
   multicast groups,... etc, are privileged operations, and MUST be



Chu & Kashyap                                                  [Page 11]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   protected by the underlying operating system. This is to prevent
   malicious, non- privileged software from hijacking important
   resources and configurations.  For example, A bogus IPoIB broadcast
   group may prevent a proper one from being created when the network
   administrator tries to set up a link.

   Controlled Q_Keys SHOULD be used in IPoIB links. This is to prevent
   non-privileged software from fabricating IP datagrams to send, as
   mentioned in section 6.2.

12.0 Acknowledgments

   The authors would like to thank Bruce Beukema, David Brean, Dan
   Cassiday, Aditya Dube, Yaron Haviv, Michael Krause, Thomas Narten,
   Erik Nordmark, Greg Pfister, Renato Recio, Satya Sharma, and David L.
   Stevens for their suggestions and many clarifications on the IBA
   specification.

13.0 References

   [AARCH]   Hinden, R. and S. Deering "IP Version 6 Addressing
             Architecture", RFC 2373, July 1998.

   [DHCP]    R. Droms "Dynamic Host Configuration Protocol", RFC 2131,
             March 1997.

   [DISC]    Narten, T., Nordmark, E. and W. Simpson, "Neighbor
             Discovery for IP Version 6 (IPv6)", RFC 2461, December
             1998.

   [HOSTS]   Braden R., "Requirements for Internet Hosts --
             Communication Layers", RFC 1122, October 1989

   [IBTA]    InfiniBand Architecture Specification, Release 1.0.a by
             InfiniBand Trade Association at www.infinibandta.org

   [IGMP2]   Fenner W., "Internet Group Management Protocol, Version 2",
             RFC 2236, November 1997.

   [IPMULT]  Deering S., "Host Extensions for IP Multicasting", RFC
             1112, August 1989.

   [IPoIB_ARCH]  draft-ietf-ipoib-architecture-01.txt

   [IPoIB_ENCAP] draft-ietf-ipoib-ip-over-infiniband-01.txt

   [IPV6]    Deering, S. and R. Hinden, "Internet Protocol, Version 6
             (IPv6) Specification", RFC 2460, December 1998.



Chu & Kashyap                                                  [Page 12]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   [IP6MLD]  Deering S., Fenner W., Haberman B., "Multicast Listener
             Discovery (MLD) for IPv6", RFC 2710, October 1999.

   [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
             Requirement Levels", BCP 14, RFC 2119, March 1997.


14.0 Author's Address

   H.K. Jerry Chu
   17 Network Circle, UMPK17-201
   Menlo Park, CA 94025
   USA

   Phone: +1 650 786-5146
   EMail: jerry.chu@sun.com


   Vivek Kashyap
   IBM
   15450, SW Koll Parkway
   Beaverton, OR 97006

   Phone: 503 578 3422
   EMail: vivk@us.ibm.com



15.0 Full Copyright Statement

   Copyright (C) The Internet Society (2003>.  All Rights Reserved.

   This document and translations of it may be copied and furnished to
   others, and derivative works that comment on or otherwise explain it
   or assist in its implementation may be prepared, copied, published
   and distributed, in whole or in part, without restriction of any
   kind, provided that the above copyright notice and this paragraph are
   included on all such copies and derivative works.  However, this
   document itself may not be modified in any way, such as by removing
   the copyright notice or references to the Internet Society or other
   Internet organizations, except as needed for the purpose of
   developing Internet standards in which case the procedures for
   copyrights defined in the Internet Standards process must be
   followed, or as required to translate it into languages other than
   English.

   The limited permissions granted above are perpetual and will not be
   revoked by the Internet Society or its successors or assigns.



Chu & Kashyap                                                  [Page 13]

draft-ietf-ipoib-link-multicast-03.txt                        March 2003


   This document and the information contained herein is provided on an
   "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
   TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
   BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
   HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
   MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.













































Chu & Kashyap                                                  [Page 14]


--Grist_of_Bees_910_000--
_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From ipoverib-admin@ietf.org  Mon Mar  3 20:48:47 2003
Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA20591
	for <ipoverib-archive@lists.ietf.org>; Mon, 3 Mar 2003 20:48:47 -0500 (EST)
Received: from www1.ietf.org (localhost.localdomain [127.0.0.1])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h241wBp12887;
	Mon, 3 Mar 2003 20:58:11 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h241qAp12693
	for <ipoverib@optimus.ietf.org>; Mon, 3 Mar 2003 20:52:10 -0500
Received: from sccrmhc03.attbi.com (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA20503
	for <ipoverib@ietf.org>; Mon, 3 Mar 2003 20:41:25 -0500 (EST)
Received: from mobilebill (12-211-30-155.client.attbi.com[12.211.30.155])
          by sccrmhc03.attbi.com (sccrmhc03) with SMTP
          id <2003030401432600300ai9ore>; Tue, 4 Mar 2003 01:43:27 +0000
From: "bill" <bill@strahm.net>
To: <ipoverib@ietf.org>
Date: Mon, 3 Mar 2003 17:43:12 -0800
Message-ID: <000001c2e1ef$6e5e91c0$6501a8c0@mobilebill>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook, Build 10.0.2627
Importance: Normal
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106
Content-Transfer-Encoding: 7bit
Subject: [Ipoverib] FW: IETF 56 Presentation Submissions
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-BeenThere: ipoverib@ietf.org
X-Mailman-Version: 2.0.12
Precedence: bulk
List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=unsubscribe>
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
List-Post: <mailto:ipoverib@ietf.org>
List-Help: <mailto:ipoverib-request@ietf.org?subject=help>
List-Subscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=subscribe>
Content-Transfer-Encoding: 7bit

If you are presenting and want your slides available before hand...
Please submit them (Guess that means doing your slides Monday night over
beer isn't considered an option anymore)

Bill

-----Original Message-----
From: owner-wgchairs@ietf.org [mailto:owner-wgchairs@ietf.org] On Behalf
Of IETF Proceedings Administrator
Sent: Monday, March 03, 2003 11:53 AM
To: wgchairs@ietf.org; bofchairs@ietf.org; Steve Coya
Subject: IETF 56 Presentation Submissions


  This message is for all group chairs who will be meeting in San
Francisco.

Presentations may be submitted for online viewing at IETF 56 along with
any interim sessions since Atlanta (55).

These materials may be sent in until Thursday, March 14th @ 5:00pm.
After this cutoff date materials may be submitted onsite at the
Registration Desk, however please allow at least 1 day for new posts to
be listed online.

Please see   www.ietf.org/proceedings/03mar/  for guidelines and the
group submission form.

Thank you,
IETF Secretariat







_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


From mailnull@www1.ietf.org  Mon Mar  3 20:48:48 2003
Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA20604
	for <ipoverib-archive@odin.ietf.org>; Mon, 3 Mar 2003 20:48:48 -0500 (EST)
Received: (from mailnull@localhost)
	by www1.ietf.org (8.11.6/8.11.6) id h241x2L12962
	for ipoverib-archive@odin.ietf.org; Mon, 3 Mar 2003 20:59:02 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h241x2p12958
	for <ipoverib-web-archive@optimus.ietf.org>; Mon, 3 Mar 2003 20:59:02 -0500
Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA20584
	for <ipoverib-web-archive@ietf.org>; Mon, 3 Mar 2003 20:48:16 -0500 (EST)
Received: from www1.ietf.org (localhost.localdomain [127.0.0.1])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h241wBp12887;
	Mon, 3 Mar 2003 20:58:11 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h241qAp12693
	for <ipoverib@optimus.ietf.org>; Mon, 3 Mar 2003 20:52:10 -0500
Received: from sccrmhc03.attbi.com (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA20503
	for <ipoverib@ietf.org>; Mon, 3 Mar 2003 20:41:25 -0500 (EST)
Received: from mobilebill (12-211-30-155.client.attbi.com[12.211.30.155])
          by sccrmhc03.attbi.com (sccrmhc03) with SMTP
          id <2003030401432600300ai9ore>; Tue, 4 Mar 2003 01:43:27 +0000
From: "bill" <bill@strahm.net>
To: <ipoverib@ietf.org>
Date: Mon, 3 Mar 2003 17:43:12 -0800
Message-ID: <000001c2e1ef$6e5e91c0$6501a8c0@mobilebill>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook, Build 10.0.2627
Importance: Normal
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106
Content-Transfer-Encoding: 7bit
Subject: [Ipoverib] FW: IETF 56 Presentation Submissions
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-BeenThere: ipoverib@ietf.org
X-Mailman-Version: 2.0.12
Precedence: bulk
List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=unsubscribe>
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
List-Post: <mailto:ipoverib@ietf.org>
List-Help: <mailto:ipoverib-request@ietf.org?subject=help>
List-Subscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=subscribe>
Content-Transfer-Encoding: 7bit
Content-Transfer-Encoding: 7bit

If you are presenting and want your slides available before hand...
Please submit them (Guess that means doing your slides Monday night over
beer isn't considered an option anymore)

Bill

-----Original Message-----
From: owner-wgchairs@ietf.org [mailto:owner-wgchairs@ietf.org] On Behalf
Of IETF Proceedings Administrator
Sent: Monday, March 03, 2003 11:53 AM
To: wgchairs@ietf.org; bofchairs@ietf.org; Steve Coya
Subject: IETF 56 Presentation Submissions


  This message is for all group chairs who will be meeting in San
Francisco.

Presentations may be submitted for online viewing at IETF 56 along with
any interim sessions since Atlanta (55).

These materials may be sent in until Thursday, March 14th @ 5:00pm.
After this cutoff date materials may be submitted onsite at the
Registration Desk, however please allow at least 1 day for new posts to
be listed online.

Please see   www.ietf.org/proceedings/03mar/  for guidelines and the
group submission form.

Thank you,
IETF Secretariat







_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From ipoverib-admin@ietf.org  Mon Mar  3 20:54:08 2003
Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA20800
	for <ipoverib-archive@lists.ietf.org>; Mon, 3 Mar 2003 20:54:08 -0500 (EST)
Received: from www1.ietf.org (localhost.localdomain [127.0.0.1])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h24212p13152;
	Mon, 3 Mar 2003 21:01:02 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2420jp13107
	for <ipoverib@optimus.ietf.org>; Mon, 3 Mar 2003 21:00:45 -0500
Received: from pheriche.sun.com (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA20687
	for <ipoverib@ietf.org>; Mon, 3 Mar 2003 20:50:00 -0500 (EST)
Received: from engmail1mpk.Eng.Sun.COM ([129.146.1.45])
	by pheriche.sun.com (8.9.3+Sun/8.9.3) with ESMTP id SAA00098
	for <ipoverib@ietf.org>; Mon, 3 Mar 2003 18:52:01 -0700 (MST)
Received: from phys-ha1sun-1 (phys-ha1sun-1.Eng.Sun.COM [129.144.135.11])
	by engmail1mpk.Eng.Sun.COM (8.12.2+Sun/8.12.2/ENSMAIL,v2.2) with ESMTP id h241q1VK006740
	for <ipoverib@ietf.org>; Mon, 3 Mar 2003 17:52:01 -0800 (PST)
Received: from Sun.COM (sr1-usun-01.Eng.Sun.COM [129.144.135.4])
 by ha1sun-mail1.eng.sun.com (iPlanet Messaging Server 5.2 (built Feb 21 2002))
 with ESMTP id <0HB700E0HBUPKK@ha1sun-mail1.eng.sun.com> for ipoverib@ietf.org;
 Mon, 03 Mar 2003 17:52:01 -0800 (PST)
Date: Mon, 03 Mar 2003 17:52:01 -0800
From: Kanoj Sarcar <Kanoj.Sarcar@Sun.COM>
To: ipoverib@ietf.org
Reply-to: Kanoj.Sarcar@Sun.COM
Message-id: <3E6406C1.BEBBBAEB@Sun.COM>
MIME-version: 1.0
X-Mailer: Mozilla 4.79C-CCK-MCD  [en] (X11; U; SunOS 5.8 sun4u)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
X-Accept-Language: en
Content-Transfer-Encoding: 7BIT
Subject: [Ipoverib] RARP usage
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-BeenThere: ipoverib@ietf.org
X-Mailman-Version: 2.0.12
Precedence: bulk
List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=unsubscribe>
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
List-Post: <mailto:ipoverib@ietf.org>
List-Help: <mailto:ipoverib-request@ietf.org?subject=help>
List-Subscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=subscribe>
Content-Transfer-Encoding: 7BIT

Hi,

In section 3.0 of 

http://www.ietf.org/internet-drafts/draft-ietf-ipoib-ip-over-infiniband-01.txt

RARP is indicated as one of the valid traffic types. This might lead people
to falsely believe that RARP needs to be implemented on IPoIB. In fact, this
might not be possible, since there is no mechanism in IBA to allocate the
same QPN to the same interface each time the machine reboots, thus the mac
address defined in the draft is variable across reboots, and thus RARP does
not make sense.

As pointed out in section 2.1.1 in

http://www.ietf.org/internet-drafts/draft-ietf-ipoib-dhcp-over-infiniband-05.txt

some implementations might choose to preserve the same QPN for the same
interface (refer to "... if the implementation is designed to keep this 
association constant across boots"), and only then RARP might make sense. 

I propose some verbiage to the effect of

"Implementation of RARP is optional for IPoIB and is only suggested on
implementations designed to keep the QPN (which is a part of the mac
address) to network interface association constant across boots. If the
implementation does support RARP, it must use the 'ETHER TYPE' specified
in this document."

be added in section 3.0 of the previous draft just after Table 1.

Thanks.

Kanoj
_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


From mailnull@www1.ietf.org  Mon Mar  3 20:54:08 2003
Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA20808
	for <ipoverib-archive@odin.ietf.org>; Mon, 3 Mar 2003 20:54:08 -0500 (EST)
Received: (from mailnull@localhost)
	by www1.ietf.org (8.11.6/8.11.6) id h2424N013322
	for ipoverib-archive@odin.ietf.org; Mon, 3 Mar 2003 21:04:23 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2424Mp13319
	for <ipoverib-web-archive@optimus.ietf.org>; Mon, 3 Mar 2003 21:04:22 -0500
Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA20781
	for <ipoverib-web-archive@ietf.org>; Mon, 3 Mar 2003 20:53:37 -0500 (EST)
Received: from www1.ietf.org (localhost.localdomain [127.0.0.1])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h24212p13152;
	Mon, 3 Mar 2003 21:01:02 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2420jp13107
	for <ipoverib@optimus.ietf.org>; Mon, 3 Mar 2003 21:00:45 -0500
Received: from pheriche.sun.com (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA20687
	for <ipoverib@ietf.org>; Mon, 3 Mar 2003 20:50:00 -0500 (EST)
Received: from engmail1mpk.Eng.Sun.COM ([129.146.1.45])
	by pheriche.sun.com (8.9.3+Sun/8.9.3) with ESMTP id SAA00098
	for <ipoverib@ietf.org>; Mon, 3 Mar 2003 18:52:01 -0700 (MST)
Received: from phys-ha1sun-1 (phys-ha1sun-1.Eng.Sun.COM [129.144.135.11])
	by engmail1mpk.Eng.Sun.COM (8.12.2+Sun/8.12.2/ENSMAIL,v2.2) with ESMTP id h241q1VK006740
	for <ipoverib@ietf.org>; Mon, 3 Mar 2003 17:52:01 -0800 (PST)
Received: from Sun.COM (sr1-usun-01.Eng.Sun.COM [129.144.135.4])
 by ha1sun-mail1.eng.sun.com (iPlanet Messaging Server 5.2 (built Feb 21 2002))
 with ESMTP id <0HB700E0HBUPKK@ha1sun-mail1.eng.sun.com> for ipoverib@ietf.org;
 Mon, 03 Mar 2003 17:52:01 -0800 (PST)
Date: Mon, 03 Mar 2003 17:52:01 -0800
From: Kanoj Sarcar <Kanoj.Sarcar@Sun.COM>
To: ipoverib@ietf.org
Reply-to: Kanoj.Sarcar@Sun.COM
Message-id: <3E6406C1.BEBBBAEB@Sun.COM>
MIME-version: 1.0
X-Mailer: Mozilla 4.79C-CCK-MCD  [en] (X11; U; SunOS 5.8 sun4u)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
X-Accept-Language: en
Content-Transfer-Encoding: 7BIT
Subject: [Ipoverib] RARP usage
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-BeenThere: ipoverib@ietf.org
X-Mailman-Version: 2.0.12
Precedence: bulk
List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=unsubscribe>
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
List-Post: <mailto:ipoverib@ietf.org>
List-Help: <mailto:ipoverib-request@ietf.org?subject=help>
List-Subscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=subscribe>
Content-Transfer-Encoding: 7BIT
Content-Transfer-Encoding: 7BIT

Hi,

In section 3.0 of 

http://www.ietf.org/internet-drafts/draft-ietf-ipoib-ip-over-infiniband-01.txt

RARP is indicated as one of the valid traffic types. This might lead people
to falsely believe that RARP needs to be implemented on IPoIB. In fact, this
might not be possible, since there is no mechanism in IBA to allocate the
same QPN to the same interface each time the machine reboots, thus the mac
address defined in the draft is variable across reboots, and thus RARP does
not make sense.

As pointed out in section 2.1.1 in

http://www.ietf.org/internet-drafts/draft-ietf-ipoib-dhcp-over-infiniband-05.txt

some implementations might choose to preserve the same QPN for the same
interface (refer to "... if the implementation is designed to keep this 
association constant across boots"), and only then RARP might make sense. 

I propose some verbiage to the effect of

"Implementation of RARP is optional for IPoIB and is only suggested on
implementations designed to keep the QPN (which is a part of the mac
address) to network interface association constant across boots. If the
implementation does support RARP, it must use the 'ETHER TYPE' specified
in this document."

be added in section 3.0 of the previous draft just after Table 1.

Thanks.

Kanoj
_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From ipoverib-admin@ietf.org  Mon Mar  3 23:32:45 2003
Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA23483
	for <ipoverib-archive@lists.ietf.org>; Mon, 3 Mar 2003 23:32:45 -0500 (EST)
Received: from www1.ietf.org (localhost.localdomain [127.0.0.1])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h244gCp22878;
	Mon, 3 Mar 2003 23:42:13 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h244frp22864
	for <ipoverib@optimus.ietf.org>; Mon, 3 Mar 2003 23:41:53 -0500
Received: from kathmandu.sun.com (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA23428
	for <ipoverib@ietf.org>; Mon, 3 Mar 2003 23:31:03 -0500 (EST)
Received: from engmail2sun.Eng.Sun.COM ([129.144.134.19])
	by kathmandu.sun.com (8.9.3+Sun/8.9.3) with ESMTP id VAA01604
	for <ipoverib@ietf.org>; Mon, 3 Mar 2003 21:33:05 -0700 (MST)
Received: from phys-ha1sun-1 (phys-ha1sun-1.Eng.Sun.COM [129.144.135.11])
	by engmail2sun.Eng.Sun.COM (8.12.8+Sun/8.12.8/ENSMAIL,v2.2) with ESMTP id h244X5J8018578
	for <ipoverib@ietf.org>; Mon, 3 Mar 2003 20:33:05 -0800 (PST)
Received: from Sun.COM (sr1-usun-01.Eng.Sun.COM [129.144.135.4])
 by ha1sun-mail1.eng.sun.com (iPlanet Messaging Server 5.2 (built Feb 21 2002))
 with ESMTP id <0HB700E05JB3KK@ha1sun-mail1.eng.sun.com> for ipoverib@ietf.org;
 Mon, 03 Mar 2003 20:33:03 -0800 (PST)
Date: Mon, 03 Mar 2003 20:33:03 -0800
From: Kanoj Sarcar <Kanoj.Sarcar@Sun.COM>
Subject: Re: [Ipoverib] draft-ietf-ipoib-link-multicast-03.txt
To: "H.K. Jerry Chu" <Jerry.Chu@eng.Sun.COM>
Cc: ipoverib@ietf.org
Reply-to: Kanoj.Sarcar@Sun.COM
Message-id: <3E642C7F.676E3368@Sun.COM>
MIME-version: 1.0
X-Mailer: Mozilla 4.79C-CCK-MCD  [en] (X11; U; SunOS 5.8 sun4u)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
X-Accept-Language: en
References: <200303040021.h240L3im307684@jurassic.eng.sun.com>
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-BeenThere: ipoverib@ietf.org
X-Mailman-Version: 2.0.12
Precedence: bulk
List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=unsubscribe>
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
List-Post: <mailto:ipoverib@ietf.org>
List-Help: <mailto:ipoverib-request@ietf.org?subject=help>
List-Subscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=subscribe>
Content-Transfer-Encoding: 7BIT

Jerry,

The following is in section 4.0:

>    Unreliable multicast is defined by IBA as an optional functionality
>    for channel adaptors (CAs) and switches. In today's IP technology,
>    link multicast has become an indispensable function for better
>    supporting a modern IP network. For this reason, it is required that
>    an IPoIB fabric supports multicast. This includes all the CAs and
>    switches that are part of an IP network.

I would suggest adding in explicit dependencies on the SM. Something like
"This includes all the CAs and switches that are part of an IP network
alongwith multicast capable SMs that provide multicast group creation
and deletion traps." I believe the IBA specs consider the traps optional,
it is better to clearly state the requirement for proper IPoIB operation.

> 
> 10.0 IP Multicast Routing
> 
>    IP multicast routing requires multicast routers to receive a copy of
>    every link multicast packet on a locally connected link [IPMULT,
>    IP6MLD].  For Ethernet this is usually achieved by turning on the
>    promiscuous multicast mode on a locally connected Ethernet interface.
> 
>    IBA does not provide any hardware support for promiscuous multicast
>    mode.  Fortunately a promiscuous multicast mode can be emulated in
>    the software running on a router through the following steps.
> 
>    A) Obtain a list of all active IB multicast groups from the local SA.
> 
>    B) Make a "NonMember" join request to the SA for every group that has
>    a signature in its MGID matching the one for either IPv4 or IPv6.
> 
>    C) Subscribe to the IB multicast group creation events using a
>    wildcarded MGID so that the router can "NonMember" join all IB
>    multicast groups created subsequently for IPv4 or IPv6.
> 

Looking at step C) in detail, the IP multicast router must make sure
to attach the QP representing the IPoIB interface to the newly created
multicast group which triggered the creation trap. Thus, there are two
parts to handling the creation trap; the first is to NonMember join 
the group, and the second is to attach the QPN to the group. The point
to note is that either the join or the attach might fail, depending
on characterstics of the switches and the hca involved. The attach is
more probable to fail due to multicast related hca limits specified in
section 11.2.1.2 in IBA. If the join/attach fails beyond a certain
limit, IP multicast routing might not work well beyond that.

To get around this problem, one solution I can think of is to make
all the nodes on the link aware that the hca running the IP mrouted
or the fabric has reached its multicast limits, thus all multicast
traffic should now use the v4 broadcast or the v6 all-node group.
If the individual nodes degrade to using link level broadcast
to achieve multicasting, IP mrouted should be able to receive all 
packets again then. If we adopt this solution, we also need to 
standardize the mechanism by which the router will indicate this 
to the nodes on the link. Section 9 already requires the nodes
to subscribe to group creation/deletion traps, thus IP mrouted
might indicate the multicast-limit condition by the creation of
a special multicast group (given the 28 bit IPv4 multicast group
id, there are a lot of IBA multicast groups that will never be
used for traffic; pick one of them, eg FF12:401B:PKEY:8000:0:0:0:0).
The router can then delete the group (at an implementation specific
time), when it can ensure that it will be able to receive all the
multicast traffic without requiring the nodes to degrade to using
broadcast.

Thanks.

Kanoj
_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


From mailnull@www1.ietf.org  Mon Mar  3 23:32:46 2003
Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA23493
	for <ipoverib-archive@odin.ietf.org>; Mon, 3 Mar 2003 23:32:46 -0500 (EST)
Received: (from mailnull@localhost)
	by www1.ietf.org (8.11.6/8.11.6) id h244h4V22926
	for ipoverib-archive@odin.ietf.org; Mon, 3 Mar 2003 23:43:04 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h244h4p22923
	for <ipoverib-web-archive@optimus.ietf.org>; Mon, 3 Mar 2003 23:43:04 -0500
Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA23460
	for <ipoverib-web-archive@ietf.org>; Mon, 3 Mar 2003 23:32:14 -0500 (EST)
Received: from www1.ietf.org (localhost.localdomain [127.0.0.1])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h244gCp22878;
	Mon, 3 Mar 2003 23:42:13 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h244frp22864
	for <ipoverib@optimus.ietf.org>; Mon, 3 Mar 2003 23:41:53 -0500
Received: from kathmandu.sun.com (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA23428
	for <ipoverib@ietf.org>; Mon, 3 Mar 2003 23:31:03 -0500 (EST)
Received: from engmail2sun.Eng.Sun.COM ([129.144.134.19])
	by kathmandu.sun.com (8.9.3+Sun/8.9.3) with ESMTP id VAA01604
	for <ipoverib@ietf.org>; Mon, 3 Mar 2003 21:33:05 -0700 (MST)
Received: from phys-ha1sun-1 (phys-ha1sun-1.Eng.Sun.COM [129.144.135.11])
	by engmail2sun.Eng.Sun.COM (8.12.8+Sun/8.12.8/ENSMAIL,v2.2) with ESMTP id h244X5J8018578
	for <ipoverib@ietf.org>; Mon, 3 Mar 2003 20:33:05 -0800 (PST)
Received: from Sun.COM (sr1-usun-01.Eng.Sun.COM [129.144.135.4])
 by ha1sun-mail1.eng.sun.com (iPlanet Messaging Server 5.2 (built Feb 21 2002))
 with ESMTP id <0HB700E05JB3KK@ha1sun-mail1.eng.sun.com> for ipoverib@ietf.org;
 Mon, 03 Mar 2003 20:33:03 -0800 (PST)
Date: Mon, 03 Mar 2003 20:33:03 -0800
From: Kanoj Sarcar <Kanoj.Sarcar@Sun.COM>
Subject: Re: [Ipoverib] draft-ietf-ipoib-link-multicast-03.txt
To: "H.K. Jerry Chu" <Jerry.Chu@eng.Sun.COM>
Cc: ipoverib@ietf.org
Reply-to: Kanoj.Sarcar@Sun.COM
Message-id: <3E642C7F.676E3368@Sun.COM>
MIME-version: 1.0
X-Mailer: Mozilla 4.79C-CCK-MCD  [en] (X11; U; SunOS 5.8 sun4u)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
X-Accept-Language: en
References: <200303040021.h240L3im307684@jurassic.eng.sun.com>
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-BeenThere: ipoverib@ietf.org
X-Mailman-Version: 2.0.12
Precedence: bulk
List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=unsubscribe>
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
List-Post: <mailto:ipoverib@ietf.org>
List-Help: <mailto:ipoverib-request@ietf.org?subject=help>
List-Subscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=subscribe>
Content-Transfer-Encoding: 7BIT
Content-Transfer-Encoding: 7BIT

Jerry,

The following is in section 4.0:

>    Unreliable multicast is defined by IBA as an optional functionality
>    for channel adaptors (CAs) and switches. In today's IP technology,
>    link multicast has become an indispensable function for better
>    supporting a modern IP network. For this reason, it is required that
>    an IPoIB fabric supports multicast. This includes all the CAs and
>    switches that are part of an IP network.

I would suggest adding in explicit dependencies on the SM. Something like
"This includes all the CAs and switches that are part of an IP network
alongwith multicast capable SMs that provide multicast group creation
and deletion traps." I believe the IBA specs consider the traps optional,
it is better to clearly state the requirement for proper IPoIB operation.

> 
> 10.0 IP Multicast Routing
> 
>    IP multicast routing requires multicast routers to receive a copy of
>    every link multicast packet on a locally connected link [IPMULT,
>    IP6MLD].  For Ethernet this is usually achieved by turning on the
>    promiscuous multicast mode on a locally connected Ethernet interface.
> 
>    IBA does not provide any hardware support for promiscuous multicast
>    mode.  Fortunately a promiscuous multicast mode can be emulated in
>    the software running on a router through the following steps.
> 
>    A) Obtain a list of all active IB multicast groups from the local SA.
> 
>    B) Make a "NonMember" join request to the SA for every group that has
>    a signature in its MGID matching the one for either IPv4 or IPv6.
> 
>    C) Subscribe to the IB multicast group creation events using a
>    wildcarded MGID so that the router can "NonMember" join all IB
>    multicast groups created subsequently for IPv4 or IPv6.
> 

Looking at step C) in detail, the IP multicast router must make sure
to attach the QP representing the IPoIB interface to the newly created
multicast group which triggered the creation trap. Thus, there are two
parts to handling the creation trap; the first is to NonMember join 
the group, and the second is to attach the QPN to the group. The point
to note is that either the join or the attach might fail, depending
on characterstics of the switches and the hca involved. The attach is
more probable to fail due to multicast related hca limits specified in
section 11.2.1.2 in IBA. If the join/attach fails beyond a certain
limit, IP multicast routing might not work well beyond that.

To get around this problem, one solution I can think of is to make
all the nodes on the link aware that the hca running the IP mrouted
or the fabric has reached its multicast limits, thus all multicast
traffic should now use the v4 broadcast or the v6 all-node group.
If the individual nodes degrade to using link level broadcast
to achieve multicasting, IP mrouted should be able to receive all 
packets again then. If we adopt this solution, we also need to 
standardize the mechanism by which the router will indicate this 
to the nodes on the link. Section 9 already requires the nodes
to subscribe to group creation/deletion traps, thus IP mrouted
might indicate the multicast-limit condition by the creation of
a special multicast group (given the 28 bit IPv4 multicast group
id, there are a lot of IBA multicast groups that will never be
used for traffic; pick one of them, eg FF12:401B:PKEY:8000:0:0:0:0).
The router can then delete the group (at an implementation specific
time), when it can ensure that it will be able to receive all the
multicast traffic without requiring the nodes to degrade to using
broadcast.

Thanks.

Kanoj
_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From ipoverib-admin@ietf.org  Thu Mar  6 06:38:39 2003
Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA04594
	for <ipoverib-archive@lists.ietf.org>; Thu, 6 Mar 2003 06:38:39 -0500 (EST)
Received: from www1.ietf.org (localhost.localdomain [127.0.0.1])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h26BmdO27528;
	Thu, 6 Mar 2003 06:48:39 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h26BchO26453
	for <ipoverib@optimus.ietf.org>; Thu, 6 Mar 2003 06:38:43 -0500
Received: from CNRI.Reston.VA.US (localhost [127.0.0.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA02874;
	Thu, 6 Mar 2003 06:27:13 -0500 (EST)
Message-Id: <200303061127.GAA02874@ietf.org>
Mime-Version: 1.0
Content-Type: Multipart/Mixed; Boundary="NextPart"
To: IETF-Announce: ;
Cc: ipoverib@ietf.org
From: Internet-Drafts@ietf.org
Reply-to: Internet-Drafts@ietf.org
Date: Thu, 06 Mar 2003 06:27:13 -0500
Subject: [Ipoverib] I-D ACTION:draft-ietf-ipoib-link-multicast-03.txt
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-BeenThere: ipoverib@ietf.org
X-Mailman-Version: 2.0.12
Precedence: bulk
List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=unsubscribe>
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
List-Post: <mailto:ipoverib@ietf.org>
List-Help: <mailto:ipoverib-request@ietf.org?subject=help>
List-Subscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=subscribe>

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the IP over InfiniBand Working Group of the IETF.

	Title		: IP link and multicast over InfiniBand networks
	Author(s)	: J. Chu, V. Kashyap
	Filename	: draft-ietf-ipoib-link-multicast-03.txt
	Pages		: 14
	Date		: 2003-3-5
	
This document specifies a method for setting up IP subnets and
multicast services over InfiniBand(TM) networks. Discussions in this
document are applicable to both IPv4 and IPv6, unless explicitly
specified. A separate document will cover unicast and encapsulation
of IP datagrams over InfiniBand networks.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-ipoib-link-multicast-03.txt

To remove yourself from the IETF Announcement list, send a message to 
ietf-announce-request with the word unsubscribe in the body of the message.

Internet-Drafts are also available by anonymous FTP. Login with the username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-ietf-ipoib-link-multicast-03.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt


Internet-Drafts can also be obtained by e-mail.

Send a message to:
	mailserv@ietf.org.
In the body type:
	"FILE /internet-drafts/draft-ietf-ipoib-link-multicast-03.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.
		
		
Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Multipart/Alternative; Boundary="OtherAccess"

--OtherAccess
Content-Type: Message/External-body;
	access-type="mail-server";
	server="mailserv@ietf.org"

Content-Type: text/plain
Content-ID:	<2003-3-5140747.I-D@ietf.org>

ENCODING mime
FILE /internet-drafts/draft-ietf-ipoib-link-multicast-03.txt

--OtherAccess
Content-Type: Message/External-body;
	name="draft-ietf-ipoib-link-multicast-03.txt";
	site="ftp.ietf.org";
	access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID:	<2003-3-5140747.I-D@ietf.org>

--OtherAccess--

--NextPart--


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


From mailnull@www1.ietf.org  Thu Mar  6 06:38:40 2003
Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA04606
	for <ipoverib-archive@odin.ietf.org>; Thu, 6 Mar 2003 06:38:40 -0500 (EST)
Received: (from mailnull@localhost)
	by www1.ietf.org (8.11.6/8.11.6) id h26BndZ27579
	for ipoverib-archive@odin.ietf.org; Thu, 6 Mar 2003 06:49:39 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h26BndO27576
	for <ipoverib-web-archive@optimus.ietf.org>; Thu, 6 Mar 2003 06:49:39 -0500
Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA04582
	for <ipoverib-web-archive@ietf.org>; Thu, 6 Mar 2003 06:38:08 -0500 (EST)
Received: from www1.ietf.org (localhost.localdomain [127.0.0.1])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h26BmdO27528;
	Thu, 6 Mar 2003 06:48:39 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h26BchO26453
	for <ipoverib@optimus.ietf.org>; Thu, 6 Mar 2003 06:38:43 -0500
Received: from CNRI.Reston.VA.US (localhost [127.0.0.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA02874;
	Thu, 6 Mar 2003 06:27:13 -0500 (EST)
Message-Id: <200303061127.GAA02874@ietf.org>
Mime-Version: 1.0
Content-Type: Multipart/Mixed; Boundary="NextPart"
To: IETF-Announce: ;
Cc: ipoverib@ietf.org
From: Internet-Drafts@ietf.org
Reply-to: Internet-Drafts@ietf.org
Date: Thu, 06 Mar 2003 06:27:13 -0500
Subject: [Ipoverib] I-D ACTION:draft-ietf-ipoib-link-multicast-03.txt
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-BeenThere: ipoverib@ietf.org
X-Mailman-Version: 2.0.12
Precedence: bulk
List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=unsubscribe>
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
List-Post: <mailto:ipoverib@ietf.org>
List-Help: <mailto:ipoverib-request@ietf.org?subject=help>
List-Subscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=subscribe>

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the IP over InfiniBand Working Group of the IETF.

	Title		: IP link and multicast over InfiniBand networks
	Author(s)	: J. Chu, V. Kashyap
	Filename	: draft-ietf-ipoib-link-multicast-03.txt
	Pages		: 14
	Date		: 2003-3-5
	
This document specifies a method for setting up IP subnets and
multicast services over InfiniBand(TM) networks. Discussions in this
document are applicable to both IPv4 and IPv6, unless explicitly
specified. A separate document will cover unicast and encapsulation
of IP datagrams over InfiniBand networks.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-ipoib-link-multicast-03.txt

To remove yourself from the IETF Announcement list, send a message to 
ietf-announce-request with the word unsubscribe in the body of the message.

Internet-Drafts are also available by anonymous FTP. Login with the username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-ietf-ipoib-link-multicast-03.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt


Internet-Drafts can also be obtained by e-mail.

Send a message to:
	mailserv@ietf.org.
In the body type:
	"FILE /internet-drafts/draft-ietf-ipoib-link-multicast-03.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.
		
		
Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Multipart/Alternative; Boundary="OtherAccess"

--OtherAccess
Content-Type: Message/External-body;
	access-type="mail-server";
	server="mailserv@ietf.org"

Content-Type: text/plain
Content-ID:	<2003-3-5140747.I-D@ietf.org>

ENCODING mime
FILE /internet-drafts/draft-ietf-ipoib-link-multicast-03.txt

--OtherAccess
Content-Type: Message/External-body;
	name="draft-ietf-ipoib-link-multicast-03.txt";
	site="ftp.ietf.org";
	access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID:	<2003-3-5140747.I-D@ietf.org>

--OtherAccess--

--NextPart--


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From ipoverib-admin@ietf.org  Fri Mar  7 09:56:06 2003
Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA07432
	for <ipoverib-archive@lists.ietf.org>; Fri, 7 Mar 2003 09:56:06 -0500 (EST)
Received: from www1.ietf.org (localhost.localdomain [127.0.0.1])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h27F6cO00501;
	Fri, 7 Mar 2003 10:06:38 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h27F3RO32643
	for <ipoverib@optimus.ietf.org>; Fri, 7 Mar 2003 10:03:27 -0500
Received: from CNRI.Reston.VA.US (localhost [127.0.0.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA06665;
	Fri, 7 Mar 2003 09:51:23 -0500 (EST)
Message-Id: <200303071451.JAA06665@ietf.org>
Mime-Version: 1.0
Content-Type: Multipart/Mixed; Boundary="NextPart"
To: IETF-Announce: ;
Cc: ipoverib@ietf.org
From: Internet-Drafts@ietf.org
Reply-to: Internet-Drafts@ietf.org
Date: Fri, 07 Mar 2003 09:51:22 -0500
Subject: [Ipoverib] I-D ACTION:draft-ietf-ipoib-ip-over-infiniband-02.txt
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-BeenThere: ipoverib@ietf.org
X-Mailman-Version: 2.0.12
Precedence: bulk
List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=unsubscribe>
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
List-Post: <mailto:ipoverib@ietf.org>
List-Help: <mailto:ipoverib-request@ietf.org?subject=help>
List-Subscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=subscribe>

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the IP over InfiniBand Working Group of the IETF.

	Title		: IP encapsulation and address resolution over 
			  InfiniBand networks
	Author(s)	: V. Kashyap, J. Chu
	Filename	: draft-ietf-ipoib-ip-over-infiniband-02.txt
	Pages		: 14
	Date		: 2003-3-6
	
This document specifies the frame format for transmission of
IP and ARP packets over InfiniBand networks. Unless explicitly
specified, the term 'IP' refers to both IPv4 and IPv6. The
term 'ARP' refers to all the ARP protocols/op-codes such as
ARP/RARP. This document also describes the method of forming
IPv6 link-local addresses, and the content of the
source/target link layer address option used in Neighbour
solicitation and advertisement, router advertisement, router
redirect and router solicitation on IPv6 over InfiniBand.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-ipoib-ip-over-infiniband-02.txt

To remove yourself from the IETF Announcement list, send a message to 
ietf-announce-request with the word unsubscribe in the body of the message.

Internet-Drafts are also available by anonymous FTP. Login with the username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-ietf-ipoib-ip-over-infiniband-02.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt


Internet-Drafts can also be obtained by e-mail.

Send a message to:
	mailserv@ietf.org.
In the body type:
	"FILE /internet-drafts/draft-ietf-ipoib-ip-over-infiniband-02.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.
		
		
Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Multipart/Alternative; Boundary="OtherAccess"

--OtherAccess
Content-Type: Message/External-body;
	access-type="mail-server";
	server="mailserv@ietf.org"

Content-Type: text/plain
Content-ID:	<2003-3-6142802.I-D@ietf.org>

ENCODING mime
FILE /internet-drafts/draft-ietf-ipoib-ip-over-infiniband-02.txt

--OtherAccess
Content-Type: Message/External-body;
	name="draft-ietf-ipoib-ip-over-infiniband-02.txt";
	site="ftp.ietf.org";
	access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID:	<2003-3-6142802.I-D@ietf.org>

--OtherAccess--

--NextPart--


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


From mailnull@www1.ietf.org  Fri Mar  7 09:56:07 2003
Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA07444
	for <ipoverib-archive@odin.ietf.org>; Fri, 7 Mar 2003 09:56:07 -0500 (EST)
Received: (from mailnull@localhost)
	by www1.ietf.org (8.11.6/8.11.6) id h27F7ex01396
	for ipoverib-archive@odin.ietf.org; Fri, 7 Mar 2003 10:07:40 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h27F7eO01393
	for <ipoverib-web-archive@optimus.ietf.org>; Fri, 7 Mar 2003 10:07:40 -0500
Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA07410
	for <ipoverib-web-archive@ietf.org>; Fri, 7 Mar 2003 09:55:35 -0500 (EST)
Received: from www1.ietf.org (localhost.localdomain [127.0.0.1])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h27F6cO00501;
	Fri, 7 Mar 2003 10:06:38 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h27F3RO32643
	for <ipoverib@optimus.ietf.org>; Fri, 7 Mar 2003 10:03:27 -0500
Received: from CNRI.Reston.VA.US (localhost [127.0.0.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA06665;
	Fri, 7 Mar 2003 09:51:23 -0500 (EST)
Message-Id: <200303071451.JAA06665@ietf.org>
Mime-Version: 1.0
Content-Type: Multipart/Mixed; Boundary="NextPart"
To: IETF-Announce: ;
Cc: ipoverib@ietf.org
From: Internet-Drafts@ietf.org
Reply-to: Internet-Drafts@ietf.org
Date: Fri, 07 Mar 2003 09:51:22 -0500
Subject: [Ipoverib] I-D ACTION:draft-ietf-ipoib-ip-over-infiniband-02.txt
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-BeenThere: ipoverib@ietf.org
X-Mailman-Version: 2.0.12
Precedence: bulk
List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=unsubscribe>
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
List-Post: <mailto:ipoverib@ietf.org>
List-Help: <mailto:ipoverib-request@ietf.org?subject=help>
List-Subscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=subscribe>

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the IP over InfiniBand Working Group of the IETF.

	Title		: IP encapsulation and address resolution over 
			  InfiniBand networks
	Author(s)	: V. Kashyap, J. Chu
	Filename	: draft-ietf-ipoib-ip-over-infiniband-02.txt
	Pages		: 14
	Date		: 2003-3-6
	
This document specifies the frame format for transmission of
IP and ARP packets over InfiniBand networks. Unless explicitly
specified, the term 'IP' refers to both IPv4 and IPv6. The
term 'ARP' refers to all the ARP protocols/op-codes such as
ARP/RARP. This document also describes the method of forming
IPv6 link-local addresses, and the content of the
source/target link layer address option used in Neighbour
solicitation and advertisement, router advertisement, router
redirect and router solicitation on IPv6 over InfiniBand.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-ipoib-ip-over-infiniband-02.txt

To remove yourself from the IETF Announcement list, send a message to 
ietf-announce-request with the word unsubscribe in the body of the message.

Internet-Drafts are also available by anonymous FTP. Login with the username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-ietf-ipoib-ip-over-infiniband-02.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt


Internet-Drafts can also be obtained by e-mail.

Send a message to:
	mailserv@ietf.org.
In the body type:
	"FILE /internet-drafts/draft-ietf-ipoib-ip-over-infiniband-02.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.
		
		
Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Multipart/Alternative; Boundary="OtherAccess"

--OtherAccess
Content-Type: Message/External-body;
	access-type="mail-server";
	server="mailserv@ietf.org"

Content-Type: text/plain
Content-ID:	<2003-3-6142802.I-D@ietf.org>

ENCODING mime
FILE /internet-drafts/draft-ietf-ipoib-ip-over-infiniband-02.txt

--OtherAccess
Content-Type: Message/External-body;
	name="draft-ietf-ipoib-ip-over-infiniband-02.txt";
	site="ftp.ietf.org";
	access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID:	<2003-3-6142802.I-D@ietf.org>

--OtherAccess--

--NextPart--


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From ipoverib-admin@ietf.org  Mon Mar 10 20:49:02 2003
Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA23836
	for <ipoverib-archive@lists.ietf.org>; Mon, 10 Mar 2003 20:49:02 -0500 (EST)
Received: from www1.ietf.org (localhost.localdomain [127.0.0.1])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2B21DO06513;
	Mon, 10 Mar 2003 21:01:13 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2B20hO06471
	for <ipoverib@optimus.ietf.org>; Mon, 10 Mar 2003 21:00:43 -0500
Received: from e34.co.us.ibm.com (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA23778
	for <ipoverib@ietf.org>; Mon, 10 Mar 2003 20:46:58 -0500 (EST)
Received: from westrelay04.boulder.ibm.com (westrelay04.boulder.ibm.com [9.17.193.32])
	by e34.co.us.ibm.com (8.12.8/8.12.2) with ESMTP id h2B1n564163826;
	Mon, 10 Mar 2003 20:49:05 -0500
Received: from d03nm122.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82])
	by westrelay04.boulder.ibm.com (8.12.8/NCO/VER6.5) with ESMTP id h2B1n4r4046226;
	Mon, 10 Mar 2003 18:49:04 -0700
Subject: Re: [Ipoverib] RARP usage
To: Kanoj.Sarcar@sun.com
Cc: ipoverib@ietf.org
X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000
Message-ID: <OF3D018AB2.FBB93F4E-ON88256CE6.00076698@us.ibm.com>
From: Vivek Kashyap <vivk@us.ibm.com>
Date: Mon, 10 Mar 2003 17:48:22 -0800
X-MIMETrack: Serialize by Router on D03NM122/03/M/IBM(Release 6.0 [IBM]|December 16, 2002) at
 03/10/2003 18:49:04
MIME-Version: 1.0
Content-type: text/plain; charset=US-ASCII
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-BeenThere: ipoverib@ietf.org
X-Mailman-Version: 2.0.12
Precedence: bulk
List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=unsubscribe>
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
List-Post: <mailto:ipoverib@ietf.org>
List-Help: <mailto:ipoverib-request@ietf.org?subject=help>
List-Subscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=subscribe>







Kanoj,

Yes... similar to the issue with DHCP. It probably is true that the QPN
will not be constant across boots - or, at least it is  likely given no
responses indicating the contrary during the DHCP discussions.

I agree that the draft can be updated to include the caution about
invariant hardware address.  Will update it.

Note that the  draft states the ethertype to be used if RARP is used  but
does not require that RARP be used.

Vivek
--
Vivek Kashyap
Linux Technology Center, IBM
vivk@us.ibm.com
kashyapv@us.ibm.com
Ph: 503 578 3422 T/L: 775 3422



                                                                                     
                      Kanoj Sarcar                                                   
                      <Kanoj.Sarcar@sun        To:       ipoverib@ietf.org           
                      .com>                    cc:                                   
                      Sent by:                 Subject:  [Ipoverib] RARP usage       
                      ipoverib-admin@ie                                              
                      tf.org                                                         
                                                                                     
                                                                                     
                      03/03/2003 05:52                                               
                      PM                                                             
                      Please respond to                                              
                      Kanoj.Sarcar                                                   
                                                                                     
                                                                                     




Hi,

In section 3.0 of

http://www.ietf.org/internet-drafts/draft-ietf-ipoib-ip-over-infiniband-01.txt


RARP is indicated as one of the valid traffic types. This might lead people
to falsely believe that RARP needs to be implemented on IPoIB. In fact,
this
might not be possible, since there is no mechanism in IBA to allocate the
same QPN to the same interface each time the machine reboots, thus the mac
address defined in the draft is variable across reboots, and thus RARP does
not make sense.

As pointed out in section 2.1.1 in

http://www.ietf.org/internet-drafts/draft-ietf-ipoib-dhcp-over-infiniband-05.txt


some implementations might choose to preserve the same QPN for the same
interface (refer to "... if the implementation is designed to keep this
association constant across boots"), and only then RARP might make sense.

I propose some verbiage to the effect of

"Implementation of RARP is optional for IPoIB and is only suggested on
implementations designed to keep the QPN (which is a part of the mac
address) to network interface association constant across boots. If the
implementation does support RARP, it must use the 'ETHER TYPE' specified
in this document."

be added in section 3.0 of the previous draft just after Table 1.

Thanks.

Kanoj
_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


From mailnull@www1.ietf.org  Mon Mar 10 20:49:03 2003
Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA23846
	for <ipoverib-archive@odin.ietf.org>; Mon, 10 Mar 2003 20:49:03 -0500 (EST)
Received: (from mailnull@localhost)
	by www1.ietf.org (8.11.6/8.11.6) id h2B22G806532
	for ipoverib-archive@odin.ietf.org; Mon, 10 Mar 2003 21:02:16 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2B22FO06529
	for <ipoverib-web-archive@optimus.ietf.org>; Mon, 10 Mar 2003 21:02:15 -0500
Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA23822
	for <ipoverib-web-archive@ietf.org>; Mon, 10 Mar 2003 20:48:31 -0500 (EST)
Received: from www1.ietf.org (localhost.localdomain [127.0.0.1])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2B21DO06513;
	Mon, 10 Mar 2003 21:01:13 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2B20hO06471
	for <ipoverib@optimus.ietf.org>; Mon, 10 Mar 2003 21:00:43 -0500
Received: from e34.co.us.ibm.com (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA23778
	for <ipoverib@ietf.org>; Mon, 10 Mar 2003 20:46:58 -0500 (EST)
Received: from westrelay04.boulder.ibm.com (westrelay04.boulder.ibm.com [9.17.193.32])
	by e34.co.us.ibm.com (8.12.8/8.12.2) with ESMTP id h2B1n564163826;
	Mon, 10 Mar 2003 20:49:05 -0500
Received: from d03nm122.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82])
	by westrelay04.boulder.ibm.com (8.12.8/NCO/VER6.5) with ESMTP id h2B1n4r4046226;
	Mon, 10 Mar 2003 18:49:04 -0700
Subject: Re: [Ipoverib] RARP usage
To: Kanoj.Sarcar@sun.com
Cc: ipoverib@ietf.org
X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000
Message-ID: <OF3D018AB2.FBB93F4E-ON88256CE6.00076698@us.ibm.com>
From: Vivek Kashyap <vivk@us.ibm.com>
Date: Mon, 10 Mar 2003 17:48:22 -0800
X-MIMETrack: Serialize by Router on D03NM122/03/M/IBM(Release 6.0 [IBM]|December 16, 2002) at
 03/10/2003 18:49:04
MIME-Version: 1.0
Content-type: text/plain; charset=US-ASCII
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-BeenThere: ipoverib@ietf.org
X-Mailman-Version: 2.0.12
Precedence: bulk
List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=unsubscribe>
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
List-Post: <mailto:ipoverib@ietf.org>
List-Help: <mailto:ipoverib-request@ietf.org?subject=help>
List-Subscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=subscribe>







Kanoj,

Yes... similar to the issue with DHCP. It probably is true that the QPN
will not be constant across boots - or, at least it is  likely given no
responses indicating the contrary during the DHCP discussions.

I agree that the draft can be updated to include the caution about
invariant hardware address.  Will update it.

Note that the  draft states the ethertype to be used if RARP is used  but
does not require that RARP be used.

Vivek
--
Vivek Kashyap
Linux Technology Center, IBM
vivk@us.ibm.com
kashyapv@us.ibm.com
Ph: 503 578 3422 T/L: 775 3422



                                                                                     
                      Kanoj Sarcar                                                   
                      <Kanoj.Sarcar@sun        To:       ipoverib@ietf.org           
                      .com>                    cc:                                   
                      Sent by:                 Subject:  [Ipoverib] RARP usage       
                      ipoverib-admin@ie                                              
                      tf.org                                                         
                                                                                     
                                                                                     
                      03/03/2003 05:52                                               
                      PM                                                             
                      Please respond to                                              
                      Kanoj.Sarcar                                                   
                                                                                     
                                                                                     




Hi,

In section 3.0 of

http://www.ietf.org/internet-drafts/draft-ietf-ipoib-ip-over-infiniband-01.txt


RARP is indicated as one of the valid traffic types. This might lead people
to falsely believe that RARP needs to be implemented on IPoIB. In fact,
this
might not be possible, since there is no mechanism in IBA to allocate the
same QPN to the same interface each time the machine reboots, thus the mac
address defined in the draft is variable across reboots, and thus RARP does
not make sense.

As pointed out in section 2.1.1 in

http://www.ietf.org/internet-drafts/draft-ietf-ipoib-dhcp-over-infiniband-05.txt


some implementations might choose to preserve the same QPN for the same
interface (refer to "... if the implementation is designed to keep this
association constant across boots"), and only then RARP might make sense.

I propose some verbiage to the effect of

"Implementation of RARP is optional for IPoIB and is only suggested on
implementations designed to keep the QPN (which is a part of the mac
address) to network interface association constant across boots. If the
implementation does support RARP, it must use the 'ETHER TYPE' specified
in this document."

be added in section 3.0 of the previous draft just after Table 1.

Thanks.

Kanoj
_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From ipoverib-admin@ietf.org  Tue Mar 11 11:27:35 2003
Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA26091
	for <ipoverib-archive@lists.ietf.org>; Tue, 11 Mar 2003 11:27:35 -0500 (EST)
Received: from www1.ietf.org (localhost.localdomain [127.0.0.1])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2BGeGO11325;
	Tue, 11 Mar 2003 11:40:16 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2BGdTO11247
	for <ipoverib@optimus.ietf.org>; Tue, 11 Mar 2003 11:39:29 -0500
Received: from kathmandu.sun.com (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA25869;
	Tue, 11 Mar 2003 11:25:25 -0500 (EST)
Received: from jurassic.eng.sun.com ([129.146.17.55])
	by kathmandu.sun.com (8.9.3+Sun/8.9.3) with ESMTP id JAA02681;
	Tue, 11 Mar 2003 09:27:32 -0700 (MST)
Received: from taipei (taipei.Eng.Sun.COM [129.146.85.178])
	by jurassic.eng.sun.com (8.12.8+Sun/8.12.8) with SMTP id h2BGRVih138031;
	Tue, 11 Mar 2003 08:27:32 -0800 (PST)
Message-Id: <200303111627.h2BGRVih138031@jurassic.eng.sun.com>
Date: Tue, 11 Mar 2003 08:27:29 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
To: agenda@ietf.org, ipoverib@ietf.org
Cc: bill@strahm.net
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: PMS9EerJZepr6oS7rEMFRA==
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.6_06 SunOS 5.8 sun4u sparc 
Subject: [Ipoverib] IPoIB Agenda 56th IETF
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-BeenThere: ipoverib@ietf.org
X-Mailman-Version: 2.0.12
Precedence: bulk
List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=unsubscribe>
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
List-Post: <mailto:ipoverib@ietf.org>
List-Help: <mailto:ipoverib-request@ietf.org?subject=help>
List-Subscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=subscribe>

56th IETF Meeting - San Francisco, CA, USA
 
IPoIB tentative agenda:
 
TUESDAY, March 18, 2003
0900-1130 Morning Sessions
INT     ipoib           IP over InfiniBand WG
==================================================================
 
Administrivia                           5 mins          Bill/Jerry
        Blue Sheets
        Minutes
        Agenda Bashing
 
WG Document Status                      5 mins          Bill/Jerry
 
IPoIB MIBs Status                       30 mins         Sean Harnedy
 
IP over IB: Connected Mode              30 mins         Vivek Kashyap
(See draft-kashyap-ipoib-connected-mode-00.txt)
 
Wrap Up                                 10 mins         Bill/Jerry

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


From mailnull@www1.ietf.org  Tue Mar 11 11:27:36 2003
Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA26100
	for <ipoverib-archive@odin.ietf.org>; Tue, 11 Mar 2003 11:27:35 -0500 (EST)
Received: (from mailnull@localhost)
	by www1.ietf.org (8.11.6/8.11.6) id h2BGf7Y11386
	for ipoverib-archive@odin.ietf.org; Tue, 11 Mar 2003 11:41:07 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2BGf7O11383
	for <ipoverib-web-archive@optimus.ietf.org>; Tue, 11 Mar 2003 11:41:07 -0500
Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA26046
	for <ipoverib-web-archive@ietf.org>; Tue, 11 Mar 2003 11:27:04 -0500 (EST)
Received: from www1.ietf.org (localhost.localdomain [127.0.0.1])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2BGeGO11325;
	Tue, 11 Mar 2003 11:40:16 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2BGdTO11247
	for <ipoverib@optimus.ietf.org>; Tue, 11 Mar 2003 11:39:29 -0500
Received: from kathmandu.sun.com (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA25869;
	Tue, 11 Mar 2003 11:25:25 -0500 (EST)
Received: from jurassic.eng.sun.com ([129.146.17.55])
	by kathmandu.sun.com (8.9.3+Sun/8.9.3) with ESMTP id JAA02681;
	Tue, 11 Mar 2003 09:27:32 -0700 (MST)
Received: from taipei (taipei.Eng.Sun.COM [129.146.85.178])
	by jurassic.eng.sun.com (8.12.8+Sun/8.12.8) with SMTP id h2BGRVih138031;
	Tue, 11 Mar 2003 08:27:32 -0800 (PST)
Message-Id: <200303111627.h2BGRVih138031@jurassic.eng.sun.com>
Date: Tue, 11 Mar 2003 08:27:29 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
To: agenda@ietf.org, ipoverib@ietf.org
Cc: bill@strahm.net
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: PMS9EerJZepr6oS7rEMFRA==
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.6_06 SunOS 5.8 sun4u sparc 
Subject: [Ipoverib] IPoIB Agenda 56th IETF
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-BeenThere: ipoverib@ietf.org
X-Mailman-Version: 2.0.12
Precedence: bulk
List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=unsubscribe>
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
List-Post: <mailto:ipoverib@ietf.org>
List-Help: <mailto:ipoverib-request@ietf.org?subject=help>
List-Subscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=subscribe>

56th IETF Meeting - San Francisco, CA, USA
 
IPoIB tentative agenda:
 
TUESDAY, March 18, 2003
0900-1130 Morning Sessions
INT     ipoib           IP over InfiniBand WG
==================================================================
 
Administrivia                           5 mins          Bill/Jerry
        Blue Sheets
        Minutes
        Agenda Bashing
 
WG Document Status                      5 mins          Bill/Jerry
 
IPoIB MIBs Status                       30 mins         Sean Harnedy
 
IP over IB: Connected Mode              30 mins         Vivek Kashyap
(See draft-kashyap-ipoib-connected-mode-00.txt)
 
Wrap Up                                 10 mins         Bill/Jerry

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From ipoverib-admin@ietf.org  Mon Mar 17 00:42:08 2003
Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA21672
	for <ipoverib-archive@lists.ietf.org>; Mon, 17 Mar 2003 00:42:08 -0500 (EST)
Received: from www1.ietf.org (localhost.localdomain [127.0.0.1])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2H5vHO24694;
	Mon, 17 Mar 2003 00:57:18 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2H5u8O24597
	for <ipoverib@optimus.ietf.org>; Mon, 17 Mar 2003 00:56:08 -0500
Received: from pheriche.sun.com (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA21618
	for <ipoverib@ietf.org>; Mon, 17 Mar 2003 00:39:22 -0500 (EST)
Received: from jurassic.eng.sun.com ([129.146.17.55])
	by pheriche.sun.com (8.9.3+Sun/8.9.3) with ESMTP id WAA27878
	for <ipoverib@ietf.org>; Sun, 16 Mar 2003 22:41:35 -0700 (MST)
Received: from unknown (vpn-129-150-17-82.SFBay.Sun.COM [129.150.17.82])
	by jurassic.eng.sun.com (8.12.8+Sun/8.12.8) with SMTP id h2H5fUp2656412;
	Sun, 16 Mar 2003 21:41:33 -0800 (PST)
Message-Id: <200303170541.h2H5fUp2656412@jurassic.eng.sun.com>
Date: Sun, 16 Mar 2003 21:46:32 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Subject: Re: [Ipoverib] draft-ietf-ipoib-link-multicast-03.txt
To: Kanoj.Sarcar@Sun.COM
Cc: ipoverib@ietf.org
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: U8jHNhqpCE9FQPRMgTcEOw==
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.2 SunOS 5.8 sun4u sparc 
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-BeenThere: ipoverib@ietf.org
X-Mailman-Version: 2.0.12
Precedence: bulk
List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=unsubscribe>
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
List-Post: <mailto:ipoverib@ietf.org>
List-Help: <mailto:ipoverib-request@ietf.org?subject=help>
List-Subscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=subscribe>

Kanoj,

>The following is in section 4.0:
>
>>    Unreliable multicast is defined by IBA as an optional functionality
>>    for channel adaptors (CAs) and switches. In today's IP technology,
>>    link multicast has become an indispensable function for better
>>    supporting a modern IP network. For this reason, it is required that
>>    an IPoIB fabric supports multicast. This includes all the CAs and
>>    switches that are part of an IP network.
>
>I would suggest adding in explicit dependencies on the SM. Something like
>"This includes all the CAs and switches that are part of an IP network
>alongwith multicast capable SMs that provide multicast group creation
>and deletion traps." I believe the IBA specs consider the traps optional,
>it is better to clearly state the requirement for proper IPoIB operation.

According to section 14.4.10 of IBA release 1.1 volume one, these traps are
part of SA's multicast capability as indicated by the IsUDMulticastSupported
bit in SA's CapabilityMask. Multicast-capable management software including
the SA is already implied by the original statement.

>
>> 
>> 10.0 IP Multicast Routing
>> 
>>    IP multicast routing requires multicast routers to receive a copy of
>>    every link multicast packet on a locally connected link [IPMULT,
>>    IP6MLD].  For Ethernet this is usually achieved by turning on the
>>    promiscuous multicast mode on a locally connected Ethernet interface.
>> 
>>    IBA does not provide any hardware support for promiscuous multicast
>>    mode.  Fortunately a promiscuous multicast mode can be emulated in
>>    the software running on a router through the following steps.
>> 
>>    A) Obtain a list of all active IB multicast groups from the local SA.
>> 
>>    B) Make a "NonMember" join request to the SA for every group that has
>>    a signature in its MGID matching the one for either IPv4 or IPv6.
>> 
>>    C) Subscribe to the IB multicast group creation events using a
>>    wildcarded MGID so that the router can "NonMember" join all IB
>>    multicast groups created subsequently for IPv4 or IPv6.
>> 
>
>Looking at step C) in detail, the IP multicast router must make sure
>to attach the QP representing the IPoIB interface to the newly created
>multicast group which triggered the creation trap. Thus, there are two
>parts to handling the creation trap; the first is to NonMember join 
>the group, and the second is to attach the QPN to the group. The point
>to note is that either the join or the attach might fail, depending
>on characterstics of the switches and the hca involved. The attach is
>more probable to fail due to multicast related hca limits specified in
>section 11.2.1.2 in IBA. If the join/attach fails beyond a certain
>limit, IP multicast routing might not work well beyond that.
>
>To get around this problem, one solution I can think of is to make
>all the nodes on the link aware that the hca running the IP mrouted
>or the fabric has reached its multicast limits, thus all multicast
>traffic should now use the v4 broadcast or the v6 all-node group.
>If the individual nodes degrade to using link level broadcast
>to achieve multicasting, IP mrouted should be able to receive all 
>packets again then. If we adopt this solution, we also need to 
>standardize the mechanism by which the router will indicate this 
>to the nodes on the link. Section 9 already requires the nodes
>to subscribe to group creation/deletion traps, thus IP mrouted
>might indicate the multicast-limit condition by the creation of
>a special multicast group (given the 28 bit IPv4 multicast group
>id, there are a lot of IBA multicast groups that will never be
>used for traffic; pick one of them, eg FF12:401B:PKEY:8000:0:0:0:0).
>The router can then delete the group (at an implementation specific
>time), when it can ensure that it will be able to receive all the
>multicast traffic without requiring the nodes to degrade to using
>broadcast.

IBA spec. already contains a number of error conditions where a
management function can fail, e.g., a FullMember/NonMember/SendOnlyNonMember
join can fail, attaching a QP to a MC group can fail..., etc. These
failure conditions either rarely happen, or are highly implementation
dependent.

Can the failure conditions related to the availability of the local
resources be treated as simply implementation limitations, or must we
address these vulnerability by adding more complexity to IPoIB protocol
to make multicast routing bullet proof?

Jerry

>
>Thanks.
>
>Kanoj

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


From mailnull@www1.ietf.org  Mon Mar 17 00:42:09 2003
Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA21680
	for <ipoverib-archive@odin.ietf.org>; Mon, 17 Mar 2003 00:42:09 -0500 (EST)
Received: (from mailnull@localhost)
	by www1.ietf.org (8.11.6/8.11.6) id h2H5wNS24742
	for ipoverib-archive@odin.ietf.org; Mon, 17 Mar 2003 00:58:23 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2H5wNO24739
	for <ipoverib-web-archive@optimus.ietf.org>; Mon, 17 Mar 2003 00:58:23 -0500
Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA21661
	for <ipoverib-web-archive@ietf.org>; Mon, 17 Mar 2003 00:41:37 -0500 (EST)
Received: from www1.ietf.org (localhost.localdomain [127.0.0.1])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2H5vHO24694;
	Mon, 17 Mar 2003 00:57:18 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2H5u8O24597
	for <ipoverib@optimus.ietf.org>; Mon, 17 Mar 2003 00:56:08 -0500
Received: from pheriche.sun.com (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA21618
	for <ipoverib@ietf.org>; Mon, 17 Mar 2003 00:39:22 -0500 (EST)
Received: from jurassic.eng.sun.com ([129.146.17.55])
	by pheriche.sun.com (8.9.3+Sun/8.9.3) with ESMTP id WAA27878
	for <ipoverib@ietf.org>; Sun, 16 Mar 2003 22:41:35 -0700 (MST)
Received: from unknown (vpn-129-150-17-82.SFBay.Sun.COM [129.150.17.82])
	by jurassic.eng.sun.com (8.12.8+Sun/8.12.8) with SMTP id h2H5fUp2656412;
	Sun, 16 Mar 2003 21:41:33 -0800 (PST)
Message-Id: <200303170541.h2H5fUp2656412@jurassic.eng.sun.com>
Date: Sun, 16 Mar 2003 21:46:32 -0800 (PST)
From: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Reply-To: "H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Subject: Re: [Ipoverib] draft-ietf-ipoib-link-multicast-03.txt
To: Kanoj.Sarcar@Sun.COM
Cc: ipoverib@ietf.org
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: U8jHNhqpCE9FQPRMgTcEOw==
X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.2 SunOS 5.8 sun4u sparc 
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-BeenThere: ipoverib@ietf.org
X-Mailman-Version: 2.0.12
Precedence: bulk
List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=unsubscribe>
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
List-Post: <mailto:ipoverib@ietf.org>
List-Help: <mailto:ipoverib-request@ietf.org?subject=help>
List-Subscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=subscribe>

Kanoj,

>The following is in section 4.0:
>
>>    Unreliable multicast is defined by IBA as an optional functionality
>>    for channel adaptors (CAs) and switches. In today's IP technology,
>>    link multicast has become an indispensable function for better
>>    supporting a modern IP network. For this reason, it is required that
>>    an IPoIB fabric supports multicast. This includes all the CAs and
>>    switches that are part of an IP network.
>
>I would suggest adding in explicit dependencies on the SM. Something like
>"This includes all the CAs and switches that are part of an IP network
>alongwith multicast capable SMs that provide multicast group creation
>and deletion traps." I believe the IBA specs consider the traps optional,
>it is better to clearly state the requirement for proper IPoIB operation.

According to section 14.4.10 of IBA release 1.1 volume one, these traps are
part of SA's multicast capability as indicated by the IsUDMulticastSupported
bit in SA's CapabilityMask. Multicast-capable management software including
the SA is already implied by the original statement.

>
>> 
>> 10.0 IP Multicast Routing
>> 
>>    IP multicast routing requires multicast routers to receive a copy of
>>    every link multicast packet on a locally connected link [IPMULT,
>>    IP6MLD].  For Ethernet this is usually achieved by turning on the
>>    promiscuous multicast mode on a locally connected Ethernet interface.
>> 
>>    IBA does not provide any hardware support for promiscuous multicast
>>    mode.  Fortunately a promiscuous multicast mode can be emulated in
>>    the software running on a router through the following steps.
>> 
>>    A) Obtain a list of all active IB multicast groups from the local SA.
>> 
>>    B) Make a "NonMember" join request to the SA for every group that has
>>    a signature in its MGID matching the one for either IPv4 or IPv6.
>> 
>>    C) Subscribe to the IB multicast group creation events using a
>>    wildcarded MGID so that the router can "NonMember" join all IB
>>    multicast groups created subsequently for IPv4 or IPv6.
>> 
>
>Looking at step C) in detail, the IP multicast router must make sure
>to attach the QP representing the IPoIB interface to the newly created
>multicast group which triggered the creation trap. Thus, there are two
>parts to handling the creation trap; the first is to NonMember join 
>the group, and the second is to attach the QPN to the group. The point
>to note is that either the join or the attach might fail, depending
>on characterstics of the switches and the hca involved. The attach is
>more probable to fail due to multicast related hca limits specified in
>section 11.2.1.2 in IBA. If the join/attach fails beyond a certain
>limit, IP multicast routing might not work well beyond that.
>
>To get around this problem, one solution I can think of is to make
>all the nodes on the link aware that the hca running the IP mrouted
>or the fabric has reached its multicast limits, thus all multicast
>traffic should now use the v4 broadcast or the v6 all-node group.
>If the individual nodes degrade to using link level broadcast
>to achieve multicasting, IP mrouted should be able to receive all 
>packets again then. If we adopt this solution, we also need to 
>standardize the mechanism by which the router will indicate this 
>to the nodes on the link. Section 9 already requires the nodes
>to subscribe to group creation/deletion traps, thus IP mrouted
>might indicate the multicast-limit condition by the creation of
>a special multicast group (given the 28 bit IPv4 multicast group
>id, there are a lot of IBA multicast groups that will never be
>used for traffic; pick one of them, eg FF12:401B:PKEY:8000:0:0:0:0).
>The router can then delete the group (at an implementation specific
>time), when it can ensure that it will be able to receive all the
>multicast traffic without requiring the nodes to degrade to using
>broadcast.

IBA spec. already contains a number of error conditions where a
management function can fail, e.g., a FullMember/NonMember/SendOnlyNonMember
join can fail, attaching a QP to a MC group can fail..., etc. These
failure conditions either rarely happen, or are highly implementation
dependent.

Can the failure conditions related to the availability of the local
resources be treated as simply implementation limitations, or must we
address these vulnerability by adding more complexity to IPoIB protocol
to make multicast routing bullet proof?

Jerry

>
>Thanks.
>
>Kanoj

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



From ipoverib-admin@ietf.org  Tue Mar 18 23:26:08 2003
Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA15059
	for <ipoverib-archive@lists.ietf.org>; Tue, 18 Mar 2003 23:26:08 -0500 (EST)
Received: from www1.ietf.org (localhost.localdomain [127.0.0.1])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2J4gNO17859;
	Tue, 18 Mar 2003 23:42:23 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2J4fHO17827
	for <ipoverib@optimus.ietf.org>; Tue, 18 Mar 2003 23:41:17 -0500
Received: from nwkea-mail-2.sun.com (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA14964
	for <ipoverib@ietf.org>; Tue, 18 Mar 2003 23:23:32 -0500 (EST)
Received: from engmail1mpk.Eng.Sun.COM ([129.146.1.45])
	by nwkea-mail-2.sun.com (8.9.3+Sun/8.9.3) with ESMTP id UAA10929
	for <ipoverib@ietf.org>; Tue, 18 Mar 2003 20:25:47 -0800 (PST)
Received: from phys-ha1sun-1 (phys-ha1sun-1.Eng.Sun.COM [129.144.135.11])
	by engmail1mpk.Eng.Sun.COM (8.12.8+Sun/8.12.8/ENSMAIL,v2.2) with ESMTP id h2J4PlcU018983
	for <ipoverib@ietf.org>; Tue, 18 Mar 2003 20:25:47 -0800 (PST)
Received: from Sun.COM (sr1-usun-02.Eng.Sun.COM [129.144.135.8])
 by ha1sun-mail1.eng.sun.com (iPlanet Messaging Server 5.2 (built Feb 21 2002))
 with ESMTP id <0HBZ00INYAYZIH@ha1sun-mail1.eng.sun.com> for ipoverib@ietf.org;
 Tue, 18 Mar 2003 20:25:47 -0800 (PST)
Date: Tue, 18 Mar 2003 20:25:47 -0800
From: Kanoj Sarcar <Kanoj.Sarcar@Sun.COM>
Subject: Re: [Ipoverib] draft-ietf-ipoib-link-multicast-03.txt
To: "H.K. Jerry Chu" <Jerry.Chu@eng.Sun.COM>
Cc: ipoverib@ietf.org
Reply-to: Kanoj.Sarcar@Sun.COM
Message-id: <3E77F14B.76A9D446@Sun.COM>
MIME-version: 1.0
X-Mailer: Mozilla 4.79C-CCK-MCD  [en] (X11; U; SunOS 5.8 sun4u)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
X-Accept-Language: en
References: <200303170541.h2H5fUp2656412@jurassic.eng.sun.com>
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-BeenThere: ipoverib@ietf.org
X-Mailman-Version: 2.0.12
Precedence: bulk
List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=unsubscribe>
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
List-Post: <mailto:ipoverib@ietf.org>
List-Help: <mailto:ipoverib-request@ietf.org?subject=help>
List-Subscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=subscribe>
Content-Transfer-Encoding: 7BIT

> >> 10.0 IP Multicast Routing
> >>
> >>    IP multicast routing requires multicast routers to receive a copy of
> >>    every link multicast packet on a locally connected link [IPMULT,
> >>    IP6MLD].  For Ethernet this is usually achieved by turning on the
> >>    promiscuous multicast mode on a locally connected Ethernet interface.
> >>
> >>    IBA does not provide any hardware support for promiscuous multicast
> >>    mode.  Fortunately a promiscuous multicast mode can be emulated in
> >>    the software running on a router through the following steps.
> >>
> >>    A) Obtain a list of all active IB multicast groups from the local SA.
> >>
> >>    B) Make a "NonMember" join request to the SA for every group that has
> >>    a signature in its MGID matching the one for either IPv4 or IPv6.
> >>
> >>    C) Subscribe to the IB multicast group creation events using a
> >>    wildcarded MGID so that the router can "NonMember" join all IB
> >>    multicast groups created subsequently for IPv4 or IPv6.
> >>
> >
> >Looking at step C) in detail, the IP multicast router must make sure
> >to attach the QP representing the IPoIB interface to the newly created
> >multicast group which triggered the creation trap. Thus, there are two
> >parts to handling the creation trap; the first is to NonMember join
> >the group, and the second is to attach the QPN to the group. The point
> >to note is that either the join or the attach might fail, depending
> >on characterstics of the switches and the hca involved. The attach is
> >more probable to fail due to multicast related hca limits specified in
> >section 11.2.1.2 in IBA. If the join/attach fails beyond a certain
> >limit, IP multicast routing might not work well beyond that.
> >
> >To get around this problem, one solution I can think of is to make
> >all the nodes on the link aware that the hca running the IP mrouted
> >or the fabric has reached its multicast limits, thus all multicast
> >traffic should now use the v4 broadcast or the v6 all-node group.
> >If the individual nodes degrade to using link level broadcast
> >to achieve multicasting, IP mrouted should be able to receive all
> >packets again then. If we adopt this solution, we also need to
> >standardize the mechanism by which the router will indicate this
> >to the nodes on the link. Section 9 already requires the nodes
> >to subscribe to group creation/deletion traps, thus IP mrouted
> >might indicate the multicast-limit condition by the creation of
> >a special multicast group (given the 28 bit IPv4 multicast group
> >id, there are a lot of IBA multicast groups that will never be
> >used for traffic; pick one of them, eg FF12:401B:PKEY:8000:0:0:0:0).
> >The router can then delete the group (at an implementation specific
> >time), when it can ensure that it will be able to receive all the
> >multicast traffic without requiring the nodes to degrade to using
> >broadcast.
> 
> IBA spec. already contains a number of error conditions where a
> management function can fail, e.g., a FullMember/NonMember/SendOnlyNonMember
> join can fail, attaching a QP to a MC group can fail..., etc. These
> failure conditions either rarely happen, or are highly implementation
> dependent.
> 
> Can the failure conditions related to the availability of the local
> resources be treated as simply implementation limitations, or must we
> address these vulnerability by adding more complexity to IPoIB protocol
> to make multicast routing bullet proof?
> 
> Jerry
> 
> >
> >Thanks.
> >
> >Kanoj


Jerry,

Note that this is not solely an IP multicast routing issue, although 
that is the most thorny case (whose failures are hardest to deal with).

The join/attach has to be done at

1. device/protocol initialization time, and on most systems, a failure
at this point is easily communicated to the sysadmin (via ifconfig 
failing etc). 
2. IP-join time, ie when a process on a host joins a IP multicast
group, and again, the failure at this point is probably communicable
to the process synchronously.
3. IP multicast sender who has not joined the IP group (case B in
section 9.0); failure at this point may or may not be communicable
to writing process (system dependent).
4. Multicast router trying to join/attach, in which case the failure
is not communicable to mrouted (since he has made the system call 
to request receipt of all multicast packets in the past and the
call succeeded).

Now, it seems to me that an hca might well have a limit in the
number of mcg's it supports in hardware, and once that limit is
reached, all the above cases will fail for new groups. Contrast
that with an ethernet fabric (which in legacy mode floods multicast
packets to all nodes) and nics (most allow promiscuous mode
to be set by software when the number of multicast groups reaches 
a limit).

To keep IP multicasting working, an easy way might be to use mcg 
create/delete traps to signal to all members to use the broadcast 
gid for transmitting multicast packets. Any node that determines it 
has reached the limit can then FullMember join the special mcg (with
same scope and attributes as the all-nodes group), and leave it 
when it can again optimally support the required number of groups 
after some of its processes terminate or stop using IP multicasting. 
The mcg traps will thus be clear indications to all subscribing nodes
whether any member requires the broadcast-fallback handling.

Specially formatted broadcast packets to signal these conditions
can be an alternative, but firstly the packets themselves can get
lost due to UD nature, and secondly will require all hosts to 
reference count how many members request broadcast-fallback handling.

Of course, if the typical case is that join/attach will not fail
(ie the number of mcg's are not too great), we can probably punt
on this, but at least with a note that a limit on the number of
IP multicast groups exists depending on member hca's and fabric
(switch/SM) capability.

Thanks.

Kanoj
_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib


From mailnull@www1.ietf.org  Tue Mar 18 23:26:09 2003
Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged))
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA15068
	for <ipoverib-archive@odin.ietf.org>; Tue, 18 Mar 2003 23:26:09 -0500 (EST)
Received: (from mailnull@localhost)
	by www1.ietf.org (8.11.6/8.11.6) id h2J4hLc17921
	for ipoverib-archive@odin.ietf.org; Tue, 18 Mar 2003 23:43:21 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2J4hLO17918
	for <ipoverib-web-archive@optimus.ietf.org>; Tue, 18 Mar 2003 23:43:21 -0500
Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA15052
	for <ipoverib-web-archive@ietf.org>; Tue, 18 Mar 2003 23:25:37 -0500 (EST)
Received: from www1.ietf.org (localhost.localdomain [127.0.0.1])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2J4gNO17859;
	Tue, 18 Mar 2003 23:42:23 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176])
	by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2J4fHO17827
	for <ipoverib@optimus.ietf.org>; Tue, 18 Mar 2003 23:41:17 -0500
Received: from nwkea-mail-2.sun.com (ietf-mx.ietf.org [132.151.6.1])
	by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA14964
	for <ipoverib@ietf.org>; Tue, 18 Mar 2003 23:23:32 -0500 (EST)
Received: from engmail1mpk.Eng.Sun.COM ([129.146.1.45])
	by nwkea-mail-2.sun.com (8.9.3+Sun/8.9.3) with ESMTP id UAA10929
	for <ipoverib@ietf.org>; Tue, 18 Mar 2003 20:25:47 -0800 (PST)
Received: from phys-ha1sun-1 (phys-ha1sun-1.Eng.Sun.COM [129.144.135.11])
	by engmail1mpk.Eng.Sun.COM (8.12.8+Sun/8.12.8/ENSMAIL,v2.2) with ESMTP id h2J4PlcU018983
	for <ipoverib@ietf.org>; Tue, 18 Mar 2003 20:25:47 -0800 (PST)
Received: from Sun.COM (sr1-usun-02.Eng.Sun.COM [129.144.135.8])
 by ha1sun-mail1.eng.sun.com (iPlanet Messaging Server 5.2 (built Feb 21 2002))
 with ESMTP id <0HBZ00INYAYZIH@ha1sun-mail1.eng.sun.com> for ipoverib@ietf.org;
 Tue, 18 Mar 2003 20:25:47 -0800 (PST)
Date: Tue, 18 Mar 2003 20:25:47 -0800
From: Kanoj Sarcar <Kanoj.Sarcar@Sun.COM>
Subject: Re: [Ipoverib] draft-ietf-ipoib-link-multicast-03.txt
To: "H.K. Jerry Chu" <Jerry.Chu@eng.Sun.COM>
Cc: ipoverib@ietf.org
Reply-to: Kanoj.Sarcar@Sun.COM
Message-id: <3E77F14B.76A9D446@Sun.COM>
MIME-version: 1.0
X-Mailer: Mozilla 4.79C-CCK-MCD  [en] (X11; U; SunOS 5.8 sun4u)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
X-Accept-Language: en
References: <200303170541.h2H5fUp2656412@jurassic.eng.sun.com>
Content-Transfer-Encoding: 7BIT
Sender: ipoverib-admin@ietf.org
Errors-To: ipoverib-admin@ietf.org
X-BeenThere: ipoverib@ietf.org
X-Mailman-Version: 2.0.12
Precedence: bulk
List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=unsubscribe>
List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
List-Post: <mailto:ipoverib@ietf.org>
List-Help: <mailto:ipoverib-request@ietf.org?subject=help>
List-Subscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
	<mailto:ipoverib-request@ietf.org?subject=subscribe>
Content-Transfer-Encoding: 7BIT
Content-Transfer-Encoding: 7BIT

> >> 10.0 IP Multicast Routing
> >>
> >>    IP multicast routing requires multicast routers to receive a copy of
> >>    every link multicast packet on a locally connected link [IPMULT,
> >>    IP6MLD].  For Ethernet this is usually achieved by turning on the
> >>    promiscuous multicast mode on a locally connected Ethernet interface.
> >>
> >>    IBA does not provide any hardware support for promiscuous multicast
> >>    mode.  Fortunately a promiscuous multicast mode can be emulated in
> >>    the software running on a router through the following steps.
> >>
> >>    A) Obtain a list of all active IB multicast groups from the local SA.
> >>
> >>    B) Make a "NonMember" join request to the SA for every group that has
> >>    a signature in its MGID matching the one for either IPv4 or IPv6.
> >>
> >>    C) Subscribe to the IB multicast group creation events using a
> >>    wildcarded MGID so that the router can "NonMember" join all IB
> >>    multicast groups created subsequently for IPv4 or IPv6.
> >>
> >
> >Looking at step C) in detail, the IP multicast router must make sure
> >to attach the QP representing the IPoIB interface to the newly created
> >multicast group which triggered the creation trap. Thus, there are two
> >parts to handling the creation trap; the first is to NonMember join
> >the group, and the second is to attach the QPN to the group. The point
> >to note is that either the join or the attach might fail, depending
> >on characterstics of the switches and the hca involved. The attach is
> >more probable to fail due to multicast related hca limits specified in
> >section 11.2.1.2 in IBA. If the join/attach fails beyond a certain
> >limit, IP multicast routing might not work well beyond that.
> >
> >To get around this problem, one solution I can think of is to make
> >all the nodes on the link aware that the hca running the IP mrouted
> >or the fabric has reached its multicast limits, thus all multicast
> >traffic should now use the v4 broadcast or the v6 all-node group.
> >If the individual nodes degrade to using link level broadcast
> >to achieve multicasting, IP mrouted should be able to receive all
> >packets again then. If we adopt this solution, we also need to
> >standardize the mechanism by which the router will indicate this
> >to the nodes on the link. Section 9 already requires the nodes
> >to subscribe to group creation/deletion traps, thus IP mrouted
> >might indicate the multicast-limit condition by the creation of
> >a special multicast group (given the 28 bit IPv4 multicast group
> >id, there are a lot of IBA multicast groups that will never be
> >used for traffic; pick one of them, eg FF12:401B:PKEY:8000:0:0:0:0).
> >The router can then delete the group (at an implementation specific
> >time), when it can ensure that it will be able to receive all the
> >multicast traffic without requiring the nodes to degrade to using
> >broadcast.
> 
> IBA spec. already contains a number of error conditions where a
> management function can fail, e.g., a FullMember/NonMember/SendOnlyNonMember
> join can fail, attaching a QP to a MC group can fail..., etc. These
> failure conditions either rarely happen, or are highly implementation
> dependent.
> 
> Can the failure conditions related to the availability of the local
> resources be treated as simply implementation limitations, or must we
> address these vulnerability by adding more complexity to IPoIB protocol
> to make multicast routing bullet proof?
> 
> Jerry
> 
> >
> >Thanks.
> >
> >Kanoj


Jerry,

Note that this is not solely an IP multicast routing issue, although 
that is the most thorny case (whose failures are hardest to deal with).

The join/attach has to be done at

1. device/protocol initialization time, and on most systems, a failure
at this point is easily communicated to the sysadmin (via ifconfig 
failing etc). 
2. IP-join time, ie when a process on a host joins a IP multicast
group, and again, the failure at this point is probably communicable
to the process synchronously.
3. IP multicast sender who has not joined the IP group (case B in
section 9.0); failure at this point may or may not be communicable
to writing process (system dependent).
4. Multicast router trying to join/attach, in which case the failure
is not communicable to mrouted (since he has made the system call 
to request receipt of all multicast packets in the past and the
call succeeded).

Now, it seems to me that an hca might well have a limit in the
number of mcg's it supports in hardware, and once that limit is
reached, all the above cases will fail for new groups. Contrast
that with an ethernet fabric (which in legacy mode floods multicast
packets to all nodes) and nics (most allow promiscuous mode
to be set by software when the number of multicast groups reaches 
a limit).

To keep IP multicasting working, an easy way might be to use mcg 
create/delete traps to signal to all members to use the broadcast 
gid for transmitting multicast packets. Any node that determines it 
has reached the limit can then FullMember join the special mcg (with
same scope and attributes as the all-nodes group), and leave it 
when it can again optimally support the required number of groups 
after some of its processes terminate or stop using IP multicasting. 
The mcg traps will thus be clear indications to all subscribing nodes
whether any member requires the broadcast-fallback handling.

Specially formatted broadcast packets to signal these conditions
can be an alternative, but firstly the packets themselves can get
lost due to UD nature, and secondly will require all hosts to 
reference count how many members request broadcast-fallback handling.

Of course, if the typical case is that join/attach will not fail
(ie the number of mcg's are not too great), we can probably punt
on this, but at least with a note that a limit on the number of
IP multicast groups exists depending on member hca's and fabric
(switch/SM) capability.

Thanks.

Kanoj
_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib



