Transmission of IP over InfiniBand (IPoIB)
updated by
- rfc-8064 — Recommendation on Stable IPv6 Interface Identifiers
Extracted elements (26)
A 'SendOnlyNonMember' join is used for multicast senders rather than 'FullMember' because it enables outgoing packet routing without causing packets to be routed to the local port, and the sender is not counted as a member for group lifecycle purposes.
IP multicast routing over IB requires emulating promiscuous multicast mode (not supported in IBA hardware) by using 'NonMember' joins to all active IPoIB multicast groups and subscribing to wildcarded multicast group creation events.
IPoIB subnets are abstract rather than physically demarcated: the link is defined by broadcast group membership (common P_Key, Q_Key, MTU) rather than physical topology. This differs from Ethernet where segment topology defines the link.
IB multicast operations (group creation, join calls, QP attachment) are subject to resource-constraint failures that can cause silent packet-forwarding failures — for example, a router's 'NonMember' join failure for a newly created group. Implementations SHOULD log errors from any IB multicast operation failure.
The QPN in the IPoIB link-layer address may not be constant across reboots or network interface resets. Cached QPN entries in static ARP or RARP servers will fail unless implementations ensure QPN invariance. It is RECOMMENDED that ARP caches be revalidated periodically.
All IB MGIDs allocated for IPoIB purpose MUST set the T-flag to 1, indicating they are transient link-layer multicast addresses.
All IP communication to the relevant IPoIB interface MUST be directed to the QPN included in the link-layer address. ARP reply packets in IPv4 subnets are also directed to this same QPN.
All IPoIB implementations MUST support an MTU of 2044 octets (2048-octet IPoIB-link MTU minus 4-octet encapsulation overhead) as the default configuration. Larger and smaller MTUs MAY be supported.
All IPoIB implementations MUST support IP over the UD (Unreliable Datagram) transport mode of IBA. UD is the only universal transmission method and supports multicasting, partitioning, and 32-bit CRC.
For IPv4 over IB, the IPoIB signature in the MGID MUST be '0x401B'. For IPv6 over IB, the signature MUST be '0x601B'. This distinguishes IPoIB multicast groups from other IB multicast groups.
It is RECOMMENDED that a controlled Q_Key with the high-order bit set be used with the broadcast-GID to prevent non-privileged software from fabricating and sending bogus IP datagrams.
Senders MUST subscribe to multicast group create and delete traps to monitor the status of specific IB multicast groups, enabling proper handling when group membership changes.
The broadcast-GID MUST be set up before an IPoIB subnet can be formed, and every IPoIB interface MUST 'FullMember' join the IB multicast group defined by the broadcast-GID.
The broadcast-GID MUST use a 'Full Membership' P_Key (high-order bit set to 1) so that all members may communicate with one another.
The IPoIB implementation MUST be able to handle packets received with or without the use of GRH (Global Routing Header). GRH is required for multicast and inter-subnet unicast, but optional for intra-subnet unicast.
The Q_Key received on joining the broadcast group MUST be used for all IPoIB communication over the particular IPoIB link.
When a node joins an IP multicast group, the new IB multicast group's MGID MUST use the same P_Key, Q_Key, SL, MTU, and HopLimit as the broadcast-GID. Other attributes SHOULD follow the broadcast-GID values as well.
When a node leaves an IP multicast group, it SHOULD make a 'FullMember' leave request to the SA to allow the SM to update forwarding tables and free resources including the MLID.
When creating an interface identifier from a GUID for IPv6 stateless autoconfiguration: if the GUID is a modified EUI-64 identifier ('u' bit toggled), the 'u' bit MUST NOT be toggled again; if it is an unmodified EUI-64, the 'u' bit MUST be toggled per RFC 3513.
For IPv4 ARP over IPoIB, the hardware type is set to 32 (decimal) as assigned by IANA. The hardware address length is 20 (octets), matching the 20-octet IPoIB link-layer address format.
For IPv6 Neighbor Discovery over IPoIB, the source/target link-layer address option uses Type 1/2, Length 3 (24 octets total), with the 20-octet IPoIB address prepended by 2 zero octets to fill the 24-octet option length.
IANA assigned hardware type number 32 (decimal) for InfiniBand in the ARP parameters registry ('Number Hardware Type' field), enabling ARP over InfiniBand. Future uses of reserved bits in the frame format and link-layer address MUST be published as RFCs.
IPoIB is vulnerable to identity spoofing and eavesdropping as with any multicast network. Higher layers or applications must implement countermeasures. SM/SA interactions for link setup MUST be protected by the OS to prevent non-privileged software from hijacking resources. Controlled Q_Keys SHOULD be used to prevent datagram fabrication.
All IP and ARP datagrams over InfiniBand are prefixed by a 4-octet encapsulation header containing a 16-bit 'Type' field (EtherType values: 0x0800=IPv4, 0x0806=ARP, 0x8035=RARP, 0x86DD=IPv6) and a 16-bit Reserved field that MUST be zero on send and ignored on receive.
The IPoIB link-layer address (hardware address) is 20 octets: 8 reserved bits (MUST be zero on send), 24-bit Queue Pair Number (QPN), and 128-bit GID. Used in both IPv4 ARP hardware address fields and IPv6 source/target link-layer address options.
The MGID (Multicast Global Identifier) for IPoIB is 128 bits: 8-bit prefix (0xFF), 4-bit flags (0001), 4-bit scope, 16-bit IPoIB signature, 16-bit P_Key, and 80-bit group ID. IPv4 multicast group IDs use only the lower 28 bits; IPv6 uses the full lower 80 bits.