InfiniBand

InfiniBand is a switched-fabric communication standard for linking computers and accelerators within a data centre, offering very high bandwidth and extremely low latency compared with conventional Ethernet. Originally built for supercomputing, it became the interconnect of choice for large AI training clusters, where thousands of GPUs must exchange gradient updates continuously across the network.^[1]

History

InfiniBand was introduced in 1999 as a merger of competing server I/O architectures and has been maintained since by the InfiniBand Trade Association. It found its lasting home in high-performance computing: for much of the 2000s and 2010s it linked the world's fastest supercomputers.^[1]

The deep-learning era made it mainstream infrastructure. When training runs expanded from single machines to thousands of GPUs, the network between GPUs became a bottleneck; NVIDIA's acquisition of Mellanox Technologies in 2020 gave it control of the leading InfiniBand vendor and, crucially, the ability to tie GPUs and networking into one optimised stack. NVIDIA's current generation, Quantum-X800 InfiniBand, delivers 800 Gb/s per port and pairs with the NVLink scale-up fabric inside a single server.^[2]

Key Concepts and Technology

Two properties distinguish InfiniBand for AI. First, remote direct memory access (RDMA) lets one GPU write into another GPU's memory without involving either node's CPU or operating system, cutting latency to microseconds. Second, in-network computing — NVIDIA's SHARP architecture — performs reductions (such as averaging gradients during all-reduce operations) inside the switches themselves, so the network does part of the maths instead of just carrying it.^[2]

InfiniBand also supports adaptive routing and self-healing fabrics that recover from link failures far faster than traditional networks — important when a single training job spans thousands of links, any one of which can fail.^[2] Its main competitor is Ethernet: NVIDIA's Spectrum-X platform applies congestion control and deterministic latency techniques to make Ethernet behave more like InfiniBand for AI clouds, and some hyperscalers have preferred that open-standard route.^[3]

Applications and Impact

InfiniBand is deployed in national supercomputers, cloud AI factories and enterprise AI clusters — any setting where distributed training or tightly coupled multi-GPU inference dominates. It remains the reference fabric against which AI networking performance is measured, and its economics (switches, cables and adapters can represent a significant share of cluster cost) are a central consideration in data centre design.^[2][3]

>See Also

• NVIDIA
• GPU Cluster
• NVLink
• AI Data Centres Malaysia
• High-Bandwidth Memory
• Liquid Cooling