High Bandwidth Memory

High Bandwidth Memory (HBM) is a computer memory technology designed to overcome the bandwidth bottleneck between processors and memory in high-performance computing. Unlike conventional DRAM modules that sit beside the processor, HBM stacks multiple memory dies vertically and connects them to the processor through a wide, short interface using through-silicon vias (TSVs), achieving significantly higher data transfer rates while consuming less power per byte. HBM is the memory of choice for AI accelerators and GPUs used in training large machine learning models, including NVIDIA's data centre products and AMD's Instinct line.^[1]

History

HBM was developed jointly by AMD and SK Hynix in the early 2010s, with the first commercial product appearing in 2015 in AMD's Fiji-based Radeon R9 Fury X graphics card. The technology was standardised by the JEDEC Solid State Technology Association as JESD235, which has been revised through successive generations: HBM2 (2016), HBM2E, HBM3 (2022), and HBM3E (2023-2024). Each generation increased both capacity per stack and data rate per pin.^[1]

HBM's importance grew dramatically with the AI boom. NVIDIA adopted HBM for its data centre accelerators, from the A100 to the H100 and H200, and the 2024 Blackwell platform (B200 and GB200) uses HBM3E to feed its massive compute throughput. The market is dominated by three memory makers — SK Hynix, Samsung Electronics, and Micron — and demand for HBM has made it one of the most supply-constrained components in AI infrastructure, with SK Hynix reporting that its HBM capacity for 2025 was already sold out early in the year.^[2][3]

Key Concepts

3D stacking: HBM stacks 8, 12, or 16 DRAM dies (known as "chips") on top of one another and connects them vertically with through-silicon vias, dramatically increasing memory density in a small footprint compared with planar DRAM.

Wide interface: Instead of a narrow 64-bit channel like standard DDR memory, each HBM stack provides a very wide interface (typically 1,024 bits per stack), which delivers high total bandwidth even at moderate clock speeds.

Interposer packaging: HBM stacks sit on a silicon interposer alongside the GPU or accelerator, connected by fine metal traces. This 2.5D packaging approach is central to HBM's performance and is a key reason that advanced packaging capacity is a strategic bottleneck in AI chip production.

Bandwidth scaling: HBM3E operates at data rates around 9.6 Gbps per pin, giving each stack roughly 1.2 TB/s of bandwidth. HBM4, ramping through 2025-2026, widens the interface further and moves to a new generation of stack configurations, with both SK Hynix and Samsung pursuing the technology.^[2]

Applications and Impact

HBM is used wherever memory bandwidth limits computation, most prominently in AI training and inference accelerators. NVIDIA's H100, H200, and Blackwell-class GPUs, AMD's MI300 series, and many custom accelerators depend on HBM to feed weights and activations to compute cores quickly enough to keep them utilised. It also appears in supercomputers (such as Fugaku, which uses HBM2E) and high-end networking and scientific computing systems.

Because AI model sizes and training runs grew faster than conventional memory bandwidth, HBM became a strategic commodity: memory makers invested billions in new fab and packaging capacity, and national governments in the US, Japan, and South Korea designated HBM production as critical to semiconductor supply chains. The tight supply of HBM and advanced packaging has shaped the roadmap of every major AI chip vendor.^[1][3]

>See Also

• NVIDIA
• NVIDIA Blackwell
• AMD
• Tensor Processing Unit
• National Semiconductor Strategy Malaysia
• AI Data Centres Malaysia

References