AI Accelerator

AI accelerator is a class of specialised semiconductor hardware designed to execute artificial intelligence workloads — chiefly the matrix and vector arithmetic behind neural network training and inference — at higher throughput and lower energy per operation than a general-purpose CPU. Market research defines the category as chips delivering "performance-per-watt efficiencies exceeding general-purpose processors" for training, inference and autonomous decision-making workloads.^[1]

Background

General-purpose CPUs proved too inefficient for deep learning as neural networks scaled in the 2010s. GPUs, originally built for graphics rendering, were repurposed for parallel matrix maths and became the default training hardware after CUDA (introduced by NVIDIA in 2006–2007) gave developers a programming model for them. The landmark AlexNet result in 2012, which trained on two consumer GPUs, is commonly cited as the moment the deep learning boom adopted GPU compute.^[3]

Hyperscalers then began designing application-specific integrated circuits (ASICs) for their own workloads: Google has developed seven generations of Tensor Processing Units with Broadcom since 2014, AWS deploys Trainium and Inferentia inside its cloud, and Microsoft Maia and Meta MTIA round out the custom-silicon field.^[1] A separate wave of inference-focused chips — Groq's Language Processing Unit, Cerebras' wafer-scale WSE-3, SambaNova and Tenstorrent — targets low-latency token generation rather than large-scale training.^[3]

Key Concepts

• GPU — massively parallel throughput; NVIDIA's Blackwell generation (B200/B300, NVL72 racks) launched on TSMC 3nm with NVLink 6 interconnect, and NVIDIA's data-centre revenue compounded from US$47.5 billion to US$193.7 billion in two years.^[1]
• Software moat — NVIDIA's CUDA ecosystem (4+ million developers, 40,000+ companies, 3,000+ accelerated applications) creates switching costs that hardware benchmarks alone do not remove.^[1][2]
• ASIC — workload-specific silicon optimised for a fixed pattern of operations; the fastest-growing chip segment at roughly 43% projected CAGR through 2035, with ASIC-based AI server shipments reaching 27.8% of the market in 2026.^[1]
• Training vs inference split — inference is on track to represent about two-thirds of all AI accelerator spending, shifting vendor roadmaps from peak FLOPs toward tokens per second per watt.^[1]
• Interconnect and memory — NVLink, high-bandwidth memory and on-chip SRAM often decide real performance: Cerebras' WSE-3 keeps 44 GB of models in SRAM at 21 PB/s bandwidth, while Groq's deterministic architecture uses roughly 230 MB of SRAM per chip.^[3]

Published market-share estimates for NVIDIA in 2025 range from about 70% (TrendForce) to 77% (Morgan Stanley, wafer consumption) to 86% (broader sector methodologies); Gartner put NVIDIA above 50% of AI-processing semiconductors in 2024. The spread reflects that "AI chip" definitions differ — merchant GPUs only, or also TPUs, Trainium and internal silicon.^[2]

Applications and Impact

Accelerators underpin every frontier model training run, every hosted LLM API and most enterprise AI deployment. Demand is concentrated in data-centre capex: AWS deployed 500,000 Trainium2 chips for Anthropic at its Indiana site, and OpenAI signed a 10 GW custom accelerator agreement in October 2025 with first deployments in 2026.^[1] At the edge, neural processing units (NPUs) in PCs and smartphones run smaller models on-device, and MLPerf remains the main cross-vendor reference benchmark for comparing accelerators under standardised conditions.^[3]

>See Also

• NVIDIA
• Google TPU
• Cerebras Systems
• Groq
• AI Data Centres Malaysia
• Penang Semiconductor and AI Ecosystem