llama.cpp

llama.cpp is an open-source software library, written in C and C++, that performs inference on large language models such as Meta's Llama family. Created by software engineer Georgi Gerganov in March 2023, it was designed to run models efficiently on ordinary computers without specialised hardware, and it has become the de facto core of most local inference tools, including Ollama and LM Studio.^[1][2]

Background

The project grew out of the GGML library, a C library for tensor algebra that Gerganov began in late 2022 with an emphasis on strict memory management and multithreading, itself inspired by Fabrice Bellard's work. Gerganov had earlier written whisper.cpp, a lightweight implementation of OpenAI's Whisper speech model, and applied the same philosophy to large language models: a dependency-free implementation that could run on a MacBook or desktop CPU.^[3][4]

The first version of llama.cpp was released on 10 March 2023, days after Meta's Llama weights became publicly available. It gained rapid traction among users who lacked GPUs, then expanded with acceleration backends for Apple Silicon, CUDA, Vulkan, ROCm and other platforms. By May 2026 the repository had accumulated more than 109,000 stars on GitHub, and later milestones included FlashAttention support (2024), renewed multimodal support through the libmtmd library (2025) and full acceleration on Android and ChromeOS devices (December 2025).^[1][2]

Key Concepts and Technology

llama.cpp is built around GGML, the tensor library that supplies its compute kernels, with a front-end that handles model loading, decoding and sampling. It supports a wide range of hardware targets — x86 and ARM CPUs, Apple Metal, NVIDIA CUDA, AMD HIP, SYCL, Vulkan and others — and uses CPU instruction sets such as AVX2, AVX-512 and ARM Neon for speed. Apple Silicon is a particularly important target.^[1][4]

Its efficiency comes largely from quantisation: models are compressed to lower-precision weights (typically four to eight bits) so that multi-billion-parameter models fit in modest amounts of memory, with further savings from KV-cache quantisation and the ability to offload part of a model to system RAM. Other features include speculative decoding, and bundled command-line tools plus a server that exposes an OpenAI-compatible HTTP interface, letting existing applications talk to locally hosted models.^[1][2]

Applications and Impact

Because it runs well on commodity hardware, llama.cpp powers a large ecosystem of local AI software — Ollama, LM Studio, GPT4All and others — and is widely used for private, offline and low-cost inference. Organisations run it on laptops, workstations and on-premises servers where data cannot leave the premises, while developers use it to test open-weight models on phones and single-board computers. Its rise paralleled the growth of the open-weight model movement, in which models such as Llama, Qwen, Mistral and DeepSeek are distributed for local use.^[1][2][5]

>See Also

• GGUF
• Ollama
• Quantisation
• Edge AI
• Small Language Models
• Llama