GGUF

GGUF (GPT-Generated Unified Format) is a binary file format for storing and distributing large language models for local inference. A GGUF file is self-contained: it packs a model's weights, its tokenizer and the metadata needed to run it into a single file that an inference engine can load quickly. It is the native format of llama.cpp and is used by most tools built on top of it, making it the de facto standard for running open-weight models on consumer hardware.^[1][2]

Background

GGUF succeeded the earlier GGML format, which had packaged model weights as raw tensors without a stable way to describe structure. As llama.cpp added support for model architectures beyond Meta's Llama, the limitations of that format became a problem: changes often broke compatibility with existing models, and each new feature required manual adjustments by users.^[3][4]

The llama.cpp project introduced GGUF on 22 August 2023 to solve this. The new format stored not just tensors but also a standardised set of key-value metadata and the tokenizer, and was designed so that new capabilities could be added without invalidating older files. Hugging Face added built-in GGUF support to its model hub shortly afterwards, and the format spread with the fast-growing ecosystem of local inference tools around llama.cpp.^[1][3]

Key Concepts and Technology

A GGUF file has a simple internal layout: a header, a block of key-value metadata, an index of tensors, and then the tensor data itself. The metadata carries everything a runtime needs to interpret the model — architecture type, context length, tokenizer vocabulary and rules, and prompt templates — so nothing needs to be reassembled from separate files. Because the tensors are laid out predictably, engines can load a model by memory-mapping the file, which keeps start-up fast and memory overhead low.^[1][2][5]

GGUF is closely tied to quantisation. Quantised weights are stored directly in the file, with a naming scheme that encodes the scheme used — for example Q4_K_M denotes a four-bit "k-quant" variant balancing size and accuracy, while the IQ series offers aggressive low-bit compression. The same model is typically published at several quantisation levels, from small files for laptops to larger, higher-fidelity versions for servers, and the format is cross-platform: one file runs on Windows, macOS and Linux, on CPU or GPU. Conversion tools such as llama-quantize and community conversion services on Hugging Face produce GGUF files from models originally trained in frameworks such as PyTorch.^[1][5][6]

Applications and Impact

GGUF is the distribution format for the local AI ecosystem. Model publishers release GGUF versions of popular open-weight families — Llama, Qwen, Mistral, Gemma, DeepSeek and others — and applications such as Ollama, LM Studio and GPT4All consume them directly. The format is a complement to safetensors: where safetensors is the standard for training and full-precision weights, GGUF is engineered for inference, packaging models at sizes that fit on phones, laptops and single-GPU workstations. For organisations, this enables offline and privacy-preserving deployments, and for students and hobbyists it lowered the cost of experimenting with language models to a download.^[2][3][5]

>See Also

• llama.cpp
• Quantisation
• Ollama
• Hugging Face
• Small Language Models
• Edge AI