Quantization

Quantization in machine learning is a model-compression technique in which the high-precision numbers that make up a neural network — typically 32-bit or 16-bit floating-point values — are converted to lower-precision representations such as 8-bit integers (INT8) or 4-bit formats. Because the majority of a neural network's memory footprint is its weights, quantized models require substantially less RAM and storage, load faster, and often run several times faster on both server and consumer hardware, at the cost of a small, usually manageable reduction in output quality.^[1]

Quantization has become one of the most important techniques for deploying large language models (LLMs) in practice. It is the reason that models such as Llama, Qwen and DeepSeek can run on a laptop, a phone or a modestly priced graphics card, and it underpins the local-inference tools that many organisations use to keep data on their own premises.

History and Background

Neural-network quantization has been studied since the 1980s and gained prominence in the deep-learning era through work on binary and ternary networks and on compression-aware training such as the "Deep Compression" line of research in the mid-2010s, which combined pruning, quantization and Huffman coding to shrink large vision models. These early techniques showed that convolutional networks could tolerate aggressive precision reduction with limited accuracy loss.^[2]

The arrival of very large language models after 2020 reframed quantization as an inference-economics problem: a 70-billion-parameter model in 16-bit precision needs about 140 GB of memory, beyond the reach of most hardware, while the same model in 4-bit needs roughly 35 GB. Landmark developments include the LLM.int8() method (2022), which showed that outlier activations were the main obstacle to 8-bit LLM inference; GPTQ, a post-training method for compressing models to 4-bit or 3-bit with minimal calibration data; and QLoRA (2023), which combined 4-bit NF4 quantization with low-rank adapters to make fine-tuning of large models possible on a single consumer GPU. Around the same time, the llama.cpp project and the GGUF format popularised quantized local inference across CPU and GPU hardware, and AWQ and SmoothQuant improved activation-friendly quantization for serving systems.^[3]

Key Concepts

Quantization maps a range of continuous values into a small set of discrete levels. In weight quantization, each parameter is stored in a low-precision format, most commonly INT8 or INT4; in activation quantization, the intermediate values flowing between layers are also reduced, which is necessary for real speed-ups on hardware such as GPUs and neural processing units. A quantized model is defined by its precision, its quantization range (symmetric or asymmetric), and its granularity — per-tensor, per-channel or per-group — with finer granularity generally preserving more accuracy.

Two families of methods exist. Post-training quantization (PTQ) converts an already-trained model using a small calibration dataset and is fast and cheap, which is why most open-source "quant" downloads use it. Quantization-aware training (QAT) simulates quantization during training or fine-tuning so the model learns to tolerate low precision, producing higher quality at very low bit-widths but requiring compute. For LLMs, additional refinements include KV-cache quantization, which compresses the attention cache during long conversations, and block-wise formats such as NF4 that improve 4-bit accuracy. Modern accelerators increasingly ship native low-precision support — FP8 on recent data-centre GPUs and INT4/FP4 matrix engines — so quantized models also run faster, not merely smaller.^[1][2]

Applications and Impact

Quantization underpins the modern local AI ecosystem: chat assistants, code helpers and document tools distributed through formats such as GGUF run quantized models on laptops, mini-PCs and phones, giving individuals and small businesses private AI without cloud subscriptions. In data centres, providers use FP8 and INT8 quantized models in serving engines such as vLLM and TensorRT-LLM to roughly double throughput per accelerator and cut energy costs, and edge devices from cameras to factory sensors rely on quantized models to classify and detect in real time within tight power budgets.^[3]

The technique is also a training enabler: QLoRA-style 4-bit fine-tuning allows domain adaptation of large open models on a single workstation, and QAT is standard practice before models are deployed to fixed-function hardware such as NPUs in phones and cars. Because quantized models are approximations, evaluation is required — quality drift, hallucination rates and numerical edge cases must be tested for the target task before production use.^[1]

>See Also

• Fine-Tuning
• Mixture of Experts
• Transformer Architecture
• MALAM (Malaysian LLM)
• ILMU (Malaysian LLM)
• Ollama