Pre-training

Pre-training is the first and most computationally intensive phase of training a large language model (LLM), in which a neural network learns general patterns, factual knowledge, and linguistic structure from massive quantities of unlabelled text before being specialised for specific tasks through subsequent phases such as fine-tuning and reinforcement learning from human feedback (RLHF). Pre-training produces what is commonly called a base model or foundation model — a general-purpose system that can predict or generate text but has not yet been optimised to follow instructions or behave as a useful assistant. This phase is responsible for the vast majority of the compute, data, and cost that goes into building a frontier AI system, and it is the stage at which a model acquires its broad knowledge of language and the world.^[1]

How Pre-training Works

Pre-training uses self-supervised learning, a technique in which the training signal is derived from the data itself rather than from human labels. The most common objective for modern LLMs is next-token prediction: the model is given a sequence of tokens and must predict the token that follows. By training on trillions of tokens of text — web pages, books, code, scientific papers, and dialogue — the model learns grammar, factual associations, reasoning patterns, and stylistic conventions.

The training loop proceeds as follows: the model is shown a batch of text sequences, it makes predictions for the next token at each position, the predictions are compared against the actual next tokens using a loss function (typically cross-entropy loss), and the model's parameters are updated via gradient descent and backpropagation to reduce the prediction error. This process is repeated billions of times over the entire corpus, often for multiple passes (epochs).

Scale and Compute

Pre-training is where the bulk of computational resources are consumed. The training of a frontier model in 2025-2026 typically involves:

• Trillions of tokens of training data (e.g., 10-15 trillion tokens for models like Llama 3 and Grok 3)
• Tens of thousands of GPUs running for weeks or months (e.g., clusters of 16,000 to 200,000 NVIDIA H100 or H200 GPUs)
• Hundreds of millions to billions of dollars in compute, energy, and hardware costs

The relationship between pre-training compute and model quality is captured by scaling laws, which demonstrate predictable improvements in model performance as a function of increased compute, data, and model parameters. These empirical findings, originally articulated by researchers at OpenAI and DeepMind, have driven the industry's sustained investment in ever-larger pre-training runs.^[2]

The Training Pipeline

Pre-training is the first of several stages in a modern model development pipeline:

Pre-training: Produces a base model from large-scale self-supervised learning.

Supervised fine-tuning (SFT): The base model is fine-tuned on curated examples of instruction-response pairs to teach it to follow instructions and behave as an assistant.

Reinforcement learning from human feedback (RLHF): The model is further refined using human preference signals to improve the quality and safety of its outputs.

Instruction tuning and alignment: Additional targeted training to improve specific capabilities (coding, mathematics, multilingual performance) and to enforce safety guidelines.

The base model produced by pre-training alone is typically not released to consumers — it is an intermediate artefact. What users interact with (ChatGPT, Claude, Gemini) are the post-trained versions that have undergone fine-tuning and alignment.

Data Composition and Curation

The composition of the pre-training corpus significantly shapes the model's capabilities and biases. Major components typically include:

• Web crawls: Broad internet text (e.g., Common Crawl, RefinedWeb)
• Books and literature: Long-form structured text
• Code repositories: Source code from GitHub and similar platforms, which improves reasoning and programming capabilities
• Scientific papers: Academic publications (e.g., arXiv, PubMed)
• Multilingual data: Text in non-English languages for global coverage
• Synthetic data: Text generated by other AI models, increasingly used to augment scarce or high-quality human data

Data quality and deduplication are critical: poorly filtered training data can introduce factual errors, toxic content, or copyrighted material into the model's knowledge. Leading labs invest heavily in data curation pipelines that filter, deduplicate, and quality-score training data before pre-training begins.^[3]

Challenges and Frontiers

Data exhaustion: Some researchers project that the pool of high-quality human-generated text suitable for pre-training may be largely consumed by the late 2020s, driving interest in synthetic data generation and curriculum learning.

Compute costs: The exponential growth in pre-training budgets has raised concerns about the concentration of AI capabilities in a small number of organisations able to afford the necessary compute.

Reproducibility: Pre-training runs are extremely difficult to reproduce exactly due to hardware nondeterminism, random initialisation, and the sheer scale of the process, complicating scientific study.

>See Also

• Fine-tuning
• RLHF
• Transformer Architecture
• Foundation Models
• AI Data Centres Malaysia

References