How Are Large Language Models Trained? From Pretraining to RLHF

Ask 2 AI assistants the same question and you may get different tones, different refusals and different mistakes. Ask why, and the answer is almost always training: what data the model learned from, what examples it was shown and what its builders rewarded.

You do not need to train a model to benefit from understanding how it is done. Knowing the stages helps you predict where a model will be strong, why it sometimes invents facts, where bias creeps in and what vendors mean when they talk about “alignment” or “RLHF.” This guide walks through each stage in plain English.

The big picture: 3 main stages

Most modern chat models go through a pipeline like this:

Companies combine and repeat these stages in different ways, and the details are often proprietary. But this structure, described in public research such as OpenAI’s 2022 InstructGPT paper, is the common backbone.

Stage 1: Pretraining builds raw knowledge

Pretraining is where a model spends most of its computing budget. The task is deceptively simple: given a stretch of text, predict the next token. The model sees a sentence fragment, guesses what comes next, checks the actual answer and adjusts billions of internal settings, called parameters, to do slightly better. Then it repeats that across an enormous dataset.

The data typically includes filtered web pages, books, scientific papers, code, reference works and other licensed or public sources. Builders clean it by removing duplicates, spam, low-quality pages and some harmful content.

To predict text well across so many topics, the model has to absorb grammar, facts, writing styles and patterns of reasoning. That is why a pretrained model “knows” so much. But it learns from patterns, not from a verified database, which is one root of AI hallucinations.

A few things pretraining decides:

• Knowledge cutoff. The model only knows what was in its data up to a certain date.
• Language coverage. Languages with more text online tend to get better performance.
• Baseline bias. The model absorbs the viewpoints, stereotypes and gaps in its sources.

Scale matters too. DeepMind’s 2022 “Chinchilla” research found that many large models were undertrained for their size, and that model size and training data should grow together. Its 70-billion-parameter model, trained on about 4 times more data than a much larger predecessor with the same compute budget, outperformed it on a wide range of tasks. Pretraining at this scale requires vast data centers, which is why energy use has become a major issue (see AI and data center energy).

The result of pretraining is a base model. Ask it “What is the capital of France?” and it might continue with more quiz questions instead of answering, because that is what such text often looks like online.

Stage 2: Supervised fine-tuning teaches it to be an assistant

Next, the model is trained on curated examples of prompts paired with high-quality responses, often written or reviewed by people. These cover answering questions, summarizing, writing code, following formatting instructions and declining clearly harmful requests.

This stage is far smaller than pretraining, but it changes behavior dramatically. The model learns the shape of a helpful answer: address the question, follow instructions, use a consistent tone. In the InstructGPT research, OpenAI used labeler-written demonstrations for this step before moving on to preference training.

Businesses can run a version of this stage themselves to customize a model’s format or style. Our guide on when to fine-tune an LLM explains when that is worth it.

Stage 3: RLHF and preference training

Example answers only go so far. It is much easier for people to judge which of 2 answers is better than to write the perfect answer from scratch. Reinforcement learning from human feedback (RLHF) takes advantage of that:

1. The model generates several responses to the same prompt.
2. Human raters rank them from best to worst, following written guidelines.
3. Those rankings train a separate reward model that predicts which responses people would prefer.
4. The main model is then trained with reinforcement learning to produce responses that score highly with the reward model, while staying close to its earlier behavior.

The InstructGPT paper showed how powerful this is: in human evaluations, outputs from a 1.3-billion-parameter model trained this way were preferred to outputs from the 175-billion-parameter GPT-3, despite having 100 times fewer parameters. The researchers also reported improved truthfulness and less toxic output, while noting the model still made simple mistakes.

Variations have since become common:

• RLAIF and Constitutional AI. Anthropic’s 2022 Constitutional AI research used a written set of principles and AI-generated feedback, rather than only human labels, to train for harmlessness.
• Direct Preference Optimization (DPO). A 2023 method that learns directly from preference pairs, skipping the separate reward model and reinforcement learning step, which makes preference training simpler.

Key takeaway: Pretraining decides what a model knows. Fine-tuning and preference training decide how it behaves: its tone, what it refuses and what it rewards itself for. Many “personality” differences between assistants come from these later stages.

Newer stage: training models to reason

Since 2024, many leading models have added training focused on reasoning. Models practice on problems with answers that can be checked automatically, such as math problems or code that must pass tests, and are rewarded for correct results. Over time, they learn to work step by step before answering.

DeepSeek’s 2025 R1 research described this openly, reporting that large-scale reinforcement learning alone produced strong reasoning behavior, with additional stages added to fix readability problems. These “reasoning” or “thinking” models tend to do better on math, coding and multi-step analysis, but use more tokens and time per answer.

What training means for behavior and bias

Each stage leaves fingerprints you will notice in daily use:

• Confident errors. Pretraining rewards plausible text, and preference training can reward answers that sound helpful. Together, they can produce fluent answers that are wrong.
• Sycophancy. Because people tend to rate agreeable answers highly, models can learn to agree with users too readily. Ask for counterarguments when it matters.
• Inherited bias. NIST’s 2022 report on AI bias groups bias into systemic, statistical and human categories and stresses that it comes not only from data but from the broader context where AI is built and used. Web-scale data carries societal stereotypes, and rater guidelines and rater pools shape what counts as a “good” answer.
• Uneven caution. Safety training can make a model refuse harmless requests that resemble risky ones, or miss harmful ones phrased in new ways.
• Stale knowledge. Training snapshots end at a cutoff date, so models need search or document retrieval for current facts.

Openness also varies. Some developers publish detailed training reports and release model weights; others share little. Our comparison of open-source vs closed AI models explains the trade-offs.

How to use this knowledge at work

1. Check the knowledge cutoff for any model you rely on, and use tools with web search or document retrieval for current information.
2. Ask vendors about training data and safety testing, especially for customer-facing or high-stakes uses.
3. Test for bias with your own scenarios, such as hiring, lending or customer service examples, before deployment.
4. Push back on agreeable answers. Ask the model to critique its own response or argue the other side.
5. Keep people reviewing outputs that affect customers, money, health or legal matters.
6. Customize behavior with the lightest tool first: prompts and retrieval before fine-tuning.

For the fundamentals behind all of this, read what a large language model is, and for where the field may be heading, see our explainer on artificial general intelligence.

The bottom line

LLMs are built in layers. Pretraining on massive text collections gives them broad knowledge and language skill. Supervised fine-tuning turns that raw predictor into an assistant. Preference training like RLHF, and newer reasoning training, shapes tone, safety and problem-solving. Every stage also brings limits: knowledge cutoffs, inherited bias, confident errors and a pull toward pleasing the user. Understanding those stages helps you choose models wisely and use them with the right checks in place.

Sources

• arXiv: Training language models to follow instructions with human feedback (Ouyang et al., 2022) — the InstructGPT pipeline of supervised fine-tuning and RLHF, and its human evaluation results
• arXiv: Training Compute-Optimal Large Language Models (Hoffmann et al., 2022) — the Chinchilla findings on balancing model size and training data
• arXiv: Constitutional AI: Harmlessness from AI Feedback (Bai et al., 2022) — training with written principles and AI feedback
• arXiv: Direct Preference Optimization (Rafailov et al., 2023) — preference training without a separate reward model
• arXiv: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025) — reinforcement learning for reasoning
• NIST SP 1270: Towards a Standard for Identifying and Managing Bias in Artificial Intelligence — systemic, statistical and human sources of AI bias