How a Model Learns
The weights do not arrive from nowhere. They are the residue of three separate training stages, each with its own data, its own objective and its own bill — and only one of them is the part most people ever touch.
Three Stages, Not One
“Training” is one word for three different processes. They run in order, and each one assumes the last has already happened.
Pretraining
Read an enormous amount of text and predict the next token, over and over. This is where knowledge and fluency come from — and where essentially all of the compute goes.
Supervised fine-tuning
Show it examples of the behaviour you want: a question and a good answer, an instruction and a correct response. This teaches format and manner, not facts.
Preference tuning
Show it pairs of answers with a judgement about which is better, and push the model toward the preferred one. This is where helpfulness and refusals are shaped.
Pretraining: Predict the Next Token
The entire objective, for months of computation, is guessing one token at a time.
Everything left of the highlight is the input. The highlighted token is the answer. Slide one position right and you have another training example — every token in the corpus is a label for the tokens before it, which is why no human has to annotate anything.
The model outputs a probability for every token in its vocabulary, and the loss is simply how much probability it assigned to the one that actually came next. Averaged over trillions of examples, minimising that single number produces grammar, world knowledge, translation, arithmetic and code — none of which were trained for directly.
This is why capability tracks scale so closely. There is no separate lesson for any particular skill; there is one objective, and skills fall out of doing it well enough across enough text. It is also why the knowledge has a hard edge: whatever was not in the corpus, or was rare in it, the model will be shaky on, and it has no way of knowing which is which.
Fine-Tuning: Teaching the Format
The same next-token objective, on a far smaller and far more deliberate pile of text.
Behaviour, not knowledge
Fine-tuning is very good at teaching a model to answer in JSON, adopt a house style, or follow a particular workflow. It is a poor and expensive way to teach it new facts — retrieval usually wins for that, and does not go stale.
LoRA and friends
Rather than updating every weight, train a small pair of low-rank matrices alongside them and leave the original model frozen. A fraction of a percent of the parameters move, one GPU is often enough, and the result is a few megabytes you can swap in and out.
Catastrophic forgetting
Push hard on a narrow dataset and general ability degrades — the model gets better at your task and worse at everything else. Small learning rates, few epochs and a held-out general benchmark are the usual defences.
Preference Tuning: RLHF and What Came After
Some qualities are easy to recognise and hard to write down. Preference tuning optimises against a judgement instead of a target string.
Collect comparisons
Sample two or more answers to the same prompt and have someone say which is better. Not what the perfect answer was — just which of these two wins. That is a far easier question to answer consistently, and it scales.
Train a reward model
Fit a second model to predict those human judgements, so preference can be scored automatically for answers no human has ever seen. The reward model is a stand-in for the annotators, with all the blind spots that implies.
Optimise against it
Update the model to score well on the reward model, with a penalty for drifting too far from where it started. Without that penalty the model finds degenerate answers that score highly and read like nonsense — reward hacking, and the reason the constraint exists.
Direct Preference Optimization collapses steps two and three: it derives a loss that acts directly on the preference pairs, with no separate reward model and no reinforcement learning loop. It is simpler, cheaper and stable enough that it has become the default for most open models.
What Any of This Costs You
The three stages differ by orders of magnitude, which decides what is realistically available to whom.
| Stage | Data | Hardware | Realistically available to |
|---|---|---|---|
| Pretraining | Trillions of tokens, scraped and filtered | Thousands of GPUs for weeks or months | A handful of labs |
| Continued pretraining | Billions of tokens in a domain | Tens of GPUs for days | Well-funded teams with a specialised corpus |
| Supervised fine-tuning | Thousands of curated examples | One to eight GPUs for hours | Most teams |
| LoRA fine-tuning | Hundreds to thousands of examples | A single consumer GPU | Anyone with a decent card |
| Preference tuning (DPO) | Thousands of preference pairs | Similar to supervised fine-tuning | Most teams, if they can source judgements |