0 Reasoning Models

A difficult problem may require several intermediate operations: deriving a relation, testing a candidate, finding an inconsistency, and trying another approach. This book studies language models trained to perform such work, and the systems that allocate and evaluate their computation.

The starting point is a distinction between what a model has learned and how much computation a particular answer receives. Both affect the result.

Training the behavior

Training changes the parameters $\theta$. Supervised examples can demonstrate useful solution patterns; reinforcement learning can reward outcomes under a chosen scoring rule.

The original DeepSeek-R1 report provides a concrete case study. Its R1-Zero experiment applied reinforcement learning directly to a pretrained base model without an initial supervised fine-tuning stage. The reported R1 pipeline added cold-start supervised data and further training stages. These are distinct training procedures; neither starts from an untrained model.

The reward matters. A mathematical answer checker and executable programming tests offer different signals from a preference judgment. Understanding a result means understanding what was rewarded, what was measured, and where those two criteria may differ.

Allocating computation at inference

With parameters fixed, a system can spend additional computation generating intermediate tokens, exploring several candidates, or checking an answer. These are different allocation strategies with different costs.

One simplified description separates intermediate tokens $z$ from a final answer $y$:

$$p_\theta(z,y\mid x) =p_\theta(z\mid x)\,p_\theta(y\mid x,z). \tag{0.1}$$(0.1)

This factorization describes a possible generation procedure. It does not establish that generated text faithfully records every internal computation.

A token budget $B$ places an upper bound on the generated sequence. Its accounting convention must be explicit: does $B$ include intermediate tokens, the final answer, or both? The same allowance can produce different latency and outcomes across models and tasks. A larger allowance creates an opportunity for more work; whether that work helps is an experimental question.

Evaluate an outcome

For a programming task, define success using held-out tests and a clear execution environment. For a derivation, check the assumptions and individual mathematical steps. Preserve unsuccessful runs as well as successful examples.

A useful comparison records:

• The exact task set and completion criterion.
• The model checkpoint, prompt, and sampling settings.
• The generation budget and number of attempts.
• Correctness, latency, and consumption per attempted task.

Keep these conditions fixed when studying one intervention. Otherwise, an apparent model improvement may come from a different budget, selection rule, or verifier.

Where this book begins

The foundations are training and inference, followed by careful experiments on computation and verification. Evaluating bounded agents offers a related worked example of budget accounting. The central question throughout is concrete: which additional computation improves which measurable outcome?