3 Training and inference

Training changes model parameters. Inference uses those parameters to compute outputs. Both may use the same transformer operations, but they have different objectives and execution patterns.

Learning a conditional distribution

For a token sequence $x_1,\ldots,x_T$, a common autoregressive training objective is the negative log-likelihood:

$$\mathcal{L}(\theta) =-\frac{1}{T}\sum_{t=1}^{T} \log p_\theta(x_t\mid x_1,\ldots,x_{t-1}). \tag{3.1}$$(3.1)

This expression describes one sequence with a simple normalization. Real training pipelines combine sequences, mask selected tokens, and choose how to weight the remaining losses. Those choices are part of the objective, not incidental bookkeeping.

Training typically presents the model with the correct previous tokens. A causal attention mask allows many next-token losses to be computed in parallel while preventing access to future tokens. Optimization then uses gradients of the loss to update the parameters.

Generating from a fixed model

During autoregressive generation, the model computes a distribution for the next token, a decoding rule chooses a token, and the chosen token joins the context. Later steps depend on earlier choices.

The model distribution and the decoding rule are different objects. Greedy decoding selects the largest logit. Sampling draws from a distribution, possibly modified by temperature or truncation. Changing decoding can change outputs without changing a single model parameter.

Prefill and decoding

A useful systems distinction is between processing the supplied prompt and generating subsequent tokens. Earlier keys and values can be cached so a decoding step does not recompute the entire prefix. The cache consumes memory, and its size depends on the architecture, context length, batch, and numerical representation.

Caching is an execution optimization. It does not teach the model a new fact or permanently update its parameters.

Post-training changes the target behavior

Pretraining usually builds broad predictive capability from a large corpus. Supervised fine-tuning can train on desired input–output examples. Preference-based methods use additional signals about which outputs are preferred. These stages change parameters; putting an instruction in a prompt changes the current input instead.

This vocabulary helps frame an engineering decision: does a problem require better information in context, a better decoding policy, a different trained behavior, or a more reliable surrounding agent system? Each intervention acts at a different layer.

Questions to carry forward

• Which tokens contribute to the loss, and with what weights?
• Which computations can happen in parallel?
• Which intermediate values can be reused?
• Is quality being measured on held-out tasks, or only on the training objective?
• Is a reported improvement due to the model, the prompt, decoding, or the surrounding system?

These questions are often more useful than beginning with a particular optimization technique.