Pretraining, Continual Training, SFT, DPO, and Alignment Methods for Large Language Models

The Multi-Stage Architecture of Modern LLM Training

Training a modern large language model (LLM) is no longer a single optimization run. Instead, it proceeds through a sequence of increasingly specialized stages, each with its own mathematical objective, data regime, and hardware footprint.

At the base of the stack lies autoregressive pretraining, where a randomly initialized network learns to predict the next token across trillions of tokens of text, code, and scientific literature. Next comes continual (or domain-adaptive) pretraining, which shifts the model toward specialized fields such as law, medicine, or underrepresented languages without erasing its general capabilities. Supervised fine-tuning (SFT) then reshapes the raw text completer into an assistant that follows instructions and formats responses cleanly. Finally, preference alignment algorithms such as RLHF, DPO, IPO, KTO, ORPO, and SPIN calibrate the model's outputs to reflect human or synthetic judgments of helpfulness, truthfulness, and safety.

This article walks through every stage of that pipeline in enough mathematical and engineering detail to implement each method, while highlighting the practical trade-offs that determine which algorithm works best under real hardware and data constraints.

Autoregressive Pretraining

The Next-Token Prediction Objective

Pretraining teaches a model the statistical and semantic structure of language, code, and symbolic reasoning. Given a sequence of tokens $x = (x_1, x_2, \dots, x_T)$, a model parameterized by $\theta$ is trained to predict the conditional probability distribution over each subsequent token $x_t$ given its preceding context.

Formally, the pretraining objective minimizes the expected negative log-likelihood across a massive corpus $\mathcal{D}$:

L_pretrain(θ) = -𝔼_{x∼D} [ Σ_{t=1}^{T} log p_θ(xₜ | x₁, ..., x_{t-1}) ]

At every position $t$, this objective reduces to the standard cross-entropy loss between the model's predicted softmax distribution over the vocabulary and the one-hot target token $x_t$.

Transformer Decoder Architecture and Scaling

Almost all contemporary frontier models build on the autoregressive Transformer decoder (Vaswani et al., 2017), refined by three standard architectural modifications:

• Rotary Position Embeddings (RoPE) (Su et al., 2024), which apply rotation matrices to query and key representations in complex space so attention scores depend on relative token distance, improving long-context extrapolation.
• Grouped-Query Attention (GQA) (Ainslie et al., 2023), which shares key and value projections across groups of query heads, drastically shrinking the key-value (KV) cache footprint during high-throughput inference.
• SwiGLU feed-forward networks (Shazeer, 2020), which replace standard ReLU layers with gated linear units using the Swish activation, improving representation capacity per parameter.

Model scale has expanded from 175 billion parameters in GPT-3 (Brown et al., 2020) to 405 billion in dense Llama 3 (Meta AI, 2024) and well beyond one trillion total parameters in sparse Mixture-of-Experts (MoE) architectures. Training at this scale requires thousands of accelerators coordinated through 3D parallelism (combining data parallelism with ZeRO sharding, tensor parallelism within nodes, and pipeline parallelism across nodes).

Corpus Engineering and Scaling Laws

Frontier pretraining runs consume between 1 and 15 trillion tokens drawn from deduplicated web crawls (Common Crawl, FineWeb), source code repositories (GitHub, The Stack), books, academic preprints (arXiv, PubMed), encyclopedic references, and curated synthetic datasets. Mix proportions matter enormously: incluir substantial proportions of source code and mathematical text noticeably improves multi-step reasoning even on non-technical tasks.

The Chinchilla scaling laws (Hoffmann et al., 2022) demonstrated that for a fixed compute budget, model parameter count $N$ and training token volume $D$ should scale roughly in proportion ($D \approx 20 N$). In practice, models intended for widespread deployment are frequently "over-trained" far beyond the Chinchilla compute-optimal point (for instance, training an 8-billion-parameter model on 15 trillion tokens) so that the resulting model delivers maximum capability at a much lower inference cost.

Optimization Mechanics and Base Model Behavior

Pretraining typically uses the AdamW optimizer (Loshchilov & Hutter, 2019) with decoupled weight decay, momentum parameters $\beta_1 = 0.9$ and $\beta_2 = 0.95$, gradient norm clipping at $1.0$, and mixed-precision arithmetic (BF16 or FP8). The learning rate warms up linearly over the first few thousand steps to a peak between $1 \times 10^{-4}$ and $3 \times 10^{-4}$ (scaled inversely with model width) before decaying along a cosine or Warmup-Stable-Decay (WSD) schedule.

The resulting artifact is a base model: an extraordinarily capable sequence completer that encodes broad world knowledge, but one that has no intrinsic bias toward answering a user's question rather than replying with three more questions in the style of an internet forum.

Continual and Domain-Adaptive Pretraining (CPT)

Injecting Domain Knowledge Without Curated Prompts

Continual pretraining (CPT), also known as domain-adaptive pretraining (DAPT), resumes the self-supervised next-token prediction objective on a target corpus $\mathcal{D}_{\text{domain}}$ whose distribution differs substantially from the base pretraining mix (Ke et al., 2025). Typical applications include adapting a general model to specialized legal statutes, clinical medical notes, financial filings, or low-resource natural languages.

Unlike supervised fine-tuning, CPT requires no human-annotated instruction pairs. Because raw domain text is usually orders of magnitude more abundant than curated Q&A datasets, CPT is the primary mechanism for teaching a model large volumes of new factual or linguistic structure.

Mitigating Catastrophic Forgetting

The central hazard of continual pretraining is catastrophic forgetting: gradient updates driven entirely by $\mathcal{D}_{\text{domain}}$ rapidly overwrite weights responsible for general reasoning and instruction following. Four practical techniques keep forgetting under control:

1. Experience replay (data mixing): Blending 5 to 15 percent general pretraining text (including code and math) into the domain corpus anchors the model's general representations (Scialom et al., 2022).
2. Conservative learning-rate schedules: Re-warming to a much lower peak learning rate (typically 10 to 30 percent of the original pretraining peak, such as $2 \times 10^{-5}$) prevents destructive weight drift during early steps.
3. Parameter regularization: Techniques such as Elastic Weight Consolidation (EWC) penalize updates to parameters with high Fisher information on the base corpus, though replay mixing is usually favored at scale for simplicity.
4. Adapter-constrained CPT: Training high-rank LoRA adapters (coupled with unfrozen embedding and language-modeling heads) isolates domain shifts while keeping the base weights intact (Hu et al., 2021).

A representative legal CPT pipeline, such as the one evaluated by Chen et al. (2025), gathers tens of billions of tokens of statutory and case-law text, blends in a 10 percent general replay buffer, trains with AdamW at $2 \times 10^{-5}$ under cosine decay, and tracks both domain benchmarks and general reasoning suites (MMLU, GSM8K) before handing the checkpoint off to instruction tuning.

Supervised Fine-Tuning (SFT)

Masked Next-Token Prediction on Demonstrations

Supervised fine-tuning (SFT), or instruction tuning (Wei et al., 2022), bridges the gap between raw text completion and interactive assistance. Given a dataset $\mathcal{D}_{\text{SFT}} = {(x_i, y_i)}$ of prompts $x_i$ (including system instructions and multi-turn conversation history) paired with target responses $y_i$, SFT minimizes the conditional negative log-likelihood of the response tokens:

L_SFT(θ) = -𝔼_{(x,y)∼D_SFT} [ Σ_{t=1}^{|y|} log p_θ(yₜ | x, y₁, ..., y_{t-1}) ]

Unlike pretraining, prompt tokens $x$ are masked out of the loss calculation. The model still attends to the entire prompt during the forward pass, but gradients are computed solely on the assistant's completion $y$.

Dataset Curation and Sequence Packing

Whereas pretraining relies on trillions of tokens, SFT is remarkably sample-efficient: as few as 10,000 to 100,000 carefully curated examples suffice to transform a base model's behavior. InstructGPT (Ouyang et al., 2022) used roughly 13,000 human-written demonstrations; subsequent open-weights recipes combine expert human annotations, synthetic distillation from frontier models, and self-instruct bootstrapping filtered by automated verifiers or unit tests.

In practice, SFT runs for 2 to 4 epochs with learning rates between $1 \times 10^{-5}$ and $5 \times 10^{-5}$. To avoid wasting GPU compute on padding tokens when example lengths vary widely, modern trainers concatenate multiple examples into a single context window (sequence packing) while applying block-diagonal attention masks so tokens from one conversation cannot attend to another.

Why SFT Alone Is Not Enough

Although an SFT checkpoint follows instructions fluently, maximum-likelihood training on static demonstrations exhibits four well-known failure modes:

• Exposure bias: During training, the model always conditions on ground-truth prefix tokens (teacher forcing); during inference, it conditions on its own generated tokens, allowing small errors to compound.
• Sycophancy: Because human demonstrations rarely contradict the user's premises, SFT models tend to agree with false presuppositions.
• Verbosity bias: Annotators often write comprehensive, hedged answers, teaching the model to default to long-winded prose.
• Absence of negative feedback: Cross-entropy pushes up the probability of the reference phrasing $y$, but provides no explicit signal about which alternative completions are harmless paraphrases versus dangerous hallucinations.

Reinforcement Learning from Human Feedback (RLHF)

From Demonstrations to Relative Preferences

To overcome the limits of teacher forcing, Christiano et al. (2017) and Ouyang et al. (2022) introduced Reinforcement Learning from Human Feedback (RLHF). Its central insight is that evaluating candidate outputs is far easier than writing an ideal output from scratch, and that pairwise comparisons teach a model explicit boundaries between desirable and undesirable behavior.

Reward Modeling Under the Bradley-Terry Framework

First, given a prompt $x$, the SFT model samples multiple candidate responses $(y_1, y_2)$. Human raters (or AI judges) indicate which response is better, yielding a dataset of preference triples $\mathcal{D}_{\text{pref}} = {(x, y_w, y_l)}$, where $y_w \succ y_l$ denotes that the winning completion $y_w$ is preferred over the losing completion $y_l$.

Second, a reward model $r_\phi(x, y)$ is initialized from the SFT checkpoint by replacing the vocabulary projection head with a linear layer that outputs a single scalar score. Under the Bradley-Terry pairwise comparison model (Bradley & Terry, 1952), the probability that $y_w$ is preferred over $y_l$ is modeled as the logistic sigmoid $\sigma$ of their reward difference:

P(y_w ≻ y_l | x) = σ(r_φ(x, y_w) - r_φ(x, y_l))

The reward model parameters $\phi$ are trained by minimizing the negative log-likelihood of the observed preference rankings across $\mathcal{D}_{\text{pref}}$:

L_RM(φ) = -𝔼_{(x,y_w,y_l)∼D_pref} [ log σ(r_φ(x, y_w) - r_φ(x, y_l)) ]

Policy Optimization with PPO and KL Regularization

In the third stage, the language model acts as an autoregressive policy $\pi_\theta(y \mid x)$ optimized via Proximal Policy Optimization (PPO) (Schulman et al., 2017) to maximize expected reward while remaining close to the frozen reference model $\pi_{\text{ref}}$ (typically the SFT checkpoint):

max_π 𝔼_{x∼D, y∼π(·|x)} [ r_φ(x, y) ] - β · KL(π(·|x) || π_ref(·|x))

The reverse Kullback-Leibler (KL) penalty weighted by $\beta$ is essential: without it, the policy quickly discovers adversarial token sequences that exploit blind spots in the imperfect reward model (reward hacking or Goodhart's law), collapsing into repetitive, high-scoring gibberish.

Despite its strong empirical ceiling, PPO-based RLHF is notoriously complex to engineer. It requires keeping four large models in distributed GPU memory simultaneously (the active policy $\pi_\theta$, the reference policy $\pi_{\text{ref}}$, the reward model $r_\phi$, and a learned value critic $V_\psi$), suffers from high variance and hyperparameter sensitivity, and spends most of its wall-clock time waiting on autoregressive rollout generation.

Direct Preference Optimization (DPO)

Deriving the Closed-Form Implicit Reward

In 2023, Rafailov et al. showed that the KL-constrained reward maximization objective used in RLHF admits a closed-form analytical solution, making it possible to align the policy directly on preference pairs without training a separate reward model or running reinforcement learning.

Specifically, for any reward function $r(x, y)$, the unique optimal policy $\pi^*$ that maximizes $\mathbb{E}[r(x,y)] - \beta,\text{KL}(\pi \parallel \pi_{\text{ref}})$ has the Boltzmann form:

π*(y|x) = (1/Z(x)) · π_ref(y|x) · exp(r(x,y) / β)

where $Z(x) = \sum_y \pi_{\text{ref}}(y \mid x)\exp(r(x,y)/\beta)$ is the intractable partition function over all possible token sequences. Solving this equation algebraically for the reward function $r(x, y)$ gives:

r(x,y) = β · log(π*(y|x) / π_ref(y|x)) + β · log Z(x)

When this expression is substituted into the Bradley-Terry preference model $P(y_w \succ y_l \mid x) = \sigma(r(x, y_w) - r(x, y_l))$, the prompt-dependent partition function $\beta \log Z(x)$ appears in both terms and cancels out completely:

P(y_w ≻ y_l | x) = σ( β · log(π*(y_w|x) / π_ref(y_w|x)) - β · log(π*(y_l|x) / π_ref(y_l|x)) )

In other words, a language model is secretly a reward model: the log-ratio of sequence probabilities between the trained policy and the reference model defines an exact implicit reward.

The DPO Loss and Its Practical Trade-Offs

Replacing the optimal policy $\pi^*$ with our trainable network $\pi_\theta$ yields the Direct Preference Optimization (DPO) loss:

L_DPO(θ) = -𝔼_{(x,y_w,y_l)∼D_pref} [ log σ( β · (log(π_θ(y_w|x) / π_ref(y_w|x)) - log(π_θ(y_l|x) / π_ref(y_l|x))) ) ]

where each sequence log-probability $\log \pi(y \mid x) = \sum_{t=1}^{|y|} \log p(y_t \mid x, y_{<t})$ is simply the sum of per-token log-probabilities. Because $\pi_{\text{ref}}$ is frozen, its log-probabilities on the dataset can be precomputed once and cached on disk, reducing DPO's memory footprint to that of standard SFT.

DPO rapidly became the default alignment method across open-weights development because it is stable, deterministic, and computationally inexpensive. However, it carries three subtle limitations:

1. Off-policy distribution shift: Standard DPO trains on a static dataset of completions rather than sampling fresh rollouts from $\pi_\theta$ as training progresses.
2. Margin overfitting on deterministic preferences: If $y_w$ is empirically always preferred over $y_l$ in a finite dataset, the sigmoid loss drives the log-ratio difference toward infinity regardless of $\beta$, which can degrade generation quality if trained for too many epochs.
3. Length exploitation: Because sequence log-probabilities sum over all tokens, DPO can inherit and amplify verbosity biases present in preference data unless length normalization is applied (Park et al., 2024).

Parameter-Efficient and Next-Generation Alignment Methods

LoRA and QLoRA: Parameter-Efficient Adaptation

Low-Rank Adaptation (LoRA) (Hu et al., 2021) is an orthogonal optimization technique applicable to CPT, SFT, or preference alignment. Observing that weight updates during fine-tuning have a low intrinsic rank, LoRA freezes each pretrained linear projection $W_0 \in \mathbb{R}^{d \times k}$ and injects a trainable low-rank bypass:

W = W_0 + (α / r) · B · A

where $A \in \mathbb{R}^{r \times k}$ and $B \in \mathbb{R}^{d \times r}$ with rank $r \ll \min(d, k)$ (typically $r \in [8, 64]$). Initializing $A$ from a Gaussian distribution and $B$ to zero ensures the adapter starts as an exact identity mapping. Because optimizer states (Adam momentum and variance) only need to be stored for $A$ and $B$, trainable parameter counts drop by more than 99 percent, and the learned update $\Delta W = (\alpha/r)BA$ can be folded back into $W_0$ after training with zero inference latency penalty.

QLoRA (Dettmers et al., 2023) pushes memory efficiency further by quantizing the frozen base weights $W_0$ to a 4-bit NormalFloat (NF4) representation, applying double quantization to the scaling constants, and using paged optimizers via unified CPU-GPU memory to absorb activation spikes on long sequences. During backpropagation, $W_0$ is dequantized on the fly to BF16 to compute gradients for the LoRA adapters, allowing a 65-billion or 70-billion parameter model to be fine-tuned on a single 48 GB workstation GPU with minimal loss in benchmark fidelity.

IPO: Identity Preference Optimization

To fix DPO's tendency to overfit when preference margins become deterministic, Azar et al. (2023) introduced Identity Preference Optimization (IPO). Rather than passing the implicit reward difference through a non-linear log-sigmoid that rewards infinite margins, IPO regresses the log-likelihood ratio difference directly toward a finite target margin determined by $\beta$:

L_IPO(θ) = 𝔼_{(x,y_w,y_l)} [ (h_θ(x,y_w) - h_θ(x,y_l) - 1/(2β))² ]

where $h_\theta(x, y) = \log(\pi_\theta(y \mid x)/\pi_{\text{ref}}(y \mid x))$. Because the quadratic loss penalizes margins that grow larger than $1/(2\beta)$ just as strongly as margins that are too small, IPO preserves the KL regularization strength $\beta$ even on noise-free preference pairs and can be trained to convergence without aggressive early stopping.

KTO: Kahneman-Tversky Optimization on Binary Signals

Collecting paired comparisons $(x, y_w, y_l)$ for the exact same prompt is expensive, whereas production systems naturally accumulate millions of unpaired binary signals (such as a user clicking thumbs-up, copying code, or regenerating a bad answer). Kahneman-Tversky Optimization (KTO) (Ethayarajh et al., 2024) adapts prospect theory from behavioral economics to align models directly on unpaired desirable or undesirable completions:

L_KTO(θ) = 𝔼_{(x,y)∼D} [ λ_y - v(x, y) ]

where the value function $v(x, y) = \lambda_D,\sigma(\beta(r_\theta(x, y) - z_{\text{ref}}))$ for desirable examples and $\lambda_U,\sigma(\beta(z_{\text{ref}} - r_\theta(x, y)))$ for undesirable examples, Evaluated relative to a batch-estimated KL reference baseline $z_{\text{ref}}$. KTO matches or exceeds DPO performance while unlocking continual alignment from single-turn production logs.

ORPO: Monolithic SFT and Preference Alignment

Standard post-training requires running SFT first, saving a checkpoint, and then loading two copies of that checkpoint ($\pi_\theta$ and $\pi_{\text{ref}}$) for DPO. Odds Ratio Preference Optimization (ORPO) (Hong et al., 2024) collapses SFT and preference alignment into a single monolithic stage without any reference model:

L_ORPO(θ) = L_SFT(θ; x, y_w) - λ · log σ( log(odds_θ(y_w|x) / odds_θ(y_l|x)) )

where $\text{odds}\theta(y \mid x) = \frac{P\theta(y \mid x)}{1 - P_\theta(y \mid x)}$ uses length-normalized sequence probabilities. The supervised cross-entropy term $L_{\text{SFT}}$ adapts the model to the target domain while the log odds-ratio penalty simultaneously pushes down the likelihood of rejected generations $y_l$ relative to $y_w$.

SPIN and RLAIF: Self-Play and Synthetic Feedback

Two complementary paradigms reduce the human annotation bottleneck from the data side:

Self-Play Fine-Tuning (SPIN) (Chen et al., 2024) eliminates preference datasets altogether. Starting from an SFT dataset, iteration $t$ treats the ground-truth human demonstrations $y \sim p_{\text{data}}$ as winning responses and synthetic completions $y' \sim \pi_{\theta_{t-1}}$ generated by the previous model checkpoint as losing responses:

L_SPIN(θ_t) = 𝔼_{(x,y)∼D_SFT, y'∼π_{t-1}(·|x)} [ -log σ( λ · (log(π_θ_t(y|x)/π_{t-1}(y|x)) - log(π_θ_t(y'|x)/π_{t-1}(y'|x))) ) ]

Training proceeds across several self-play iterations until the policy distribution becomes indistinguishable from the human target distribution.

Reinforcement Learning from AI Feedback (RLAIF) (Lee et al., 2023) keeps the pairwise preference framework of RLHF or DPO, but replaces human raters with a frontier judge model guided by an explicit rubric or constitution. While RLAIF cuts annotation costs by orders of magnitude and scales effortlessly to code and math verification, practitioners must guard against judge biases such as position preference, length bias, and stylistic self-preference.

Comparative Overview of Post-Training Methods

Method	Reward Model	Online RL	Reference Model	Data Format	Primary Strength	Main Engineering Caveat
RLHF (PPO)	Yes	Yes	Yes	Paired $(x, y_w, y_l)$ + prompts	On-policy exploration, strong ceiling	High memory (4 models) and rollout cost
DPO	No	No	Yes	Paired $(x, y_w, y_l)$	Simple, stable, fast offline training	Off-policy shift, length bias, margin overfitting
IPO	No	No	Yes	Paired $(x, y_w, y_l)$	Mathematically bounded margin	Still offline and requires paired completions
KTO	No	No	Yes	Unpaired binary $(x, y, \pm 1)$	Uses abundant thumbs-up/down logs	Slightly weaker gradient signal per sample
ORPO	No	No	No	Paired $(x, y_w, y_l)$	Single-stage SFT + alignment, low memory	Less flexibility to tune SFT and alignment separately
SPIN	No	No	Yes (prior step)	SFT pairs $(x, y)$ only	Extracts extra performance without preference data	Bounded by the ceiling of the SFT distribution
RLAIF	Optional	Optional	Optional	AI-labeled pairs $(x, y_w, y_l)$	Scales preference curation at low cost	Inherits blind spots and biases of the judge LLM

Synthesis and Open Challenges

Putting these stages together yields the modern post-training playbook: start from a compute-scaled pretrained base model, apply replay-mixed continual pretraining only when deep domain or language shifts require it, run supervised fine-tuning on a compact set of verified demonstrations, and finish with preference alignment via DPO, KTO, ORPO, or online RL. Parameter-efficient adapters (LoRA and QLoRA) make every post-training stage accessible on modest GPU clusters.

Despite rapid algorithmic simplification, one fundamental challenge cuts across every method in this survey: distribution shift between training and inference. Offline algorithms like SFT, DPO, IPO, and ORPO optimize over static text distributions, leaving the model vulnerable when its own generations drift into unfamiliar territory at deployment time. Bridging the gap between the simplicity of offline preference losses and the robustness of iterative on-policy exploration remains the central frontier of LLM alignment research.