1. Theoretical Motivation & Foundations

Next-token prediction alone does not produce an obedient assistant. Preference alignment steers models toward human values. We explore the classical RLHF pipeline (supervised fine-tuning, reward model training, and PPO policy optimization) and derive Direct Preference Optimization (DPO), which analytically solves for the optimal policy in closed form without requiring a separate reward model.

2. Mathematical Formulations & Derivations

The governing analytical formulations and proof frameworks for this module:

Bradley-Terry Human Preference Probability: P(y_w ≻ y_l | x) = σ(r(x, y_w) - r(x, y_l)) Direct Preference Optimization (DPO) Objective: L_{DPO} = -E_{(x, y_w, y_l)} [log σ( β log(π_θ(y_w|x) / π_{ref}(y_w|x)) - β log(π_θ(y_l|x) / π_{ref}(y_l|x)) )]

3. From-Scratch Reference Implementation

Executable, production-tested reference code without magic libraries:

import torch
import torch.nn.functional as F

def dpo_loss(policy_chosen_logps, policy_rejected_logps,
             ref_chosen_logps, ref_rejected_logps, beta=0.1):
    pi_logratios = policy_chosen_logps - policy_rejected_logps
    ref_logratios = ref_chosen_logps - ref_rejected_logps
    logits = beta * (pi_logratios - ref_logratios)
    return -F.logsigmoid(logits).mean()

4. Systems Complexity & Memory Footprint

DPO eliminates the need to train a 2nd reward model and actor-critic networks, stabilizing convergence and cutting VRAM overhead by 50%.

5. Canonical Literature & Primary Research

Original research papers and foundational texts recommended for advanced study:

Rafailov, R., et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS.
Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS.