How Fine-Tuning Works: SFT vs. RLHF vs. DPO vs. LoRA Low-Rank Weight Updates

When a base language model (such as Llama 3 Base or DeepSeek-V3 Base) completes its pre-training phase, it has digested tens of trillions of tokens across web scrapes, books, and code repositories. It is extraordinarily knowledgeable, yet practically unusable as an assistant: if you prompt a base model with "What is the capital of France?", it may simply autocomplete with "What is the capital of Germany? What is the capital of Italy?"

Post-training transforms this raw text-completion engine into an instruction-following, aligned reasoning agent. This guide breaks down the mathematical mechanics of the four dominant post-training methodologies: SFT, RLHF, DPO, and LoRA.

1. The Post-Training Pipeline

Industrial model alignment proceeds through three sequential phases:

[Base Pre-Trained Model] (Unconstrained Next-Token Prediction)
         |
         v
[Supervised Fine-Tuning (SFT)] (Teaches Conversational Formatting & Intent)
         |
         v
[Preference Alignment (DPO / RLHF)] (Suppresses Hallucinations, Refines Reasoning)
         |
         v
[Production Aligned Checkpoint] (Deployable Agent)

2. Supervised Fine-Tuning (SFT) & Gradient Masking

SFT trains the base model on curated question-answer pairs: $(x, y)$, where $x$ is the user prompt and $y = (y_1, y_2, \dots, y_M)$ is the target assistant response.

Loss Formulation with Target Masking

A critical engineering detail is gradient masking. The model must not be penalized or rewarded for predicting the user prompt $x$. Gradients are only computed over assistant tokens $y$:

3. Reinforcement Learning from Human Feedback (RLHF via PPO)

Pioneered by Christiano et al. (2017) and scaled by OpenAI for InstructGPT and ChatGPT, classical RLHF formalizes alignment as a reinforcement learning Markov Decision Process:

1. Step 1: Reward Model Training: Human annotators rank model completions $y_w \succ y_l$ (winner vs. loser). A reward model $r_\psi(x, y)$ is trained via the Bradley-Terry preference loss: $$\mathcal{L}_{RM}(\psi) = -\mathbb{E}_{(x, y_w, y_l)} \left[ \log \sigma(r_\psi(x, y_w) - r_\psi(x, y_l)) \right]$$
2. Step 2: Policy Optimization via PPO: The active language model $\pi_\theta$ generates candidate responses and receives scalar rewards from $r_\psi$. To prevent the policy from collapsing into nonsensical adversarial token loops (reward hacking), a Kullback-Leibler (KL) divergence penalty enforces proximity to the frozen reference model $\pi_{\text{ref}}$:

The PPO Memory Crisis: PPO requires loading four full models simultaneously into GPU VRAM: the active Actor policy ($\pi_\theta$), the Critic value network ($V_\phi$), the Reference model ($\pi_{\text{ref}}$), and the Reward model ($r_\psi$). This massive hardware footprint made RLHF inaccessible to non-hyperscalers.

4. Direct Preference Optimization (DPO): Eliminating the Reward Model

In 2023, Rafael Rafailov, Archit Sharma, and Eric Mitchell at Stanford published Direct Preference Optimization (DPO), solving the RLHF complexity crisis with an elegant mathematical reparameterization.

Rafailov et al. proved mathematically that the exact optimal reward function under the KL-constrained PPO objective can be expressed directly in terms of the optimal policy itself:

By substituting this closed-form identity directly into the Bradley-Terry preference loss, the unknown partition function $Z(x)$ cancels out identically, yielding the DPO Loss Function:

5. LoRA: Low-Rank Adaptation Matrix Factorization

When fine-tuning a 70-billion parameter model, full weight fine-tuning requires updating all 70B weights. Using the AdamW optimizer (which maintains 32-bit floating point momentum and variance buffers), storing optimizer states alone consumes 16 bytes per parameter:

This necessitates an 8-GPU cluster of NVIDIA H100 80GB cards simply to hold the optimizer states during training. LoRA (Hu et al., 2021) resolves this via the Intrinsic Rank Hypothesis.

The Mathematical Factorization

Edward Hu et al. observed that although weight matrices $W_0 \in \mathbb{R}^{d \times k}$ are high-dimensional, the weight updates $\Delta W$ during domain adaptation have an extremely low "intrinsic rank" $r \ll \min(d, k)$ (often $r \in [8, 64]$).

Instead of updating $W_0$ directly, LoRA freezes $W_0$ and decomposes $\Delta W$ into the product of two low-rank matrices:

Original Weight:     W0  in R^[4096 x 4096]   ==> 16,777,216 Parameters (Frozen)
Decomposed Adapters: A   in R^[r=16 x 4096]   ==>     65,536 Parameters (Trainable)
                     B   in R^[4096 x r=16]   ==>     65,536 Parameters (Trainable)
                     ------------------------------------------------------------
                     Total Trainable Parameters:     131,072 Parameters (99.2% Reduction!)

Initialization & Scaling Invariant

• Matrix $A$ is initialized from a Gaussian random distribution $\mathcal{N}(0, \sigma^2)$.
• Matrix $B$ is initialized strictly to zero ($B = 0$). This guarantees that at the start of training, $\Delta W = B \cdot A = 0$, so the model's pre-trained behavior is unaltered.
• The constant scalar $\frac{\alpha}{r}$ scales adapter updates, eliminating the need to retune learning rates when experimenting with different rank values $r$.

Zero Inference Latency Overhead

At deployment time, the adapter weights can be permanently fused into the base model weights via a single linear matrix addition: $W_{\text{deploy}} = W_0 + \frac{\alpha}{r} B A$. During production serving, LoRA incurs zero additional inference latency or memory overhead.

6. QLoRA: 4-Bit NormalFloat Quantization

Tim Dettmers et al. (2023) introduced QLoRA, enabling engineers to fine-tune a 70B parameter model on a single consumer NVIDIA RTX 3090/4090 GPU (24 GB VRAM):

• 4-Bit NormalFloat (NF4): An information-theoretically optimal quantile quantization data type designed for normally distributed neural weights.
• Double Quantization (DQ): Quantizes the quantization constants themselves, saving 0.37 bits per parameter.
• Paged Optimizers: Automatically offloads memory spikes from GPU VRAM to system CPU RAM using CUDA Unified Memory, preventing Out-Of-Memory (OOM) crashes during long context sequences.

7. Post-Training Comparison Matrix