1. Theoretical Motivation & Foundations

Full parameter fine-tuning of 70B+ parameter models requires massive GPU clusters simply to store optimizer states. Low-Rank Adaptation (LoRA) hypothesizes that weight updates have an intrinsic low latent rank. By decomposing delta weights into low-rank matrices W = W_0 + B A, LoRA reduces trainable parameters by up to 99.9% while preserving downstream performance.

2. Mathematical Formulations & Derivations

The governing analytical formulations and proof frameworks for this module:

LoRA Forward Formulation: h = W_0 x + ΔW x = W_0 x + (B A) x * (α / r) where W_0 ∈ ℝ^{d × k}, B ∈ ℝ^{d × r}, A ∈ ℝ^{r × k}, and r ≪ min(d, k) Zero Initialization Guarantee: A ~ N(0, σ^2), B = 0 ⇒ ΔW = 0 at t=0

3. From-Scratch Reference Implementation

Executable, production-tested reference code without magic libraries:

import torch
import torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, in_features, out_features, rank=8, alpha=16):
        super().__init__()
        self.linear = nn.Linear(in_features, out_features, bias=False)
        self.linear.weight.requires_grad = False  # Freeze base
        self.lora_A = nn.Parameter(torch.randn(rank, in_features) * 0.01)
        self.lora_B = nn.Parameter(torch.zeros(out_features, rank))
        self.scaling = alpha / rank

    def forward(self, x):
        base_out = self.linear(x)
        lora_out = (x @ self.lora_A.T @ self.lora_B.T) * self.scaling
        return base_out + lora_out

4. Systems Complexity & Memory Footprint

Base model weights are frozen and shared across multiple tasks; only the tiny adapter matrices A and B are swapped in VRAM during multi-tenant inference.

5. Canonical Literature & Primary Research

Original research papers and foundational texts recommended for advanced study:

Hu, E. J., et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. ICLR.
Dettmers, T., et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS.