1. Theoretical Motivation & Foundations

Modern state-of-the-art LLMs (Llama 3, Gemma, DeepSeek) depart significantly from the original 2017 transformer. Absolute sinusoidal embeddings have been replaced by Rotary Position Embeddings (RoPE). Standard LayerNorm is replaced by RMSNorm, and Multi-Head Attention is replaced by Grouped-Query Attention (GQA) to minimize KV-cache overhead. This course assembles every component into a unified, clean PyTorch codebase.

2. Mathematical Formulations & Derivations

The governing analytical formulations and proof frameworks for this module:

Rotary Position Embedding (RoPE): R_{Θ,m}^d x = (x_1 + i x_2) e^{i m θ_k} Root Mean Square Normalization (RMSNorm): a_i = (x_i / RMS(x)) * g_i, where RMS(x) = √( (1/d) ∑_{j=1}^d x_j^2 + ε ) Grouped-Query Attention Ratio: G = n_heads_q / n_heads_kv (KV cache compressed by factor G)

3. From-Scratch Reference Implementation

Executable, production-tested reference code without magic libraries:

import torch
import torch.nn as nn

class RMSNorm(nn.Module):
    def __init__(self, dim: int, eps: float = 1e-6):
        super().__init__()
        self.eps = eps
        self.weight = nn.Parameter(torch.ones(dim))

    def forward(self, x):
        variance = x.pow(2).mean(-1, keepdim=True)
        return x * torch.rsqrt(variance + self.eps) * self.weight

4. Systems Complexity & Memory Footprint

GQA reduces KV cache memory footprint by 4x to 8x, allowing larger batch sizes and long context generation on single GPUs.

5. Canonical Literature & Primary Research

Original research papers and foundational texts recommended for advanced study:

Su, J., et al. (2024). RoFormer: Enhanced Transformer with Rotary Position Embedding. Neurocomputing.
Ainslie, J., et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. EMNLP.