1. Theoretical Motivation & Foundations
Modern state-of-the-art LLMs (Llama 3, Gemma, DeepSeek) depart significantly from the original 2017 transformer. Absolute sinusoidal embeddings have been replaced by Rotary Position Embeddings (RoPE). Standard LayerNorm is replaced by RMSNorm, and Multi-Head Attention is replaced by Grouped-Query Attention (GQA) to minimize KV-cache overhead. This course assembles every component into a unified, clean PyTorch codebase.
2. Mathematical Formulations & Derivations
The governing analytical formulations and proof frameworks for this module:
3. From-Scratch Reference Implementation
Executable, production-tested reference code without magic libraries:
import torch
import torch.nn as nn
class RMSNorm(nn.Module):
def __init__(self, dim: int, eps: float = 1e-6):
super().__init__()
self.eps = eps
self.weight = nn.Parameter(torch.ones(dim))
def forward(self, x):
variance = x.pow(2).mean(-1, keepdim=True)
return x * torch.rsqrt(variance + self.eps) * self.weight
4. Systems Complexity & Memory Footprint
GQA reduces KV cache memory footprint by 4x to 8x, allowing larger batch sizes and long context generation on single GPUs.
5. Canonical Literature & Primary Research
Original research papers and foundational texts recommended for advanced study:
- Su, J., et al. (2024). RoFormer: Enhanced Transformer with Rotary Position Embedding. Neurocomputing.
- Ainslie, J., et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. EMNLP.