1. Theoretical Motivation & Foundations
Pretraining converts compute into generalizable representations. In this course, we analyze tokenization algorithms (Byte-Pair Encoding, WordPiece) and their subtle failure modes. We study Chinchilla scaling laws, deriving the optimal balance between model parameter count N and dataset token volume D given a compute budget C.
2. Mathematical Formulations & Derivations
The governing analytical formulations and proof frameworks for this module:
BPE Compression Iteration:
arg max_{(c_1, c_2)} Count(c_1, c_2)
Chinchilla Compute Scaling Law:
C ≈ 6 N D FLOPs
Optimal Parameters: N_{opt} ∝ C^a, Optimal Tokens: D_{opt} ∝ C^b
Empirically: a ≈ 0.5, b ≈ 0.5 (Scale N and D in equal proportion)
3. From-Scratch Reference Implementation
Executable, production-tested reference code without magic libraries:
from collections import Counter
def get_bpe_stats(vocab):
pairs = Counter()
for word, freq in vocab.items():
symbols = word.split()
for i in range(len(symbols) - 1):
pairs[symbols[i], symbols[i+1]] += freq
return pairs
4. Systems Complexity & Memory Footprint
Chinchilla showed that older models (GPT-3 175B on 300B tokens) were severely undertrained; optimal performance requires ~20 tokens per parameter.
5. Canonical Literature & Primary Research
Original research papers and foundational texts recommended for advanced study:
- Hoffmann, J., et al. (2022). Training Compute-Optimal Large Language Models (Chinchilla). arXiv:2203.15556.
- Kaplan, J., et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361.