1. Theoretical Motivation & Foundations

Machine learning is statistical inference performed under computational constraints. Language models estimate joint probability distributions over sequences of discrete tokens. Understanding Shannon entropy, cross-entropy, and KL divergence explains why we minimize negative log-likelihood (NLL) during pretraining, how temperature scaling reshapes categorical distributions during sampling, and how preference alignment constrains policy divergence.

2. Mathematical Formulations & Derivations

The governing analytical formulations and proof frameworks for this module:

Bayes' Theorem: P(θ | D) = P(D | θ) P(θ) / P(D) Maximum Likelihood Estimation (MLE): θ_{MLE} = argmax_θ ∑_{i=1}^N log P(x_i | θ) Shannon Entropy: H(P) = -∑_{x} P(x) log_2 P(x) Kullback-Leibler Divergence: D_{KL}(P || Q) = ∑_{x} P(x) log(P(x) / Q(x)) Cross-Entropy Equivalence: H(P, Q) = H(P) + D_{KL}(P || Q) = -∑_{x} P(x) log Q(x)

3. From-Scratch Reference Implementation

Executable, production-tested reference code without magic libraries:

import numpy as np

def cross_entropy_loss(probabilities: np.ndarray, target_indices: np.ndarray) -> float:
    """Compute cross-entropy loss with numerical stabilization."""
    N = probabilities.shape[0]
    eps = 1e-15
    clipped = np.clip(probabilities[np.arange(N), target_indices], eps, 1.0 - eps)
    return -np.mean(np.log(clipped))

def kl_divergence(p: np.ndarray, q: np.ndarray) -> float:
    """Compute relative entropy D_KL(P || Q)."""
    eps = 1e-15
    p = np.clip(p, eps, 1.0)
    q = np.clip(q, eps, 1.0)
    return np.sum(p * np.log(p / q))

4. Systems Complexity & Memory Footprint

Entropy computation scales linearly with vocabulary size: O(|V|). Numerical clipping prevents log(0) IEEE 754 floating-point underflow.

5. Canonical Literature & Primary Research

Original research papers and foundational texts recommended for advanced study:

Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal 27, 379-423.
Cover, T. M., & Thomas, J. A. (2006). Elements of Information Theory. Wiley-Interscience.