1. Theoretical Motivation & Foundations

Why do deep neural networks train reliably at scale? This course explores gradient stability across deep graphs. We analyze non-linear activation functions (from logistic sigmoids to Gaussian Error Linear Units and SwiGLU), prove why decoupled weight decay in AdamW prevents regularized gradient decay, and examine learning rate schedules.

2. Mathematical Formulations & Derivations

The governing analytical formulations and proof frameworks for this module:

SwiGLU Gated Activation: SwiGLU(x) = (x W_1) ⊙ Swish(x W_2) = (x W_1) ⊙ ((x W_2) * σ(x W_2)) AdamW Update Rules: m_t = β_1 m_{t-1} + (1 - β_1) g_t v_t = β_2 v_{t-1} + (1 - β_2) g_t^2 m̂_t = m_t / (1 - β_1^t), v̂_t = v_t / (1 - β_2^t) θ_t = θ_{t-1} - η (m̂_t / (√(v̂_t) + ε) + λ θ_{t-1})

3. From-Scratch Reference Implementation

Executable, production-tested reference code without magic libraries:

import numpy as np

class AdamWOptimizer:
    def __init__(self, params, lr=1e-3, beta1=0.9, beta2=0.999, eps=1e-8, weight_decay=0.01):
        self.params = params
        self.lr = lr
        self.beta1 = beta1
        self.beta2 = beta2
        self.eps = eps
        self.weight_decay = weight_decay
        self.m = {k: np.zeros_like(v) for k, v in params.items()}
        self.v = {k: np.zeros_like(v) for k, v in params.items()}
        self.t = 0

4. Systems Complexity & Memory Footprint

AdamW maintains 2 state tensors (first and second moments) per parameter, tripling memory overhead from 4 bytes (FP32) to 12-16 bytes per parameter.

5. Canonical Literature & Primary Research

Original research papers and foundational texts recommended for advanced study:

Loshchilov, I., & Hutter, F. (2019). Decoupled Weight Decay Regularization. ICLR.
Shazeer, N. (2020). GLU Variants Improve Transformer. arXiv:2002.05202.