1. Theoretical Motivation & Foundations

Machine learning training is fundamentally an unconstrained optimization problem in billions of variables. Gradient descent navigates non-convex, high-dimensional loss surfaces where local minima, plateaus, and saddle points dominate. Understanding first-order conditions (gradients), second-order curvature (the Hessian), and Taylor series expansions reveals why simple SGD oscillates in narrow valleys and why momentum and adaptive learning rate mechanics are necessary for convergence.

2. Mathematical Formulations & Derivations

The governing analytical formulations and proof frameworks for this module:

Gradient Vector: ∇f(x) = [∂f/∂x_1, ∂f/∂x_2, ..., ∂f/∂x_n]^T The Jacobian Matrix for vector function f: ℝ^n → ℝ^m: J_{ij} = ∂f_i / ∂x_j The Hessian Curvature Matrix for scalar loss L: ℝ^n → ℝ: H_{ij} = ∂^2 L / (∂x_i ∂x_j) Gradient Descent Update with Classical Momentum: v_{t+1} = β v_t + ∇f(x_t) x_{t+1} = x_t - α v_{t+1}

3. From-Scratch Reference Implementation

Executable, production-tested reference code without magic libraries:

import numpy as np

class MomentumOptimizer:
    def __init__(self, lr=0.01, beta=0.9):
        self.lr = lr
        self.beta = beta
        self.velocity = None

    def step(self, params, grads):
        if self.velocity is None:
            self.velocity = np.zeros_like(params)
        self.velocity = self.beta * self.velocity + grads
        params -= self.lr * self.velocity
        return params

# Rosenbrock non-convex test surface
def rosenbrock(x, y, a=1.0, b=100.0):
    return (a - x)**2 + b * (y - x**2)**2

def rosenbrock_grad(x, y, a=1.0, b=100.0):
    dx = -2 * (a - x) - 4 * b * x * (y - x**2)
    dy = 2 * b * (y - x**2)
    return np.array([dx, dy])

4. Systems Complexity & Memory Footprint

Gradient Descent is O(n) per step. Second-order Newton methods require computing and inverting H, which is O(n^3)—computationally intractable for billion-parameter models.

5. Canonical Literature & Primary Research

Original research papers and foundational texts recommended for advanced study:

Boyd, S., & Vandenberghe, L. (2004). Convex Optimization. Cambridge University Press.
Nesterov, Y. (1983). A method of solving a convex programming problem with convergence rate O(1/k^2). Soviet Math. Dokl.
Ruder, S. (2016). An overview of gradient descent optimization algorithms. arXiv:1609.04747.