1. Theoretical Motivation & Foundations

Supervised learning constructs mapping functions from labeled input-output pairs. This course analyzes both closed-form linear regression and non-linear logistic classification. We derive the normal equations, prove the geometric shrinkage properties of L2 Ridge (weight decay) versus L1 Lasso (sparse feature selection), and implement iterative reweighted least squares (IRLS).

2. Mathematical Formulations & Derivations

The governing analytical formulations and proof frameworks for this module:

Ordinary Least Squares Closed Form: w^* = (X^T X)^{-1} X^T y Ridge Regression (L2 Penalty): w_{Ridge} = (X^T X + λ I)^{-1} X^T y Lasso Regression (L1 Penalty): min_w ||y - X w||_2^2 + λ ||w||_1 Logistic Hypothesis & Odds Ratio: P(y=1 | x) = σ(w^T x) = 1 / (1 + e^{-w^T x}) log(P / (1 - P)) = w^T x

3. From-Scratch Reference Implementation

Executable, production-tested reference code without magic libraries:

import numpy as np

class RidgeRegressionFromScratch:
    def __init__(self, l2_penalty: float = 1.0):
        self.l2_penalty = l2_penalty
        self.weights = None

    def fit(self, X: np.ndarray, y: np.ndarray):
        # Add bias column
        X_bias = np.c_[np.ones(X.shape[0]), X]
        n_features = X_bias.shape[1]
        # Regularization identity without penalizing bias
        I = np.eye(n_features)
        I[0, 0] = 0.0
        # Solve (X^T X + lambda I)^-1 X^T y
        self.weights = np.linalg.inv(X_bias.T @ X_bias + self.l2_penalty * I) @ X_bias.T @ y

    def predict(self, X: np.ndarray) -> np.ndarray:
        X_bias = np.c_[np.ones(X.shape[0]), X]
        return X_bias @ self.weights

4. Systems Complexity & Memory Footprint

Matrix inversion requires O(d^3) operations. For large feature spaces (d > 10,000), iterative stochastic gradient descent replaces matrix inversion.

5. Canonical Literature & Primary Research

Original research papers and foundational texts recommended for advanced study:

Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. Royal Statistical Society B, 267-288.
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning. Springer.