1. Theoretical Motivation & Foundations
Supervised learning constructs mapping functions from labeled input-output pairs. This course analyzes both closed-form linear regression and non-linear logistic classification. We derive the normal equations, prove the geometric shrinkage properties of L2 Ridge (weight decay) versus L1 Lasso (sparse feature selection), and implement iterative reweighted least squares (IRLS).
2. Mathematical Formulations & Derivations
The governing analytical formulations and proof frameworks for this module:
Ordinary Least Squares Closed Form:
w^* = (X^T X)^{-1} X^T y
Ridge Regression (L2 Penalty):
w_{Ridge} = (X^T X + λ I)^{-1} X^T y
Lasso Regression (L1 Penalty):
min_w ||y - X w||_2^2 + λ ||w||_1
Logistic Hypothesis & Odds Ratio:
P(y=1 | x) = σ(w^T x) = 1 / (1 + e^{-w^T x})
log(P / (1 - P)) = w^T x
3. From-Scratch Reference Implementation
Executable, production-tested reference code without magic libraries:
import numpy as np
class RidgeRegressionFromScratch:
def __init__(self, l2_penalty: float = 1.0):
self.l2_penalty = l2_penalty
self.weights = None
def fit(self, X: np.ndarray, y: np.ndarray):
# Add bias column
X_bias = np.c_[np.ones(X.shape[0]), X]
n_features = X_bias.shape[1]
# Regularization identity without penalizing bias
I = np.eye(n_features)
I[0, 0] = 0.0
# Solve (X^T X + lambda I)^-1 X^T y
self.weights = np.linalg.inv(X_bias.T @ X_bias + self.l2_penalty * I) @ X_bias.T @ y
def predict(self, X: np.ndarray) -> np.ndarray:
X_bias = np.c_[np.ones(X.shape[0]), X]
return X_bias @ self.weights
4. Systems Complexity & Memory Footprint
Matrix inversion requires O(d^3) operations. For large feature spaces (d > 10,000), iterative stochastic gradient descent replaces matrix inversion.
5. Canonical Literature & Primary Research
Original research papers and foundational texts recommended for advanced study:
- Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. Royal Statistical Society B, 267-288.
- Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning. Springer.