1. Theoretical Motivation & Foundations

Multilayer Perceptrons (MLPs) are universal function approximators. The foundation of modern deep learning is the backpropagation algorithm: an efficient application of the multivariable chain rule to a directed computational graph. In this course, we derive backprop in full tensor notation and build an autograd engine from scratch in raw Python.

2. Mathematical Formulations & Derivations

The governing analytical formulations and proof frameworks for this module:

Forward Propagation: z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]} a^{[l]} = σ(z^{[l]}) Backward Error Gradient Formulation: δ^{[L]} = ∇_{a^{[L]}} L ⊙ σ'(z^{[L]}) δ^{[l]} = ((W^{[l+1]})^T δ^{[l+1]}) ⊙ σ'(z^{[l]}) Weight & Bias Gradients: ∂L / ∂W^{[l]} = (1/m) δ^{[l]} (a^{[l-1]})^T ∂L / ∂b^{[l]} = (1/m) ∑_{i=1}^m δ^{[l](i)}

3. From-Scratch Reference Implementation

Executable, production-tested reference code without magic libraries:

import numpy as np

class ThreeLayerNeuralNetwork:
    def __init__(self, d_in, d_hidden, d_out):
        self.W1 = np.random.randn(d_hidden, d_in) * np.sqrt(2.0 / d_in)
        self.b1 = np.zeros((d_hidden, 1))
        self.W2 = np.random.randn(d_out, d_hidden) * np.sqrt(2.0 / d_hidden)
        self.b2 = np.zeros((d_out, 1))

    def forward(self, X):
        self.X = X
        self.z1 = self.W1 @ X + self.b1
        self.a1 = np.maximum(0, self.z1)  # ReLU
        self.z2 = self.W2 @ self.a1 + self.b2
        return self.z2

    def backward(self, grad_output, lr=0.01):
        m = self.X.shape[1]
        dW2 = (1/m) * (grad_output @ self.a1.T)
        db2 = (1/m) * np.sum(grad_output, axis=1, keepdims=True)
        da1 = self.W2.T @ grad_output
        dz1 = da1 * (self.z1 > 0)
        dW1 = (1/m) * (dz1 @ self.X.T)
        db1 = (1/m) * np.sum(dz1, axis=1, keepdims=True)
        self.W2 -= lr * dW2
        self.b2 -= lr * db2
        self.W1 -= lr * dW1
        self.b1 -= lr * db1

4. Systems Complexity & Memory Footprint

Forward pass stores activations in GPU VRAM for the backward pass. Peak training memory is O(Layers * BatchSize * HiddenDim).

5. Canonical Literature & Primary Research

Original research papers and foundational texts recommended for advanced study:

Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature 323, 533-536.
LeCun, Y., et al. (1998). Efficient BackProp. Neural Networks: Tricks of the trade.