1. Theoretical Motivation & Foundations

Convolutional Neural Networks exploit two fundamental physical properties of images: local connectivity and translation equivariance. We derive discrete 2D convolutions, compute effective receptive fields across deep layers, and prove why He et al.'s identity skip connections in ResNet mathematically eliminate vanishing gradient bottlenecks.

2. Mathematical Formulations & Derivations

The governing analytical formulations and proof frameworks for this module:

Discrete 2D Cross-Correlation: S(i, j) = (I * K)(i, j) = ∑_{m} ∑_{n} I(i+m, j+n) K(m, n) Residual Formulation: y = F(x, {W_i}) + x Identity Shortcut Gradient Preservation: ∂L / ∂x = (∂L / ∂y) * (∂F / ∂x + I)

3. From-Scratch Reference Implementation

Executable, production-tested reference code without magic libraries:

import numpy as np

def conv2d_forward_naive(X, K, stride=1, padding=1):
    N, C, H, W = X.shape
    F, _, HH, WW = K.shape
    X_pad = np.pad(X, ((0,0), (0,0), (padding, padding), (padding, padding)))
    out_h = 1 + (H + 2 * padding - HH) // stride
    out_w = 1 + (W + 2 * padding - WW) // stride
    out = np.zeros((N, F, out_h, out_w))
    return out

4. Systems Complexity & Memory Footprint

Optimized conv implementations use im2col matrix multiplication (GEMM) to map receptive patches to dense GPU tensor cores.

5. Canonical Literature & Primary Research

Original research papers and foundational texts recommended for advanced study:

He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep Residual Learning for Image Recognition. CVPR.
LeCun, Y., et al. (1989). Backpropagation applied to handwritten zip code recognition. Neural Computation.