1. Theoretical Motivation & Foundations

Self-attention revolutionized natural language processing by enabling O(1) sequential path length between any two tokens in a sequence. Rather than compressing context through a recurrent hidden state, attention computes a dynamic, content-based routing matrix. We derive Queries, Keys, and Values, explain why scaling by 1/sqrt(d_k) prevents softmax gradient saturation, and implement causal masking.

2. Mathematical Formulations & Derivations

The governing analytical formulations and proof frameworks for this module:

Scaled Dot-Product Attention Equation: Attention(Q, K, V) = softmax((Q K^T) / √(d_k)) V Softmax Temperature Scaling Derivation: If q_i, k_i ~ N(0, 1) i.i.d., then Var(q · k) = d_k. Dividing by √(d_k) restores Var((q · k) / √(d_k)) = 1.0, preventing softmax inputs from exploding into regions with near-zero gradients. Causal Autoregressive Mask: M_{ij} = 0 for j ≤ i, and -∞ for j > i

3. From-Scratch Reference Implementation

Executable, production-tested reference code without magic libraries:

import torch
import torch.nn as nn
import math

def scaled_dot_product_attention(Q, K, V, mask=None):
    """Compute scaled dot-product attention in pure PyTorch."""
    d_k = Q.size(-1)
    # Scores shape: (batch, heads, seq_len_q, seq_len_k)
    scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(d_k)
    
    if mask is not None:
        scores = scores.masked_fill(mask == 0, -1e9)
        
    weights = torch.softmax(scores, dim=-1)
    output = torch.matmul(weights, V)
    return output, weights

4. Systems Complexity & Memory Footprint

Standard attention requires O(N^2) memory and FLOPs relative to sequence length N. FlashAttention mitigates memory footprint via GPU SRAM tiling.

5. Canonical Literature & Primary Research

Original research papers and foundational texts recommended for advanced study:

Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS 30.
Dao, T., et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS.