1. Theoretical Motivation & Foundations

Deep learning software does not run in an abstract mathematical vacuum; it is strictly bounded by silicon architecture and thermal dynamics. While CPUs dedicate most of their die area to cache hierarchy and complex branch prediction logic to minimize latency for single-threaded serial tasks, GPUs allocate silicon directly to thousands of Arithmetic Logic Units (ALUs) and specialized matrix multiply-accumulate engines (Tensor Cores) to maximize parallel throughput. Understanding the Roofline Model reveals a fundamental truth of modern AI: while the initial prompt processing (prefill) phase is compute-bound (dense matrix-matrix multiplication GEMM), autoregressive token generation (decode) is strictly memory-bandwidth bound (sparse matrix-vector GEMV). This guide deconstructs GPU memory hierarchies, HBM3e interconnects, Apple Silicon unified memory, and the mathematical formulas governing VRAM and Key-Value (KV) cache allocation.

2. Mathematical Formulations & Derivations

The governing analytical formulations and proof frameworks for this module:

The Roofline Model & Operational Intensity: I = FLOPs / Memory_Traffic_Bytes Attainable Performance P = min(Peak_Compute_FLOPs/s, Memory_Bandwidth_Bytes/s × I) Inference Memory-Bandwidth Bound (Token Generation Latency): T_token ≈ Total_Model_Weight_Bytes / Memory_Bandwidth_Bytes_per_sec Example: 70B model in 4-bit (35 GB) on an RTX 4090 (1,008 GB/s bandwidth): T_token ≈ 35 GB / 1,008 GB/s ≈ 0.0347s → Max Decode Throughput ≈ 28.8 tokens/sec Key-Value (KV) Cache Memory Allocation per Token per Batch: Memory_KV = 2 × 2 × n_layers × n_kv_heads × d_head × L_seq × batch_size × bytes_per_elem Where first 2 accounts for Keys and Values, and second 2 accounts for FP16 precision.

3. From-Scratch Reference Implementation

Executable, production-tested reference code without magic libraries:

# VRAM Footprint & KV Cache Memory Budget Calculator
def calculate_llm_vram(
    param_count_billions: float,
    quant_bits_per_param: float,
    n_layers: int,
    n_kv_heads: int,
    d_head: int,
    seq_len: int,
    batch_size: int = 1,
    bytes_per_kv: int = 2  # FP16 = 2 bytes
) -> dict:
    # 1. Model Weights Footprint
    weight_bytes = (param_count_billions * 1e9) * (quant_bits_per_param / 8.0)
    weight_gb = weight_bytes / (1024**3)
    
    # 2. KV Cache Footprint
    # 2 tensors (K & V) * n_layers * n_kv_heads * d_head * seq_len * batch * bytes_per_kv
    kv_bytes = 2 * n_layers * n_kv_heads * d_head * seq_len * batch_size * bytes_per_kv
    kv_gb = kv_bytes / (1024**3)
    
    # 3. Activation & CUDA Context Overhead (~20% headroom recommended)
    cuda_overhead_gb = 1.2 + (0.1 * weight_gb)
    total_vram_gb = weight_gb + kv_gb + cuda_overhead_gb
    
    return {
        'weights_gb': round(weight_gb, 2),
        'kv_cache_gb': round(kv_gb, 2),
        'overhead_gb': round(cuda_overhead_gb, 2),
        'total_vram_required_gb': round(total_vram_gb, 2)
    }

# Example: Llama 3 70B (Grouped Query Attention: 8 KV heads, 80 layers, head_dim 128) in 4-bit at 32k context
spec = calculate_llm_vram(
    param_count_billions=70.6,
    quant_bits_per_param=4.5,
    n_layers=80,
    n_kv_heads=8,
    d_head=128,
    seq_len=32768,
    batch_size=1
)
print('Llama 3 70B VRAM Requirement:', spec)

4. Systems Complexity & Memory Footprint

Interconnect Latency & Bandwidth: PCIe Gen 5 delivers 64 GB/s bidirectional. NVIDIA NVLink 4 (Hopper) delivers 900 GB/s bidirectional per GPU (14x PCIe). NVIDIA NVLink 5 (Blackwell) delivers 1.8 TB/s per GPU across 72-GPU compute trays.

5. Canonical Literature & Primary Research

Original research papers and foundational texts recommended for advanced study:

Williams, S., Waterman, A., & Patterson, D. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures. Communications of the ACM, 52(4).
Jouppi, N. P., et al. (2017). In-Datacenter Performance Analysis of a Tensor Processing Unit. ACM/IEEE ISCA.
NVIDIA Corporation (2022). NVIDIA H100 Tensor Core GPU Architecture Whitepaper.
Kwon, W., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. ACM SOSP.