1. Theoretical Motivation & Foundations

Large language model inference exhibits a fundamental duality governed by computer architecture: the prefill phase and the decode phase operate in completely separate computational regimes. During prefill, all input prompt tokens are processed simultaneously using batched matrix multiplications (GEMM). This creates high arithmetic intensity (typically 80 to 150 FLOPs per byte transferred from DRAM), pushing GPU Tensor Cores to their theoretical compute saturation ceiling. Conversely, auto-regressive token generation (decode) emits exactly one token at a time per sequence. Each forward pass must transfer all model weights from High Bandwidth Memory (HBM) into on-chip SRAM to multiply against a single input vector (GEMV). The resulting arithmetic intensity collapses to approximately 1 to 2 FLOPs per byte, leaving over 95% of GPU Tensor Cores completely idle while the memory bus is 100% saturated. This guide derives the operational roofline model, calculates the critical hardware ridge point across modern accelerators (H100, H200, B200, RTX 4090, Apple Silicon), and demonstrates how batching and tensor parallelism shift workloads across the roofline boundary.

2. Mathematical Formulations & Derivations

The governing analytical formulations and proof frameworks for this module:

Operational Roofline Formulation: Attainable Performance P (TFLOP/s) = min(Peak Compute P_peak, Arithmetic Intensity I × Memory Bandwidth B_mem) Arithmetic Intensity: I = Total Floating Point Operations (FLOPs) / Total DRAM Bytes Transferred (Bytes) Hardware Ridge Point (Operational Boundary): I* = P_peak / B_mem • If I < I*: Memory Bandwidth Bound (Attainable P = I × B_mem) • If I ≥ I*: Compute Bound (Attainable P = P_peak) Decode vs. Prefill Arithmetic Intensity (FP16, Model Parameters P, Batch B): • Single-Token Decode (B=1): I_decode = (2 × P FLOPs) / (2 × P Bytes) = 1.0 FLOP/Byte • H100 SXM5 Ridge Point: I* = 1979 TFLOP/s / 3.35 TB/s ≈ 590.7 FLOPs/Byte -> Decode operates at 1/590th of the ridge point (0.17% compute efficiency!) • Batched Decode (Batch B): I_batched ≈ (2 × P × B) / (2 × P + 2 × B × KV_bytes) ≈ B FLOPs/Byte • Batched Prefill (Prompt Length S): I_prefill ≈ S FLOPs/Byte (Compute saturated when S ≥ 600)

3. From-Scratch Reference Implementation

Executable, production-tested reference code without magic libraries:

# Roofline Regime Analyzer & Attainable Throughput Calculator
def calculate_roofline_operating_point(
    model_params_b: float,
    precision_bytes: float,
    batch_size: int,
    gpu_tflops_peak: float,
    gpu_bandwidth_tb_s: float
) -> dict:
    total_flops = 2.0 * model_params_b * 1e9 * batch_size
    model_weight_bytes = model_params_b * 1e9 * precision_bytes
    # Assuming weights dominate memory traffic for moderate KV sizes
    arithmetic_intensity = total_flops / model_weight_bytes
    
    ridge_point = gpu_tflops_peak / gpu_bandwidth_tb_s
    is_compute_bound = arithmetic_intensity >= ridge_point
    
    if is_compute_bound:
        attainable_tflops = gpu_tflops_peak
        step_latency_sec = total_flops / (gpu_tflops_peak * 1e12)
    else:
        attainable_tflops = arithmetic_intensity * gpu_bandwidth_tb_s
        step_latency_sec = model_weight_bytes / (gpu_bandwidth_tb_s * 1e12)
        
    compute_efficiency = (attainable_tflops / gpu_tflops_peak) * 100.0
    tokens_per_sec = batch_size / step_latency_sec
    
    return {
        'arithmetic_intensity_flops_per_byte': round(arithmetic_intensity, 2),
        'ridge_point': round(ridge_point, 2),
        'regime': 'COMPUTE-BOUND' if is_compute_bound else 'MEMORY-BANDWIDTH-BOUND',
        'compute_efficiency_pct': round(compute_efficiency, 2),
        'throughput_tokens_per_sec': round(tokens_per_sec, 1)
    }

# Benchmark Llama-3-70B (FP8 = 1 byte/param) on NVIDIA H100 (1979 TFLOPS, 3.35 TB/s)
res_b1 = calculate_roofline_operating_point(70, 1.0, 1, 1979, 3.35)
res_b64 = calculate_roofline_operating_point(70, 1.0, 64, 1979, 3.35)
print('Batch 1 Decode:', res_b1)
print('Batch 64 Decode:', res_b64)

4. Systems Complexity & Memory Footprint

Single-stream LLM generation is inherently memory-bandwidth bound: faster GPUs with identical memory bandwidth yield zero latency improvement during decode. To increase throughput and arithmetic intensity, serving engines must aggregate requests into dynamic batches or employ tensor parallelism to divide weight matrices across high-speed NVLink interconnects.

5. Canonical Literature & Primary Research

Original research papers and foundational texts recommended for advanced study:

Williams, S., Waterman, A., & Patterson, D. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures. CACM.
Pope, R., et al. (2023). Efficiently Scaling Transformer Inference on TPU v4. MLSys.
NVIDIA Corporation. (2024). NVIDIA Hopper H100 Tensor Core GPU Architecture Whitepaper.