Home Blog Spatial Lab Disciplines Agentic Tools
Learn • AI Academy
↳ AI Glossary & Lexicon (57 Terms) ↳ Hardware & VRAM Calculator ↳ Token & Context Lab ↳ Inference Latency & TTFT Lab
IP Network Infrastructure About Connect
Home > Learn > Inference Latency & TTFT Lab
SYSTEMS ARCHITECTURE • PERFORMANCE ENGINEERING

LLM Inference Latency, Throughput & TTFT Systems Lab

Size your inference serving pipelines, analyze memory-bandwidth rooflines, evaluate Time-to-First-Token (TTFT) vs. inter-token decode latency, and simulate speculative decoding speedups across 10 GPU architectures.

10 GPU Profiles
H100, H200, B200, 4090, Apple M4
Roofline Model
Prefill vs Autoregressive Decode
Speculative ROI
Draft Model Tree Verification
Zero Fabrication
Hardware Bandwidth Physics
Estimated Total Response Latency
3.12 s
Single-Stream Decode: 88.4 tok/s • TTFT: 292 ms
Timing Distribution (TTFT vs Decode Duration) 9.4% Prefill / 90.6% Decode
Time to First Token (TTFT) 292 ms
Decode Phase Duration 2,828 ms
System Bottleneck: Memory-Bandwidth Bound (Autoregressive Decode)

Generating each token requires sweeping the entire model weight tensor through GPU memory. At batch size 1, operational intensity is only ~1 FLOP/byte, causing the execution units to wait on HBM memory bandwidth.

Inter-Token Latency (ITL)
11.3 ms
per generated token
Generation Speed
88.4 tok/s
single-stream throughput
Aggregate Throughput
88.4 tok/s
across 1 concurrent stream
Speculative Gain
2.28×
effective speedup multiplier
GPU Inference Hardware Comparison Matrix
Full Fleet Metrics
GPU Architecture Tier Bandwidth VRAM FP16 TFLOPS Est. Decode Speed Est. TTFT (1.5k)
Curriculum Integration

Theoretical Foundations & Architecture Guides

Master the engineering principles behind memory hierarchies, FlashAttention tiling, continuous batching, and speculative tree verification.

Level 506 • Advanced Systems

Roofline Analysis: Memory Bandwidth vs Compute Bound LLMs

Deriving the arithmetic intensity boundary: why GEMM prefill achieves high hardware utilization while GEMV autoregressive decode idles 90% of tensor cores.

Read Level 506 Guide → Level 507 • Hardware Kernels

FlashAttention-3 & Chunked Prefill Architecture

SRAM tiling algorithms that avoid round-trips to high-bandwidth memory (HBM), combined with chunked prefill to eliminate serving bubbles.

Read Level 507 Guide → Level 508 • Speculative Systems

Speculative Decoding, Medusa Heads & Tree Verification

Mathematical bounds on acceptance rates, draft-model distillation, and parallel candidate tree verification algorithms that break the autoregressive barrier.

Read Level 508 Guide → Playbook A8 • Production Systems

Continuous Batching, vLLM PagedAttention & Production SLAs

Eliminating memory fragmentation via virtual memory paging for KV caches, iteration-level continuous batching, and production latency SLAs.

Read Playbook A8 →