Connect
Home About Disciplines Agentic Tools Learn • AI Academy AI Glossary & Lexicon (57 Terms) AI Hardware & VRAM Calculator Infrastructure Blog Connect
Interactive Systems Engineering Tool

Frontier AI Hardware & Local LLM VRAM Calculator

Calculate exact memory footprints for model weights, Grouped-Query Attention (GQA) KV caches, and activation buffers across quantization levels. Project memory-bandwidth bound generation speeds (tokens/sec) and verify physical hardware compatibility before deploying.

20+ Models
Llama 3.3, DeepSeek, Qwen 2.5
18+ Rigs
RTX 4090, Apple Silicon, H100
First-Principles
Exact GQA & Roofline Math
100% Offline
Pure Reactive Client-Side

Model & Context Deck

2K 4K 8K 16K 32K 64K 128K
Total Required Memory Footprint
0.00 GiB
0.00 GB decimal (Safety Headroom: Recommended +15%)
VRAM Allocation Breakdown Weights Dominant
Model Weights
0.00 GiB
0% of total
KV Cache (GQA)
0.00 GiB
0% of total
Activations
0.00 GiB
0% of total
Driver Overhead
0.60 GiB
0% of total

Hardware Compatibility & Generation Speed

Estimated autoregressive decode speed (tokens/sec) based on physical memory bandwidth.

First Principles Systems Engineering

The Computer Architecture Behind LLM Memory & Speed

Why does a model take twice as much VRAM at 64k context? Why does token generation speed depend entirely on memory bandwidth rather than GPU compute TFLOPS? Understand the underlying physics of machine learning hardware.

Bottleneck Theory • The Roofline Model

Why Memory Bandwidth Dictates Generation Speed

During LLM generation (the decode phase), the model predicts one token at a time. To emit a single token, every single parameter weight must be read from VRAM into the GPU's register files and tensor cores.

Max Tokens/Sec = Hardware Bandwidth (GB/s) / Model Weights Size (GB)

For example, an RTX 4090 has 1,008 GB/s memory bandwidth. A 70B model at Q4_K_M is roughly 40 GB. To generate 1 token, the GPU must stream 40 GB across the bus. At 1,008 GB/s, the theoretical upper bound is:

1,008 GB/s ÷ 40 GB = 25.2 tokens/second

No matter how many TFLOPS of compute you have, generation speed is physically locked to the speed of your memory bus.

Architecture Optimization • Attention Math

How Grouped-Query Attention (GQA) Saves VRAM

In standard Multi-Head Attention (MHA), every query head has its own dedicated Key and Value heads ($n_{kv} = n_{heads}$). At 128k context, standard MHA causes the KV cache to balloon past 100 GB.

KV Cache = 2 × Batch × Context × Layers × KV_Heads × Head_Dim × Bytes

Modern models like Llama 3.3 and Qwen 2.5 use Grouped-Query Attention (GQA), where multiple query heads share a single key/value head (e.g. 8 KV heads for 64 query heads = 8:1 ratio).

This reduces the KV cache footprint by an astonishing 87.5% (8× smaller), allowing long-context documents to fit on consumer GPUs.

Numerical Precision • GGUF & AWQ

The Mathematics of Weight Quantization

Standard model training operates in 16-bit floating point (FP16/BF16), requiring 2.0 bytes per parameter. Quantization compresses weights into lower bit representations:

  • FP16 (16-bit): 70B model = 140 GB VRAM
  • FP8 (8-bit): 70B model = 70 GB VRAM
  • Q4_K_M (4.5-bit): 70B model = 40 GB VRAM
  • Q2_K (2.6-bit): 70B model = 24 GB VRAM (fits single RTX 3090)

Modern k-quants use block-level scales and offsets so that sensitive layers (attention matrices) retain higher bit precision while feedforward layers are compressed.

Foundational Coursework

Dive deeper into GPU microarchitecture and AI factories

Study Hardware 600: The Silicon & Hardware Behind AI for tensor core scheduling, systolic arrays, and HBM memory controllers.

Hardware 600 Module → AI Glossary (57) →