Frontier AI Scaling Laws & Compute Budget Planner

A first-principles engineering tool modeling pre-training dynamics across dense and mixture-of-experts architectures. Calculate exact floating-point operations (FLOPs), Chinchilla compute-optimal frontiers, cluster wall-clock training duration, and training CapEx for leading GPU accelerators.

Model & Hardware Sizing Inputs

Network Architecture

MoE decouples total parameter capacity from FLOPs consumed per forward pass.

Total Model Parameters 70.0 B

Active weights stored in memory. Total weights $N_{\text{total}}$.

Active Parameters per Token 37.0 B

Weights evaluated per token across selected routed experts ($N_{\text{active}}$).

Training Tokens ($D$) 15.00 T

Total pre-training corpus tokens consumed during optimization.

Accelerator Architecture

Tensor Core Precision

FP8 doubles peak mathematical execution throughput relative to 16-bit.

Cluster Size ($N_{\text{GPUs}}$) 16,384

Concurrent GPUs networked via NVLink / InfiniBand fabrics.

Model FLOPs Utilization (MFU) 42 %

Efficiency after 3D tensor, pipeline, and data parallel communication overhead.

GPU Cloud Rate ($ / GPU-hour)

Blended cloud rental rate (Tier-1 provider with interconnect).

Total Pre-Training Compute
6.30e+24
6,300 ZettaFLOPs ($6 \times N_{\text{active}} \times D$)

Chinchilla Optimal Ratio
10.7 ×
Optimal: 1.40T Tokens ($D \approx 20N$)

Cluster Wall-Clock Time
53.8 Days
1,291 Total Continuous Hours

Estimated Training CapEx
$60.28 M
21.15 Million GPU-Hours

Empirical Loss vs. Compute Scaling Frontier

Hoffmann et al. (Chinchilla) Power-Law Loss Projection

Chinchilla Optimal

Selected Configuration

Pre-Training Token Allocation Strategy Inference-Optimal Over-Trained (10.7×)

Chinchilla Baseline: 1.4T Tokens Over-Trained Surplus: 13.6T Tokens

Active GPU Compute Power

13.62 EFLOPS

Peak theoretical: 32.43 ExaFLOPs

Predicted Test Perplexity

5.82

Estimated Loss: 1.761 nats

Inference Compute Amortization

Optimal

Downstream queries amortize training cost

Energy Footprint (MWh)

14,717 MWh

PUE ~1.2 with liquid-assisted cooling

First-Principles Mathematical Foundations

Understanding the physics and numerical dynamics governing deep neural network scaling. From early power laws to modern inference-optimal over-training paradigms.

FLOPs Physics

The 6ND Pre-Training Compute Rule

During standard autoregressive transformer pretraining, computing the forward pass across $N$ non-embedding parameters requires approximately $2N$ floating-point operations per token:

FLOPs_forward ≈ 2 × N × D

The backward pass requires both computing gradients with respect to activations and computing gradients with respect to weights, taking $4N$ FLOPs per token. Summing both passes gives the fundamental invariant:

C_total ≈ 6 × N_active × D

In Mixture-of-Experts (MoE) architectures, only the routed and shared active parameters $N_{\text{active}}$ execute computation for each token.

Chinchilla vs. Kaplan

The Compute-Optimal Frontier

Kaplan et al. (2020) originally argued that model parameters should scale significantly faster than dataset tokens ($N \propto C^{0.73}, D \propto C^{0.27}$). However, Hoffmann et al. (2022) proved this was biased by sub-optimal learning rate schedules:

L(N, D) = E + A / N^α + B / D^β

With fitted constants $E=1.69, A=406.4, B=410.7, \alpha=0.34, \beta=0.28$. Under a fixed compute budget $C$, both parameters and tokens must scale in equal proportion:

D_optimal ≈ 20 × N

Economics of Scale

The Inference-Optimal Over-Training Pivot

While Chinchilla optimizes compute during the training phase alone, real-world deployment incurs substantial inference costs. A model trained past the Chinchilla boundary is physically smaller, meaning:

Less VRAM required per inference node (fits on fewer GPUs).
Significantly lower KV cache memory footprint.
Substantially higher tokens per second per dollar.

Meta's Llama 3 8B (15T tokens) was over-trained by $93.7\times$, and DeepSeek-V3 by $19.9\times$, driving down the marginal serving cost for billions of downstream queries.