The Second Scaling Law: Inference as Deliberation
For six years, the dominant paradigm in artificial intelligence was governed by Chinchilla and Kaplan pretraining scaling laws: performance improved as a smooth power-law function of dataset size (tokens) and parameter count (weights). But as high-quality human web text approached exhaustion and pretraining cluster costs exceeded hundreds of millions of dollars, the marginal returns on raw pretraining compute began to diminish.
The arrival of frontier reasoning models (OpenAI o1/o3-mini, DeepSeek-R1, Claude 3.7 Sonnet) unlocked an entirely orthogonal scaling axis: Test-Time Compute.
Outcome Verifiers vs. Process Reward Models (PRMs)
Rather than outputting the first probable completion in a single forward pass, reasoning architectures spend additional computational energy during generation:
- Thinking Tokens: The model generates thousands of intermediate reasoning steps, hypothesizing solutions, checking for internal inconsistencies, and actively refuting erroneous sub-paths.
- Monte Carlo Tree Search (MCTS): Generating a tree of candidate steps where a Process Reward Model (PRM) scores every intermediate thought, allowing the inference engine to prune dead branches and back-track.
The Economic Transformation of the Inference Business Model
This architectural shift completely rewires cloud economics:
1. Elastic Pricing per Query: Historically, an API call cost a fixed fraction of a cent based on static input and output lengths. With test-time compute, a user or agent can dynamically specify a "reasoning effort budget" (e.g. 1,000 tokens for a trivia question vs. 64,000 tokens for a formal cryptographic audit), dynamically trading computational dollars for accuracy.
2. Capital Rebalancing: Datacenter capacity is rebalancing from massive synchronous pretraining runs (which require thousands of GPUs linked by zero-latency InfiniBand fabrics) to massive distributed inference server farms running speculative decoding and tree searches across commodity networks.