1. Theoretical Motivation & Foundations
Pretraining scaling is encountering data bottlenecks. The new frontier of AI capabilities is test-time compute: spending additional FLOPs during generation to search, evaluate multiple trajectories, and verify steps. We analyze search strategies, Process Reward Models (PRMs), and self-correction verification loops.
2. Mathematical Formulations & Derivations
The governing analytical formulations and proof frameworks for this module:
Inference Scaling Formula:
Compute_{infer} = N_{samples} × Depth_{search} × Verifier_{FLOPs}
Process Reward Step Validation:
R_{step} = P(step_i \text{ is correct} | x, step_{1..i-1})
Monte Carlo Tree Search Value Backprop:
Q(s, a) = (1 / N(s, a)) ∑_{i=1}^{N(s, a)} r_i
3. From-Scratch Reference Implementation
Executable, production-tested reference code without magic libraries:
import heapq
class BeamSearchReasoner:
def __init__(self, beam_width=5):
self.beam_width = beam_width
def search(self, initial_state, step_generator, verifier):
beam = [(0.0, initial_state)] # (negative_score, state)
for _ in range(10): # max reasoning depth
candidates = []
for score, state in beam:
next_steps = step_generator(state)
for step in next_steps:
step_score = verifier(state, step)
candidates.append((score + step_score, step))
beam = heapq.nlargest(self.beam_width, candidates, key=lambda x: x[0])
return beam[0][1]
4. Systems Complexity & Memory Footprint
Test-time compute transforms static LLMs into search engines over thought space. Latency increases from hundreds of milliseconds to tens of seconds in exchange for mathematical reliability.
5. Canonical Literature & Primary Research
Original research papers and foundational texts recommended for advanced study:
- Yao, S., et al. (2023). Tree of Thoughts: Deliberate Problem Solving with Large Language Models. NeurIPS.
- Lightman, H., et al. (2023). Let's Verify Step by Step. OpenAI Research.