1. Theoretical Motivation & Foundations

Evaluating frontier reasoning models has transitioned from static multiple-choice questions (MMLU, GSM8K) to dynamic, execution-based evaluation harnesses. Static benchmarks saturate rapidly and suffer from severe contamination. Modern gold-standard benchmarks measure verifiable outcomes: SWE-bench Verified evaluates an agent's ability to resolve real GitHub issues from production Python repositories inside isolated Docker execution sandboxes with unit-test verification; GPQA Diamond tests graduate-level physics, chemistry, and biology questions specifically written to be Google-proof, defeating experts outside their specific narrow domain. In subjective evaluation, LLM-as-a-Judge architectures deploy frontier models to score candidate outputs, requiring strict position-bias calibration, chain-of-thought grading rubrics, and Cohen's Kappa inter-rater agreement validation against human expert panels.

2. Mathematical Formulations & Derivations

The governing analytical formulations and proof frameworks for this module:

Pass@k Metric Formulation (n samples per problem, c correct solutions): pass@k = E[1 - \binom{n - c}{k} / \binom{n}{k}] Cohen's Kappa Inter-Annotator Agreement (LLM Judge vs. Human Expert): κ = (P_o - P_e) / (1 - P_e) where P_o is relative observed agreement and P_e is hypothetical chance agreement. SWE-bench Verification Criterion: Solve(Issue) = 1 if (Fail_to_Pass_Tests(Patch) == PASSED and Pass_to_Pass_Tests(Patch) == PASSED) else 0

3. From-Scratch Reference Implementation

Executable, production-tested reference code without magic libraries:

# Pass@k Estimator & Evaluation Harness Simulator
import math

def calculate_pass_at_k(n: int, c: int, k: int) -> float:
    # Unbiased pass@k estimator from Chen et al. (HumanEval)
    if n - c < k:
        return 1.0
    return 1.0 - (math.comb(n - c, k) / math.comb(n, k))

sample_sizes = [1, 3, 5, 10]
correct_runs = 4
total_samples = 10

print(f'Results for {correct_runs}/{total_samples} correct solutions:')
for k in sample_sizes:
    p_k = calculate_pass_at_k(total_samples, correct_runs, k)
    print(f'pass@{k} = {p_k * 100.0:.2f}%')

4. Systems Complexity & Memory Footprint

Production evaluation requires reproducible execution environments. Never trust LLM benchmark claims that rely on regex parsing or string matching. True verification requires containerized sandbox test runners where unit tests, linting, and compile steps run independently on verified isolated hardware.

5. Canonical Literature & Primary Research

Original research papers and foundational texts recommended for advanced study:

Jimenez, C. E., et al. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR.
Rein, D., et al. (2023). GPQA: A Graduate-Level Google-Proof Q&A Benchmark. arXiv:2311.12022.
Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS.