Test-Driven Agentic Code Synthesis & Self-Healing Loops

Eliminating hallucinated patches in autonomous coding agents. How test-driven verification, fail-to-pass test generation, stack trace differential analysis, and closed-loop self-repair convert stochastic LLM generations into deterministic software solutions.

1. The Principle of Fail-to-Pass (F2P) Test Reproduction

In industry-standard autonomous evaluation suites like SWE-bench, over 60% of agent failures stem from plausible-sounding hallucinated edits: modifications that look syntactically elegant to a human reviewer but either fail to solve the actual root cause or cause regressions elsewhere in the codebase.

To prevent this, production-grade coding agents enforce a rigorous Test-Driven Development (TDD) invariant:

Step 1: Ingest Issue & Generate Minimal Reproducing Test

Before touching any production source file, the agent generates a standalone test case that directly asserts the buggy behavior described in the issue specification.

Step 2: Execute in Sandbox & Confirm Failure (F2P Verification)

The reproduction test is executed against the unpatched codebase. The test must fail with the expected exception (e.g. AssertionError, KeyError). If the test passes prior to editing, the test itself is invalid and must be discarded.

Step 3: Synthesize Surgical Implementation Patch

The agent edits the codebase with the strict goal of making the reproducing test pass while preserving all existing pass-to-pass (P2P) baseline tests.

2. The Self-Healing Feedback Loop & Convergence Dynamics

When a generated patch fails the test suite, the agent does not start from scratch. Instead, it enters a closed-loop Markovian self-repair state:

S_{t+1} = \mathcal{A}(S_t, \Delta_{\text{code}}, \mathcal{E}(\text{stderr}))

Where \(S_t\) is the agent's context state at step \(t\), \(\Delta_{\text{code}}\) is the candidate diff, and \(\mathcal{E}(\text{stderr})\) is the pruned execution feedback. The probability of resolving an issue within \(k\) reflection rounds scales according to an exponential decay of residual entropy:

P(\text{Resolved} \le k) = 1 - \prod_{i=1}^k (1 - p_i)

Empirical benchmarks show that \(p_1 \approx 0.38\), \(p_2 \approx 0.24\), and \(p_3 \approx 0.12\). Beyond \(k = 4\) iterations, the likelihood of convergence collapses rapidly due to context window contamination, making strict backtrack pruning essential.

3. Stack Trace Differential Pruning

Raw terminal output from test runners (e.g. pytest) often spans hundreds of lines of framework internals, mocking wrappers, and environmental boilerplate. Dumping raw tracebacks into the prompt window wastes thousands of tokens and distracts the agent's attention heads.

Agents utilize AST-guided traceback parsers that isolate:

The Triggering Frame: The exact file, line number, and AST expression within the user's codebase where the failure originated.
The Exception Value: The precise error message and operand types (e.g., TypeError: unsupported operand type(s) for +: 'int' and 'str').
Local Variable State: The values of active variables in the top frame at the moment of exception raise.

4. Pass-to-Pass (P2P) Regression Prevention

A fix that resolves the issue but breaks three adjacent features represents a negative net value. To enforce zero regression, the test harness evaluates candidate patches against two distinct test sets:

\mathcal{L}_{\text{eval}}(\Delta) = \begin{cases} 1 & \text{if } \forall t \in T_{\text{F2P}}, \text{Pass}(t, \Delta) \land \forall t' \in T_{\text{P2P}}, \text{Pass}(t', \Delta) \\ 0 & \text{otherwise} \end{cases}

If any test in \(T_{\text{P2P}}\) fails, the patch is immediately rejected, and the traceback of the regressed test is supplied to the agent as high-priority negative feedback.

5. Production Python Implementation: Autonomous Self-Healing Harness

The following standalone Python engine implements an automated test-runner and self-repair loop with execution timeouts, traceback extraction, and rollback capability:

import subprocess
import sys
import tempfile
import os
from typing import Tuple, Optional

class AgentTestHarness:
    """
    Executes code in an isolated subprocess, captures return codes and tracebacks,
    and manages iterative self-repair attempts up to a maximum round limit.
    """
    def __init__(self, max_retries: int = 3, timeout_sec: int = 15):
        self.max_retries = max_retries
        self.timeout_sec = timeout_sec

    def execute_test(self, script_path: str) -> Tuple[bool, str]:
        """Runs test script and extracts concise failure diagnostics."""
        try:
            res = subprocess.run(
                [sys.executable, script_path],
                capture_output=True,
                text=True,
                timeout=self.timeout_sec
            )
            if res.returncode == 0:
                return True, "All assertions passed successfully."
            else:
                # Prune stderr to isolate traceback exception
                lines = res.stderr.strip().splitlines()
                pruned = "\n".join(lines[-10:]) if len(lines) > 10 else res.stderr
                return False, f"Test failed (exit {res.returncode}):\n{pruned}"
        except subprocess.TimeoutExpired:
            return False, f"Execution timed out after {self.timeout_sec}s."

    def run_repair_loop(self, initial_code: str, test_code: str) -> Tuple[bool, str, int]:
        """Simulates autonomous test-driven self-healing."""
        current_code = initial_code
        with tempfile.TemporaryDirectory() as tmpdir:
            test_file = os.path.join(tmpdir, "test_target.py")
            
            for attempt in range(1, self.max_retries + 1):
                # Write current code combined with test suite
                with open(test_file, "w") as f:
                    f.write(current_code + "\n\n" + test_code)
                
                success, diagnostic = self.execute_test(test_file)
                if success:
                    return True, current_code, attempt
                
                print(f"[Attempt {attempt} Failed] Diagnostic: {diagnostic.splitlines()[-1]}")
                # Autonomous agent patch reflection step (simulated fix for demonstration)
                current_code = self.agent_repair_patch(current_code, diagnostic)
                
            return False, current_code, self.max_retries

    def agent_repair_patch(self, code: str, diagnostic: str) -> str:
        """Applies targeted patch based on parsed diagnostic."""
        if "ZeroDivisionError" in diagnostic:
            return code.replace("return a / b", "return a / b if b != 0 else 0.0")
        return code

if __name__ == "__main__":
    buggy_implementation = """
def compute_ratio(a: float, b: float) -> float:
    return a / b
"""
    f2p_test_suite = """
assert compute_ratio(10, 2) == 5.0
assert compute_ratio(10, 0) == 0.0  # Fails with ZeroDivisionError
print("TESTS_PASSED")
"""
    harness = AgentTestHarness(max_retries=3)
    passed, repaired_code, attempts = harness.run_repair_loop(buggy_implementation, f2p_test_suite)
    print(f"\nRepair Result: {'PASSED' if passed else 'FAILED'} in {attempts} attempt(s)")
    if passed:
        print(f"Repaired Implementation:\n{repaired_code.strip()}")

Next in Curriculum

Level 609: MicroVM Sandboxing & Secure Multi-Tenant Tool Execution

Explore hardware-level virtualization, Firecracker microVMs, gVisor syscall interception, and seccomp-bpf filters that isolate untrusted autonomous agent code execution.

Proceed to Level 609 →