Hugues Clouâtre - AI & Platform Engineering

Open-Weight LLMs Reach the Structured Output Quality Ceiling

Hugues Clouâtre — Fri, 10 Jul 2026 10:55:00 GMT

Three models entered a pre-registered blind evaluation: Gemma 4 26B-A4B at $0.00042 per run, Claude Haiku 4.5 at $0.0119, and Claude Sonnet 5 at $0.0399. Across 30 samples and 8 rubric criteria, all three scored identically on 7 criteria. The single divergence on criterion C8 traced entirely to token-limit truncation, not to any reasoning failure. When quality converges at the ceiling, cost is the only rational differentiation axis.

That cost gap is not marginal. The 28x spread between Gemma 4 and Haiku 4.5, and the 95x spread between Gemma 4 and Sonnet 5, translate directly to operational scale. A pipeline processing one million structured-output calls per month costs $39,900 against Sonnet 5, $11,900 against Haiku 4.5, and $420 against Gemma 4 26B-A4B via managed API; this represents a monthly delta of $39,480 for identical rubric performance on this task class. The MMLU gap between the best open and closed models shrank from 17.5 percentage points in late 2023 to effectively zero by early 2026. For structured-output pipelines specifically, that convergence appears to be complete.

Two additional forces make this moment structurally different from prior open-weights cycles. Apache 2.0 licensing removes the license-review step that once slowed open-model procurement, though self-hosting transfers other friction onto the deploying team. These hurdles include GPU qualification, indemnification gaps, and inference engineering ownership. EU AI Act GPAI enforcement powers activate August 2, 2026, making data residency a compliance requirement rather than an operational preference. Self-hosted open-weight models satisfy data residency; so do managed closed-model providers with EU-resident infrastructure.

Contents

What Does It Mean for Structured-Output Quality to Hit a Ceiling?
How Was Experiment 9 Designed to Avoid Benchmark Gaming?
What Did the Blind Evaluation Reveal About Cost Versus Quality?
- What Does a 95x Cost Spread Mean at Scale?
When Does Switching to Open-Weight Models Make Economic Sense?
Why Is C8 a Token Budget Problem, Not a Reasoning Problem?
- What Does C8 Mean for Self-Hosted vs. Managed Inference?
How Wide Is the Remaining Gap Between Open and Closed Models?
- What Are the Scope Boundaries?
When Do Closed Models Still Win?
What Does Apache 2.0 Licensing Change for Enterprise Procurement?
- What Does Self-Hosting Provide for Audit Trails?
How Does EU AI Act Enforcement Shift the Data Residency Calculus?
- Does Using a Third-Party API Create a Data Residency Risk?
Takeaways
References

What Does It Mean for Structured-Output Quality to Hit a Ceiling?

The quality ceiling is the finding: Gemma 4 26B-A4B matched Claude Haiku 4.5 and Claude Sonnet 5 on 7 of 8 rubric criteria at 95x lower cost. Benchmark scores can mislead in two directions, and understanding what “ceiling” means here determines whether the result generalizes to your workload.

JSON Pass Rate is not a discriminating metric at modern quality levels (Singh et al., 2026): almost every current model clears 95% or above, yet valid JSON and correct JSON diverge by 15-30 points on every frontier model (Interfaze AI, 2026). The SOB (Structured Output Benchmark) leaderboard shows the top six models within one point of each other on overall score, with structural metrics such as Path Recall and Type Safety reading 99% while 20-30% of leaf values remain wrong.

Quality ceiling, in this context, means the task-relevant rubric criteria are saturated across models of materially different cost tiers. It does not mean all models are interchangeable for all tasks. Reasoning-heavy chains, multi-step tool orchestration, and complex disambiguation tasks continue to stratify models. The ceiling applies to structured extraction and generation against a defined schema, the domain where Exp9 was conducted.

Model size is not a reliable predictor of value accuracy: open-weight GLM-5.1, GLM-4.7, and Qwen3.5-35B beat GPT-5, Claude-Opus-4.7, and Claude-Sonnet-4.6 on unified Value Accuracy in the SOB benchmark (Interfaze AI, 2026). Exp9 adds a cost-controlled data point: when Gemma 4 26B-A4B, a mixture-of-experts (MoE) model activating 4 billion of its 26 billion parameters per forward pass, operates at the same rubric level as models priced 28x and 95x higher, the ceiling is the signal.

How Was Experiment 9 Designed to Avoid Benchmark Gaming?

Exp9 (Clouatre, 2026) applied a pre-registered rubric before any model results were observed. The rubric covered eight criteria, C1 through C8, capturing schema compliance, value accuracy, field completeness, format adherence, instruction following, edge-case handling, output coherence, and response length within budget. All three models were evaluated under a blind protocol: n=10 samples per model (30 total), routed through the OpenRouter endpoint, with no post-hoc rubric adjustments.

The pre-registration constraint is the key methodological safeguard. Rubric criteria were fixed before evaluation commenced, precluding the selection of metrics that favor any particular model after results were visible. LLM-judge scoring was applied to criteria C6 and C7, using a separate model as the evaluator. Haiku 4.5 and Sonnet 5 each scored 20/20 on those criteria; Gemma 4 scored 18/20, a two-point difference that appears in the LLM-judge outputs but not in the objective scores for C1-C5 or C8.

On criteria C1 through C7, all three models scored identically: pass on every objective criterion. The LLM-judge scores for C6 and C7 showed Haiku 4.5 and Sonnet 5 at 10/10 each (20/20 judge points) and Gemma 4 at 18/20, with one missed point on C6 and one on C7, both from run-91. That two-point gap is within LLM-judge variance and does not represent a functional quality difference on the task.

**Table 1:** Exp9 per-model results (n=10 per model, 30 total, pre-registered rubric). Sonnet 5 at intro rates ($2/$10 per MTok input/output, valid through Aug 31, 2026); standard is $3/$15. Cost/run computed from Exp9 mean token counts: 1,723 input and 3,646 output tokens per Sonnet 5 call; 1,128 input and 996 output for Gemma 4.
Model	C6/C7 judge	C8	Cost/run	vs Gemma 4
Gemma 4 26B-A4B	18/20	Truncation (10%)	$0.00042	baseline
Haiku 4.5	20/20	Pass	$0.0119	28x
Sonnet 5	20/20	Truncation (20%)	$0.0399	95x

What Does a 95x Cost Spread Mean at Scale?

Seven of eight rubric criteria produce no differentiation. Selecting Sonnet 5 over Gemma 4 for structured-output workloads matching the Exp9 task profile delivers no measurable quality gain at 95x the cost. The one criterion where differentiation appeared is addressed in the following section.

Code Snippet 1: Monthly cost calculator using OpenRouter 2026-07-06 pricing

COST_PER_RUN = {
    "gemma-4-26b-a4b": 0.00042,
    "claude-haiku-4-5": 0.01190,
    "claude-sonnet-5":  0.03990,
}

def monthly_cost(model: str, runs_per_month: int) -> float:
    """Return total monthly cost in USD."""
    return COST_PER_RUN[model] * runs_per_month

def cost_per_run(model: str) -> float:
    """Return per-run cost in USD."""
    return COST_PER_RUN[model]

# Example: 1 million runs/month
for m, c in COST_PER_RUN.items():
    print(f"{m}: ${monthly_cost(m, 1_000_000):,.0f}/month")scripts/model_cost_convergence.py

When Does Switching to Open-Weight Models Make Economic Sense?

The cost decision tree below maps monthly token volume to the go/no-go threshold. At Exp9’s mean of ~2,100 tokens per call, 50 million tokens per month corresponds to roughly 24,000 structured-output calls. Under that threshold, the API cost delta does not justify the operational overhead of self-hosting; GPU infrastructure, continuous batching tuning, and MLOps staffing add costs that typically exceed token savings at low volume. Above 500 million tokens per month, the $530 vs $18,000 per month comparison makes the case; the intermediate path is a managed open-weight inference provider (OpenRouter, Together AI, Baseten), not bare-metal self-hosting.

Figure 1: Cost decision tree by monthly token volume. Under 50M tokens/month the cost delta does not justify a switch; the 50M-500M band requires quality and latency evaluation (see Figure 2); above 500M the open-weights case is strong.

Why Is C8 a Token Budget Problem, Not a Reasoning Problem?

C8 failures (response length within budget) in Gemma 4 26B-A4B were exclusively token-limit truncations. Outputs were structurally correct and semantically accurate until the token budget was exhausted; responses did not produce incoherence or malformed output. They simply stopped. Haiku 4.5 had no C8 failures. Sonnet 5 also hit the 4096-token cap in run-112 and run-113; its mean output of 3,652 tokens per run versus 996 for Gemma 4 makes it the most verbose model and the one most exposed to truncation risk at higher completion budgets.

What Does C8 Mean for Self-Hosted vs. Managed Inference?

The fix is a configuration change, not a model substitution. Increasing the max_tokens parameter to the 95th-percentile response length eliminates the failure mode. That said, C8 illustrates a broader operational reality: production inference is not a configuration exercise. Naive self-hosted deployments achieve 30 to 40 percent GPU utilization; a production-tuned stack using continuous batching and PagedAttention reaches 80 to 90 percent, a 2 to 3x cost differential on identical hardware (Kiely, 2026). C8 on a managed API is an operational parameter; C8 on a self-hosted stack is the entry point to inference engineering.

Code Snippet 2: max_tokens configuration fix for C8, targeting the 95th-percentile response length

import httpx

# Before: default completion limit caused C8 truncation failures
# payload_before = {"model": "google/gemma-4-26b-a4b", "max_tokens": 1024}

# After: set max_tokens to 95th-percentile observed response length
# Heuristic: sample 100 runs, take p95 token count, round up to nearest 512
payload = {
    "model": "google/gemma-4-26b-a4b",
    "max_tokens": 4096,  # p95 for Exp9 task distribution
    "messages": [{"role": "user", "content": "..."}],
}

response = httpx.post(
    "https://openrouter.ai/api/v1/chat/completions",
    json=payload,
    headers={"Authorization": "Bearer $OPENROUTER_API_KEY"},
)scripts/openrouter_invoke.py

How Wide Is the Remaining Gap Between Open and Closed Models?

The structured-output quality ceiling does not imply open-weights have reached parity on all tasks. Open-weight models trail closed-source frontier by approximately 4 months as of May 2026, measured by Epoch AI’s Epoch Capabilities Index (ECI; Epoch AI, 2026). That gap is real and worth quantifying before deployment decisions.

The reasoning gap between open and closed models measures approximately 13 points on structured benchmarks as of June 2026 (LLM Stats, 2026). This gap manifests primarily in multi-step reasoning, adversarial prompts, and tasks requiring synthesis across long contexts. Structured-output extraction against a defined schema, the Exp9 task class, falls below the threshold where that gap becomes operational.

Figure 2: Quality and latency gate. P50 latency requirement and rubric score thresholds determine whether fine-tuning is needed before open-weights promotion.

What Are the Scope Boundaries?

Two scope boundaries apply to Exp9 specifically. Tool calling was not tested; Gemma 4 26B-A4B scores 68.2% on tau2-bench agentic benchmarks vs higher scores on closed alternatives, and that gap has not been evaluated under the same controlled conditions. Exp9 results generalize to structured extraction and generation tasks; extension to multi-step agentic pipelines requires separate experimentation.

The practical heuristic: use open-weights for tasks where the Exp9 rubric class applies (structured extraction, schema validation, templated generation). Evaluate closed frontier models for tasks requiring the top 13 reasoning points or tool-orchestration depth that currently differentiate them. Five independent open model families reached frontier quality simultaneously by early 2026 (LetsDataScience, 2026), confirming the convergence is structural rather than attributable to any single model or provider.

When Do Closed Models Still Win?

Five conditions favor closed-model APIs regardless of token-cost arithmetic:

No inference engineering team. Self-hosting a 26B MoE model in production requires continuous batching configuration, KV-cache tuning, GPU health monitoring, and model rollback procedures, at minimum 0.5 FTE MLOps. Without that capability, infrastructure cost exceeds token savings.
Contractual SLA required. Self-hosted deployments carry no vendor uptime guarantee and no incident escalation path. Closed-model APIs provide contractual SLAs; for production systems with business continuity obligations, that is a hard requirement.
Interactive latency below 2,000 ms P50 required. Self-hosted MoE inference on commodity hardware does not reliably meet sub-2-second P50 targets. Closed API endpoints with dedicated capacity are the appropriate choice for latency-sensitive workloads.
Reasoning-heavy or agentic tasks. Multi-step tool orchestration, adversarial disambiguation, and long-context synthesis fall in the region where the benchmark gap translates to production failures. Gemma 4 26B-A4B scores 68.2% on tau2-bench agentic benchmarks versus higher scores on closed alternatives.
GPU generation mismatch. Open-weight model releases are coupled to GPU generation cycles; a successor model may ship kernels optimized for a different architecture, requiring months of porting before stable performance on the incumbent stack (Kiely, 2026).

What Does Apache 2.0 Licensing Change for Enterprise Procurement?

All four Gemma 4 models ship under the Apache 2.0 license with native function calling and structured JSON output support (Google DeepMind, 2026). Apache 2.0 permits commercial use, modification, and redistribution without requiring downstream source disclosure. That removes one procurement bottleneck. It does not provide vendor indemnification for model outputs, GPU procurement lead-time relief (H100 SXM5 lead time is 2-6 weeks), or a patch cadence for weight-level security vulnerabilities; those responsibilities transfer to the deploying organization.

What Does Self-Hosting Provide for Audit Trails?

The audit trail advantage is concrete. A self-hosted model with a SHA-256 hash of the weights, version-pinned dependencies, and deterministic inference configuration produces a reproducible artifact satisfying change-management requirements. Cloud API endpoints provide no equivalent: model versions can be updated by the provider without notice, and the inference path is opaque. Open-weight GPAI models released under a free open-source license are exempt from Annex XI technical documentation and Annex XII downstream provider information requirements under the EU AI Act (Alation, 2026), though copyright compliance and training data summaries are still required.

**Table 2:** Regulatory posture by deployment model
Regulation	Open-weights (Apache 2.0)	Proprietary API
EU AI Act GPAI (Aug 2026)	Exempt from Annex XI/XII documentation	Full documentation required
Audit trail	SHA-256 weight pin, reproducible	Provider-controlled, opaque
Data residency	Self-hosted on EU infrastructure	Cross-border transfer triggers GDPR Art. 46

How Does EU AI Act Enforcement Shift the Data Residency Calculus?

GPAI enforcement powers activate August 2, 2026 under Articles 101 and 113. The fine schedule is compact:

**Table 3:** EU AI Act enforcement schedule (GPAI, 2026)
Enforcement Date	Provision	Fine Standard	Fine Serious
Aug 2, 2026	GPAI obligations (Art. 101/113); Transparency (Art. 50)	EUR 15M or 3% global turnover	EUR 35M or 7% global turnover
Dec 2, 2027	High-risk systems (Art. 9-17)	EUR 15M or 3% global turnover	EUR 35M or 7% global turnover

The serious-violation ceiling exceeds GDPR maximums (Alation, 2026). US-based institutions produced 40 notable AI models in 2024 vs 15 from China and 3 from Europe, meaning enforcement pressure falls disproportionately on US cloud providers serving European organizations (EWSolutions, 2026).

Does Using a Third-Party API Create a Data Residency Risk?

Data processed through a third-party API in a foreign jurisdiction creates a transfer record triggering GDPR Article 46 analysis. Data processed through a self-hosted model in the deploying organization’s own infrastructure does not. Managed closed-model providers with EU-resident deployments (Azure OpenAI EU data zones, Vertex AI EU regions, Bedrock Frankfurt) satisfy the same Article 46 requirement without the infrastructure burden, at the cost of weight-level auditability. The choice is between operational simplicity and documentation control, not between compliant and non-compliant.

Takeaways

Quality ceiling reached for this task class: Gemma 4 26B-A4B matched Claude Haiku 4.5 and Claude Sonnet 5 on C1-C7 across n=30 blind trials (10 per model); results do not generalize to reasoning-heavy, agentic, or latency-sensitive workloads.
C8 is a configuration problem, not a capability problem: Token-limit truncations are correctable by adjusting completion budgets; they do not indicate a reasoning gap.
API token cost favors open weights above volume thresholds; TCO requires separate modelling. The 28x-95x spread reaches $39,480 per million monthly runs at identical rubric quality via managed API. Self-hosted TCO adds GPU infrastructure, 0.5 or more FTE MLOps, and idle-utilization loss; break-even sits at 35 to 50 million tokens per month for a single-GPU MoE deployment; multi-GPU open-weight models require ~256M tokens per month against a premium API (LetsDataScience, 2026).
Reasoning gap exists and is bounded: Open-weights trail closed frontier by approximately 4 months and 8 ECI points; that gap surfaces in reasoning-heavy tasks, not in the Exp9 structured-output class.
Apache 2.0 self-hosting is a viable EU AI Act compliance path, not the only one. SHA-256 weight pins and GPAI open-source exemptions provide documentation control; managed EU-resident providers satisfy the same data-residency requirement with contractual SLAs and no inference infrastructure ownership.

References

Alation, “EU AI Act Compliance: A Practical Guide for 2026-2027” (2026) — https://www.alation.com/blog/eu-ai-act-compliance-guide
Clouatre, H., “Exp9: Gemma 4 26B-A4B vs Haiku 4.5 vs Sonnet 5 structured-output blind evaluation” (2026) — https://github.com/clouatre-labs/llm-agent-experiments#replication-study-experiment-9
DeepInfra, “Gemma 4 on DeepInfra: Fast & Scalable Open AI Models” (2026) — https://deepinfra.com/blog/gemma-4-on-deepinfra
Epoch AI, “Open models lag state-of-the-art closed models by 4 months” (2026) — https://epoch.ai/data-insights/open-closed-eci-gap
EWSolutions, “EU AI Act Updates 2026: What US Firms Must Do” (2026) — https://www.ewsolutions.com/eu-ai-act-updates-2026
Google DeepMind, “Gemma 4 model card” (2026) — https://ai.google.dev/gemma/docs/core/model_card_4
Interfaze AI, “Introducing SOB: A Multi-Source Structured Output Benchmark for LLMs” (2026) — https://interfaze.ai/blog/introducing-structured-output-benchmark
Kiely, P., “Inference Engineering” (2026) — https://www.baseten.co/inference-engineering/
LetsDataScience, “Open Source vs Closed LLMs: The 2026 Decision Framework” (2026) — https://letsdatascience.com/blog/open-source-vs-closed-llms-choosing-the-right-model-in-2026
LLM Stats, “LLM Benchmark Leaderboard” (2026) — https://llm-stats.com/
Singh, A. K. et al., “SOB: A Benchmark for Evaluating Structured Output Quality in Large Language Models” (2026) — https://doi.org/10.48550/arXiv.2604.25359

AI Approval Gates: Engineering Oversight at Machine Speed

Hugues Clouâtre — Tue, 07 Jul 2026 11:42:00 GMT

In April 2026, a PocketOS coding agent deleted a production database and its only backup in nine seconds, triggering a 30-hour outage (Crane, 2026). No gate existed to stop it. Tiering approval gates by action reversibility, not risk category, is the structural fix. This post gives a three-tier gate model, the reviewer atrophy research behind it, and the four metrics that tell you when the gates are holding.

Contents

Why Are AI Agents Causing Production Incidents Without Any External Attacker?
Why Does Adding a Human Reviewer Sometimes Make Things Worse?
- Why Debugging Skill Is the Specific Casualty
- What Does DORA Say About External Approval Processes?
What Is Reversibility, and Why Does It Define the Gate Design?
- Why Risk Category Fails as a Gate Axis
- What Does the EU AI Act Require From This Design?
How Does a Three-Tier Reversibility Gate Work in Practice?
- What Happens When an Action Is Mis-Tiered?
How Do You Prevent Reviewer Atrophy From Hollowing Out the Judgment Tier?
How Do You Measure Whether Your Gates Are Holding?
What Should Engineering Leaders Act On?
References

Why Are AI Agents Causing Production Incidents Without Any External Attacker?

Cyera’s dataset of 7,246 publicly reported AI incident records isolates 188 cases where corporate production environments were harmed by the AI system itself (Cyera Research, 2026). These are not adversarial attacks. They are authorized agents completing assigned tasks with insufficient constraint on action scope.

CodeRabbit’s analysis of 470 pull requests found AI-generated code produces 2.74 times more XSS vulnerabilities and 1.57 times more security findings overall (CodeRabbit, 2025). Faros AI’s telemetry across 22,000 developers documents a 242.7% increase in incidents per PR (Faros AI, 2026).

A second threat is behavioral: when an agent is blocked from one execution path, it finds another. An agent prevented from calling an API directly may route the same action through browser automation or a chained tool call. Static permission scopes cannot anticipate every path an agent will discover; runtime enforcement intercepts actions as they execute, not only at task classification, complementing what a blocklist-only model cannot cover.

The production incident rate is not a model quality problem alone. Regulated industries built every governance control on a deterministic assumption: the same input produces the same output, every time. Agents are probabilistic. Every audit trail, approval gate, and access policy in your stack was designed for a world that no longer exists once agents enter it. In practice, operators compounded this by defining permissions at the identity level (what the agent could authenticate to) rather than the action level: what it could execute without human confirmation.

Why Does Adding a Human Reviewer Sometimes Make Things Worse?

The Vaccaro et al. meta-analysis examined 106 experiments across 74 studies and found human-AI combinations were outperformed by either humans or AI alone (Vaccaro et al., 2024). When AI was stronger than the reviewer, adding a human created approval theater: a process that looks like oversight but delivers none.

The Anthropic RCT by Shen and Tamkin sharpens the mechanism: 52 junior engineers randomized to AI-assisted conditions showed 17% lower comprehension scores on debugging tasks (Shen and Tamkin, 2026).

Why Debugging Skill Is the Specific Casualty

Debugging requires tracing causation backward from a symptom without the model’s assistance. When AI supplies those intermediate steps, the engineer’s causal-reasoning muscle goes unexercised. The more AI assistance a reviewer receives in their own daily work, the less that muscle is maintained.

Figure 1: Reviewer atrophy feedback loop and the structural break point.

What Does DORA Say About External Approval Processes?

The 2019 DORA report found organizations using formal external approval processes, including CABs and senior manager sign-off, were 2.6 times more likely to be low delivery performers, with no reduction in change failure rates (DORA, 2019). Heavyweight gates added latency without improving stability. The 2025 DORA report adds the AI dimension: adoption increases delivery instability even as throughput improves (DORA, 2025). 90% of developers now use AI tools; 30% report little to no trust in the output. Instability drives burnout even when individual productivity metrics improve.

AI reintroduces instability at machine speed, and the structural answer applies friction at the reversibility boundary, not uniformly at every change.

What Is Reversibility, and Why Does It Define the Gate Design?

Before tiering gates, scope agent permissions to the minimum required for their task. Agents routinely receive access equivalent to ten times a developer’s typical permissions by default; a reversibility gate operating against that blast radius is correcting for a structural miscalibration, not solving it. Reversibility is the property that determines whether a mistake can be corrected after it is made. A database deletion that destroys the only backup is not reversible; a customer notification already delivered is not.

Risk category requires human judgment to assess and shifts with business context. Reversibility is a structural property of the action, encodable as a constraint and evaluated before execution. No sharp reviewer is required, no model confidence score, and the property does not erode as queue depth increases. Unlike risk category, the classification is binary: an action either has a rollback path or it does not. Tier-based interrupts add the execution-time gate that AGENTS.md policy and repository controls do not provide (AI-Assisted Development).

Why Risk Category Fails as a Gate Axis

In an original benchmark of 60 agentic action scenarios seeded from documented production incidents, a multi-factor risk-label classifier (Classifier A) evaluates each action on severity, blast radius, and authorization level. Blast radius here is measured by data volume and system count; the operationally relevant dimension is system-of-record criticality. An agent writing a log file and an agent with Salesforce write access score identically on volume metrics; only one can corrupt enterprise data at scale through a single hallucinated action. All three signals were low for the missed cases, and the axis cannot encode whether a mistake can be undone: Classifier A missed 13.8%; the reversibility blocklist missed none (Clouatre, 2026). The PocketOS incident is one instance: low risk by any standard rubric, and irreversible (Crane, 2026).

Tassey (2002) found production defect fixes cost two or more orders of magnitude more than design-phase fixes; the cost compounds as downstream state propagates. DO-178C and IEC 61508 encode the same logic: gate on reversibility consequence before execution, not on risk category label (RTCA / EUROCAE, 2011).

Code Snippet 1: Reversibility classifier — actions matching any IRREVERSIBLE_MARKERS property route to Tier 3.

from enum import Enum

class ReversibilityTier(Enum):
    TIER1 = "reversible"
    TIER2 = "bounded_reversible"
    TIER3 = "irreversible"

# Action properties that force Tier 3 regardless of risk label
IRREVERSIBLE_MARKERS = {
    "mutates_persistent_state",
    "external_side_effect",
    "no_rollback_procedure",
}

def classify(action: dict) -> ReversibilityTier:
    if action.get("properties", set()) & IRREVERSIBLE_MARKERS:
        return ReversibilityTier.TIER3
    if action.get("confidence", 1.0) < action.get("threshold", 0.85):  # calibrate threshold to your FP baseline
        return ReversibilityTier.TIER2
    return ReversibilityTier.TIER1gates/reversibility_classifier.py

The three-marker blocklist above was tested against that same 60-scenario benchmark. The Tier 2 confidence threshold implements the SLM-for-governance pattern: a model tuned for the risk/safe decision boundary rather than general language generation. Embedding organizational policy at inference time, rather than retraining per deployment, is the practical path to multi-tenant calibration. Table 1 shows results across three classifier designs; Classifier B’s 96.7% halt rate is a blocklist artifact, not a production threshold.

**Table 1:** Classifier comparison across 60 agentic action scenarios.
Classifier	Design	Miss rate	False-positive rate
A	Multi-factor risk label	13.8%	64.5%
B	Reversibility blocklist	0.0%	93.6%
C	Combined A OR B	0.0%	71.0%

Classifier C reduces that false-positive rate by 22 percentage points at the same zero miss. The difference is statistically supported but should be treated as directional given the pilot scale. Results reflect a single model family (Claude Sonnet 4.6).

What Does the EU AI Act Require From This Design?

Article 14 of the EU AI Act requires that high-risk AI systems allow humans to understand capabilities and limitations, detect and address issues, decide not to use the output, and halt operation (European Parliament, 2024). These obligations enter into force on August 2, 2026, with fines up to 15 million euros or 3% of global annual turnover for non-compliance with high-risk system obligations.

This gate aligns with Article 14 structurally; legal counsel should confirm applicability to specific system classifications. Tier 1 provides the audit trail. Tier 2’s interrupt provides the “decide not to use” surface. Tier 3’s mandatory hold provides “halt operation” where the cost of a mistake is highest. A uniform review process satisfies the letter of the requirement; approval theater at scale means the human is nominally present but substantively absent.

How Does a Three-Tier Reversibility Gate Work in Practice?

The three tiers below govern action consequence, not pipeline security. They are distinct from the security framework in AI-Augmented CI/CD, which controls what context AI receives during code review. Here, tier determines latency budget and reviewer type; risk category informs judgment at Tier 3 but does not select the tier.

**Table 2:** Three-tier reversibility gate model.
Tier	Action type	Gate mechanism
1 — Reversible	Feature flag, config with instant rollback	Automated pass-through + audit log
2 — Bounded	PR merge, dependency update	Confidence-threshold interrupt
3 — Irreversible	Schema change, data mutation, external API call	Mandatory expert hold; four-eyes

The LangGraph interrupt pattern supports four decision types at Tier 2: approve, reject, edit, or respond with additional context (LangChain, 2025). At Tier 3, risk category, business context, and regulatory requirements inform the expert’s judgment; the human brings context the classification cannot encode. The checkpoint is the structural guarantee: when a Tier 3 review takes hours, the agent resumes from exact state rather than replaying the full plan. The Zero-Downtime DNS Migration post documents two named Tier 3 holds in a production workflow, each requiring a reviewable artifact, not a yes/no click.

Code Snippet 2: LangGraph interrupt pattern for Tier 2 and Tier 3 actions.

from langgraph.graph import StateGraph
from langgraph.checkpoint.memory import MemorySaver

def build_gated_graph():
    builder = StateGraph(AgentState)
    builder.add_node("plan", plan_node)
    builder.add_node("human_review", interrupt_node)
    builder.add_node("execute", execute_node)

    def route_by_tier(state):
        tier = classify(state["next_action"])
        if tier == ReversibilityTier.TIER3:
            return "human_review"
        if tier == ReversibilityTier.TIER2:
            if not above_confidence_threshold(state):
                return "human_review"
        return "execute"

    builder.add_conditional_edges("plan", route_by_tier)
    builder.add_edge("human_review", "execute") # resumes from checkpoint 
    return builder.compile(checkpointer=MemorySaver())gates/reversibility_gate.py

What Happens When an Action Is Mis-Tiered?

The classifier in Code Snippet 1 addresses mis-tiering with a blocklist of consequence markers: any action matching even one marker routes to Tier 3 regardless of its assigned risk label. The post-approval incident rate metric in Table 3 closes the feedback loop: Tier 1 post-approval incidents trigger automatic reclassification of that action type to Tier 2.

A harder case is agent chaining: multiple agents, each operating within individually scoped permissions, executing in sequence to achieve what no single agent is authorized to do alone. The gate model handles this by classifying each action independently at execution time. Each step that crosses a reversibility marker halts; the blast radius of the chain is bounded by whichever step first triggers Tier 3.

How Do You Prevent Reviewer Atrophy From Hollowing Out the Judgment Tier?

That same comprehension gap concentrates precisely in Tier 3 review tasks. Three structural controls counter reviewer atrophy: rotate Tier 3 reviewers through agent-free work to maintain baseline judgment; require a comprehension check before approval (the reviewer explains the action, the agent’s reasoning, and the rollback procedure); and audit Tier 3 decisions against outcomes, moving reviewers whose approvals correlate with post-deployment incidents to supervised review.

None of these controls are self-monitoring; the next section defines the four metrics that detect when they are failing.

How Do You Measure Whether Your Gates Are Holding?

Four metrics provide early warning before an incident and close the feedback loop between agent throughput and stability. The first two are leading indicators; the last two confirm whether tier classifications remain valid.

**Table 3:** Gate health metrics. Calibrate thresholds to your baseline.
Metric	Source	Alert signal
Tier 3 queue depth	Gate event log	Rising faster than reviewer capacity
Tier 2 trigger rate	Gate event log	Rising rate — agent scope expanding
Post-approval incident rate	Incident + gate log	Any Tier 3 incident triggers reviewer audit and tier classification review
Tier 1 rollback success	Deployment log	Any failure reclassifies action to Tier 2

A rising Tier 2 trigger rate signals task framing investigation, not threshold adjustment: agent scope is expanding. A Tier 1 rollback failure means the action was never truly reversible and must be reclassified. In Figure 2, the Hold branch fires on gate.tier3_queue_depth or gate.tier2_trigger_rate breaches; the Alert branch fires when gate.post_approval_incident is true or gate.tier1_rollback_success drops below 100%.

Figure 2: Four gate health signals and their alert routing.

What Should Engineering Leaders Act On?

Engineering leaders cannot solve reviewer atrophy with cultural mandates; it requires architectural constraints. Organizations that solve agentic governance systematically deploy faster than those that solve it ad hoc. Startups ship agents in days; enterprises stall for months because their committees cannot iterate faster than the technology evolves. The governance gap is a deployment velocity problem as much as a compliance one.

Scope permissions, then audit action space. Before classifying actions, scope each agent to the minimum permissions its task requires. Then identify every action that mutates persistent state, fires external side effects, or has no rollback procedure, and blocklist it into Tier 3. Code Snippet 1 provides the starting classifier.
Implement checkpointed interrupts. Deploy LangGraph’s durable checkpointer (Code Snippet 2) for Tier 2 and Tier 3 gates. The checkpoint survives reviewer session boundaries.
Instrument the four gate metrics. Add the OpenTelemetry span in Code Snippet 3: Tier 3 queue depth, Tier 2 trigger rate, post-approval incident rate, and Tier 1 rollback success are your early-warning system.

Code Snippet 3: Gate health OTel (OpenTelemetry) span. The four highlighted attributes are the early-warning signals; tier_misclassification_detected triggers immediate review.

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def record_gate_event(action: dict, tier: str, outcome: str) -> None:
    with tracer.start_as_current_span("gate_health") as span:
        span.set_attribute("gate.tier", tier)
        span.set_attribute("gate.outcome", outcome)
        span.set_attribute("gate.tier3_queue_depth",
            get_queue_depth("tier3"))
        span.set_attribute("gate.tier2_trigger_rate",
            get_trigger_rate("tier2"))
        span.set_attribute("gate.post_approval_incident",
            outcome == "incident_post_approval")
        span.set_attribute("gate.tier1_rollback_success",
            get_rollback_success_rate("tier1"))
        span.set_attribute("gate.tier_misclassification_detected",
            outcome == "incident" and tier == "tier1")gates/gate_health_span.py

The approval gate methodology described in this post is the subject of Canadian patent application CA 3315358, filed Jun 17, 2026 (CIPO).

References

Clouatre, H., “Reversibility Benchmark: Risk-Label vs. Reversibility Gate on 60 Agentic Action Scenarios” (2026) — https://doi.org/10.5281/zenodo.20644042 — https://github.com/clouatre-labs/reversibility-benchmark
CodeRabbit, “State of AI vs Human Code Generation Report” (2025) — https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report
Crane, J. (via Cerbos), “PocketOS AI Coding Agent Deleted a Production Database in 9 Seconds” (2026) — https://www.cerbos.dev/blog/ai-coding-agent-deleted-a-production-database-in-9-seconds
Cyera Research, “Agent-Inflicted Damage: Inside the Real-World Failures of Enterprise AI Systems” (2026) — https://www.cyera.com/research/agent-inflicted-damage-inside-the-real-world-failures-of-enterprise-ai-systems
DORA (Google), “2019 Accelerate State of DevOps Report” (2019) — https://dora.dev/research/2019/dora-report/
DORA (Google), “State of AI-assisted Software Development 2025” (2025) — https://dora.dev/research/2025/dora-report/
European Parliament and Council of the EU, “Regulation (EU) 2024/1689 (AI Act), Article 14: Human Oversight” (2024) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng
Faros AI, “AI Engineering Report 2026: The Acceleration Whiplash” (2026) — https://www.faros.ai/research/ai-acceleration-whiplash
LangChain / LangGraph, “Human-in-the-Loop” (2025) — https://langchain-ai.github.io/langgraph/concepts/human_in_the_loop/
MindStudio, “How a Data Science Team Achieved Massive ROI with AI Agents” (vendor case study, 2025) — https://www.mindstudio.ai/blog/data-science-roi
RTCA / EUROCAE, “DO-178C: Software Considerations in Airborne Systems and Equipment Certification” (2011) — https://www.rtca.org/content/do-178c; see also FAA Advisory Circular AC 20-115D — https://www.faa.gov/documentLibrary/media/Advisory_Circular/AC_20-115D.pdf
Shen, J.H. and Tamkin, A. (Anthropic), “How AI Impacts Skill Formation” (2026) — https://doi.org/10.48550/arXiv.2601.20245
Tassey, G., “The Economic Impacts of Inadequate Infrastructure for Software Testing,” NIST (2002) — https://www.nist.gov/system/files/documents/director/planning/report02-3.pdf
Vaccaro et al. (MIT Center for Collective Intelligence), “When combinations of humans and AI are useful: A systematic review and meta-analysis,” Nature Human Behaviour (2024) — https://www.nature.com/articles/s41562-024-02024-1

AI Adoption in Engineering: Breaking the 50% Plateau

Hugues Clouâtre — Mon, 01 Jun 2026 16:25:00 GMT

Purpose-built AI tooling cuts per-task cost 21-68%, yet only 13% of organizations have AI agents broadly integrated into workflows (BCG, 2025) and self-reported velocity diverges from measured outcomes by 39 percentage points (Becker et al., 2025). Frontline use has stagnated at 51% across three annual editions of BCG’s survey. This post gives a three-cohort segmentation model, a four-phase operating model, and the benchmark data behind those numbers.

Contents

Why Do Enterprise AI Programs Stall at 50%?
- The Self-Reporting Trap
- The Deployment-Before-Instrumentation Pattern
Who Is Actually Blocked, and Why?
- Three Cohorts, Three Interventions
- The Policy Document as Unlocker
What Does Purpose-Built Tooling Change?
- Benchmark Methodology
- Benchmark Results
How Do You Instrument Before You Deploy?
- Five Numbers Before Day One
- What Generic Monitoring Misses
What Operating Model Sustains Adoption?
Which AI Adoption Plays Waste Budget?
How Do You Measure Adoption, Not Just Activity?
- The Five-Number Rule
- The KPI Scorecard
What Should Engineering Leaders Do Next?
References

Why Do Enterprise AI Programs Stall at 50%?

The frontline adoption stall is not primarily a culture problem; it is a measurement problem. When leaders rely on survey confidence instead of workflow telemetry, they misclassify blocked engineers as resistant.

The Self-Reporting Trap

The METR perception gap explains why the stall persists. A 2025 randomized controlled trial (Becker et al., 2025) across experienced open-source developers found that AI tools caused tasks to take 19% longer, while developers simultaneously reported believing AI made them 20% faster. A February 2026 follow-up found likely speedup from late-2025 tools but with severe selection effects, and the perception gap finding stands.

The Deployment-Before-Instrumentation Pattern

When programs stall, the default response is a new communications campaign, a lunch-and-learn series, or an expanded license rollout. None of these address the actual distribution of blockers across the engineering population. Gartner predicts over 40% of agentic AI projects will be canceled by end of 2027 due to escalating costs, unclear business value, or inadequate risk controls. The pattern is consistent: organizations deploy before they instrument, then cannot make a defensible scale-or-stop decision.

DORA 2025 found AI impact depends on the quality of the underlying organizational system; platform maturity gates adoption returns.

Who Is Actually Blocked, and Why?

Before any intervention, segment the engineering population by blocker type. Three cohorts emerge, each with a distinct barrier type independently validated across enterprise deployments (OECD/BCG, 2025); in larger organizations, the cohort map also surfaces AI expertise concentration risk.

Three Cohorts, Three Interventions

The Blocked cohort is the highest-leverage and least-served by standard programs. Policy uncertainty, not skepticism, is the barrier.

Figure 1: Three cohorts mapped to three barrier types. Distribution is illustrative; ranges informed by Cisco (2024) data.

The Policy Document as Unlocker

A one-page tier policy unblocks engineers faster than another launch event because it answers the operational questions they face at commit time: which data category, which approved provider agreement, and which approval path. The three tiers (internal productivity, institutional data, and regulated or restricted) each carry a distinct approved-provider list and approval path (see Snippet 2).

Without policy clarity, engineers route work to unsanctioned tools. Harmonic Security (2025) found this at over 90% of organizations. Free tools lack the data agreements, context windows, and authenticated access that enterprise workflows require.

What Does Purpose-Built Tooling Change?

Generic AI access via a chat interface is not the same as purpose-built tooling for a specific workflow.

Benchmark Methodology

We ran two benchmark tasks using Goose (Agentic AI Foundation) with the aptu-coder MCP (Model Context Protocol) server: an auth migration analysis against the official Django codebase, and an AeroDyn integration audit against OpenFAST, a public Fortran repository. The MCP server provides on-demand, structured access to codebases via AST (Abstract Syntax Tree) queries: only the symbols, call graphs, and file ranges each task requires are loaded into context.

Benchmark Results

aptu-coder benchmarks show purpose-built tooling reduced per-task cost 21-68% across two production codebases. Dell’Acqua et al. (2023) found AI assistance lifted output quality 40% on tasks inside the capability frontier and degraded it outside. Tooling determines which side of that line a task lands on. Routing planning to a capable model and execution to a faster one, with structured handoffs between specialized agents, compounds that reduction. Bain (2025) found teams pairing AI with end-to-end process transformation reported 25-30% gains vs. 10% for single-tool augmentation.

Model selection compounds the effect. Aptu benchmarks (Clouatre, 2026) comparing a structured, schema-enforced prompt with Mercury 2 against a raw Claude Opus 4.6 call across six fixtures show: 4.8/5 mean quality vs 2.2/5, at 17x lower cost and 8x lower latency. The structured prompt gives the smaller diffusion model the context it needs.

How Do You Instrument Before You Deploy?

Instrumentation has to precede rollout because the pre-change baseline is the only defensible reference point. Without it, leaders can report usage but cannot prove whether AI changed cost, quality, or delivery time.

Five Numbers Before Day One

Before deploying any AI tooling to a cohort, pre-register five numbers for each target workflow: baseline time or cost per task, target improvement percentage, acceptable risk threshold for defect regression, adoption target as a percentage of the cohort by a fixed date, and the decision date for scale or stop.

Code Snippet 1: PilotRecord dataclass and OTel (OpenTelemetry) span. The five highlighted fields are pre-registration inputs; actual_secs is measured at task completion.

from dataclasses import dataclass
from datetime import date
from opentelemetry import trace
from opentelemetry.trace import StatusCode

@dataclass
class PilotRecord:
    task: str
    baseline_secs: float
    target_reduction_pct: float
    risk_threshold_regression_pct: float
    cohort_adoption_target_pct: float
    decision_date: date           

tracer = trace.get_tracer(__name__)

def record_task_completion(record: PilotRecord, actual_secs: float) -> None:
    with tracer.start_as_current_span("ai_task_completion") as span:
        try:
            span.set_attribute("pilot.task", record.task)
            span.set_attribute("pilot.baseline_secs", record.baseline_secs)
            span.set_attribute("pilot.actual_secs", actual_secs)
            span.set_attribute("pilot.reduction_pct",
                round((1 - actual_secs / record.baseline_secs) * 100, 1))
            span.set_attribute("pilot.decision_date", str(record.decision_date))
            span.set_status(StatusCode.OK)
        except Exception as exc:
            span.record_exception(exc)
            span.set_status(StatusCode.ERROR)
            raiseinstrumentation/pilot_record.py

What Generic Monitoring Misses

The AI observability gaps that block measurement at the agent level apply equally at the program level. Generic monitoring tools capture latency and error rates; they do not capture task completion rates, acceleration ratios, or abandonment.

What Operating Model Sustains Adoption?

Sustained adoption requires a sequenced operating model, not a launch event. Each phase removes the precondition that blocks the next.

**Table 1:** Four-phase operating model with timelines, deliverables, and success signals
Phase	Timeline	Key deliverables	Success signal
1: Baseline and segmentation	Days 1-30	Cohort map, friction audit, policy gap register	Top 5 friction items identified
2: Friction removal	Days 30-60	Data-category policy doc, approved tool list, vendor certifications, role-specific prompt libraries	Blocked cohort begins moving
3: Workflow integration	Days 60-120	Task-specific templates, acceleration ratio tracking, defect quality delta	Measurable throughput in 3+ task categories
4: Sustaining	Days 90-180	Manager KPI inclusion, AI-first sprint planning, monthly adoption review	80% sustained for 60+ days

Phase 1: Baseline and Segmentation

Run a friction audit: structured interviews with a sample of Passive and Blocked engineers, focused on what specifically prevents use of tools already available. The policy gap register captures every approval, certification, or data-handling question with no documented answer. Output is the cohort map, a ranked list of the top five friction items, and a use-case inventory scored on five axes: business value, feasibility, data readiness, risk level, and named sponsorship.

Classify candidate use cases into three lanes by governance exposure. Lane 1 covers internal productivity: code assist, documentation, knowledge search. No regulated data; immediate confidence gains. Lane 2 covers operational workflows: support summarization, knowledge bases, implementation tooling. Institutional data in scope; requires Tier 2 policy coverage. Lane 3 covers product-embedded AI: features delivered to end users. External or regulated data; formal risk review is a precondition. Pilot sequencing follows lane order. Lane 3 is a distinct governance regime, not a later phase of Lane 1. Conflating them is where programs in regulated industries produce incidents (ISG, 2025; PwC, 2025).

Phase 2: Friction Removal

Policy clarity for the blocked cohort is a governance question, not a technology one. The data-category policy document needs to answer three questions per category: which approved provider agreements cover it, whether output may be retained, and what approval is required before a new vendor or tool class is introduced. DX Research (Tacho, 2025) found that daily AI users hit their 10th pull request in 49 days vs. 91 days for non-users, cutting onboarding time by 46%. Reaching that outcome requires role-differentiated enablement (Anthropic, 2025; DORA, 2026):

Executives: governance framing and outcome visibility.
Managers: workflow redesign patterns and inspection criteria for AI-assisted output.
Practitioners: standardized agentic workflows, context engineering patterns, and AI SDLC integration.

McKinsey (2025) found only 1% of companies have reached AI maturity and identified leadership steering, not employee readiness, as the primary gap.

Code Snippet 2: Three-tier data policy template. Tier 1 requires no case-by-case review.

tiers:
  - tier: 1
    label: Internal productivity
    data_types: [code, docs, internal-comms]
    approved_providers: any-approved
    audit_logging: false
    approval_path: none

  - tier: 2
    label: Institutional data
    data_types: [architecture-docs, anonymized-datasets, internal-kb]
    approved_providers: [enterprise-agreement-only]
    audit_logging: true
    approval_path: team-lead-once-per-tool-class

  - tier: 3
    label: Restricted or regulated
    data_types: [pii, regulated-records, confidential]
    approved_providers: [isolated-endpoints-only]
    audit_logging: true
    approval_path: security-review-once-per-vendorpolicy/ai-data-policy.yaml

Without a published tier definition, organizations default to Tier 3 overhead on Tier 1 tasks. In regulated environments, Tier 3 maps to any data class with statutory retention or confidentiality obligations: isolated model endpoints and exportable audit logs are entry criteria before any workflow touches that data. RAG applied to an existing architecture documentation corpus reduced manual compliance documentation effort from weeks to near-automated throughput per migration phase (see RAG for Legacy Systems for the full architecture). That result required Tier 2 classification and an approved provider agreement.

Phase 3: Workflow Integration

Deploy purpose-built tooling for the highest-volume tasks for Active engineers and role-specific templates with explicit examples for the Passive cohort. Track acceleration ratios and defect delta from day one. An AGENTS.md file instructs the agent on commit conventions, identity requirements, and policy boundaries. Orchestrator hooks (in tools like Goose, Claude Code, or Codex) enforce those rules at execution time; local git hooks verify the same contract at commit time; and repository controls (GPG signing, DCO, required code owner review, branch rulesets, provenance attestation) enforce them again at the server. All three layers are intentionally aligned: a well-configured agent should never trip a hook, preserving smooth developer experience without sacrificing accountability.

Phase 4: Sustaining

Include adoption metrics in manager team health reviews and AI-assisted task identification in sprint planning. Without management inclusion, adoption reverts to those who would have adopted regardless. For Tier 3 production AI, uptime guarantees, audit logging, model versioning, and data lineage are entry criteria, not optional features. When an AI-assisted workflow in that tier degrades, the rollback path and escalation owner must be documented before go-live, not defined during the incident. SRE practices for AI agents in production covers the error budget and trust ladder model that operationalizes this.

Which AI Adoption Plays Waste Budget?

Four interventions consume program budget while producing no durable change in the blocked and passive cohorts.

**Table 2:** High-visibility interventions with low conversion impact
Intervention	Why it fails	What to do instead
Hackathons	Attract Active engineers only; do not address Blocked cohort’s actual barrier	Friction audit and policy doc for Blocked cohort
Performance review linkage in year one	Engineers game the metric before templates exist; self-reported numbers inflate	Telemetry-based measurement only until workflow templates are established
Deploying all tools simultaneously	Decision fatigue, shallow engagement; no cohort gets a complete workflow	Sequence by cohort readiness; one complete workflow per cohort before expanding
Skipping policy clarity on data categories	Self-censorship is the correct default in the absence of guidance	Publish the tier model before any rollout begins

How Do You Measure Adoption, Not Just Activity?

Activity metrics, license activations, prompt volumes, and satisfaction scores are easy to collect and tell you nothing about delivery outcomes. Adoption, defined as sustained workflow change, maps directly to business value.

The Five-Number Rule

The pre-registration record below is the program management counterpart to the telemetry pipeline: the committed numbers that turn a pilot into a decision.

Code Snippet 3: Pilot pre-registration record for a code-review workflow. decision_date is the go/no-go gate.

task: ai-assisted-code-review
baseline_minutes: 45
target_reduction_pct: 30
risk_threshold_defect_regression_pct: 0
cohort_size: 20
adoption_target_pct: 75
decision_date: "2026-07-25"
scale_signal: ">=75% cohort active AND defect_regression == 0"
stop_signal: "<50% cohort active OR defect_regression > 0"pilots/code-review-pilot.yaml

Clouatre (2026) found no effect of prompt repetition across three pre-registered experiments. Techniques that improve published benchmarks do not transfer to production agentic systems without internal measurement.

The KPI Scorecard

The adoption metric must come from tool telemetry, not self-report. The METR perception gap (Becker et al., 2025) makes this non-negotiable: survey data is not a proxy for productivity. The burnout index is required: programs that drive throughput without monitoring team health create a different kind of debt.

**Table 3:** KPI scorecard for AI adoption programs. 80% sustained adoption is the threshold at which the practice propagates without continued program intervention.
Dimension	Metric (source)	Target
Delivery velocity	Time per task category (sprint logs)	20-40% reduction by Day 120
Quality (dev)	Defect rate vs. baseline (defect tracker)	No regression
Quality (prod)	Defect escape rate to production (incident tracker)	No increase from pre-AI baseline
AI adoption	% active in past 30 days (tool telemetry)	80% by Day 120
Policy compliance	% use within approved boundaries (audit logs)	100% from Day 1
Team health	Burnout index (anonymous quarterly survey)	No regression

What Should Engineering Leaders Do Next?

Segment before you intervene. The three-cohort model determines which intervention produces return. A single program applied to all cohorts is the primary reason generic programs stall.
Publish a tier model before rollout. A one-page policy that classifies data by tier and names approved provider agreements removes that blocker.
Instrument before you deploy. Build the telemetry pipeline first: OTel spans, task logs, and a pre-registered decision date. The data you collect before rollout is the only baseline you will ever have.
Purpose-built agentic tooling reduces cost. Specialize agents per phase over MCP and AGENTS.md. In regulated environments, vet vendors on data agreements, model provenance, and audit log exportability before any restricted-data pilot.
Measure adoption from telemetry, not self-report. Track acceleration ratio and defect delta, not prompt volume.

For AI governance and decision sequencing in delivery contexts, see Decision Frameworks for AI Delivery.

References

Anthropic, “Effective Context Engineering for AI Agents” (2025) — https://www.anthropic.com/engineering/effective-context-engineering
Bain & Company, “From Pilots to Payoff: Generative AI in Software Development” (2025) — https://www.bain.com/insights/from-pilots-to-payoff-generative-ai-in-software-development-technology-report-2025/
BCG, “AI at Work 2025: Momentum Builds, But Gaps Remain” (2025) — https://www.bcg.com/publications/2025/ai-at-work-momentum-builds-but-gaps-remain
Becker, J. et al., “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity” (2025) — https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study
Cisco, “AI Readiness Index” (2024) — https://www.cisco.com/c/m/en_us/solutions/ai/readiness-index/archive/2024-m11.html
Clouatre, H., “Aptu Benchmarks: aptu+mercury-2 vs raw claude-opus-4.6” (2026) — https://github.com/clouatre-labs/aptu/blob/main/docs/BENCHMARKS.md
Clouatre, H., “aptu-coder benchmark results” (2026) — https://github.com/clouatre-labs/aptu-coder#benchmarks
Clouatre, H., “Ceiling Effects and Convergence: Null Results for Instruction Repetition in LLM-Agent Pipelines” (2026) — https://doi.org/10.5281/zenodo.20039271
Dell’Acqua, F. et al., “Navigating the Jagged Technological Frontier” (2023) — https://doi.org/10.2139/ssrn.4573321
DORA (Google Cloud), “2025 State of DevOps Report” (2025) — https://dora.dev/research/2025/dora-report/
DORA (Google Cloud), “Moving from AI Adoption to Effective SDLC Use” (2026) — https://dora.dev/research/2026/ai-sdlc/
Gartner, “Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027” (2025) — https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
Harmonic Security, “What 22 Million Enterprise AI Prompts Reveal About Shadow AI” (2025) — https://www.harmonic.security/resources/what-22-million-enterprise-ai-prompts-reveal-about-shadow-ai-in-2025
ISG, “State of Enterprise AI Adoption” (2025) — https://isg-one.com/research/state-of-enterprise-ai-adoption
McKinsey, “Superagency in the Workplace: Empowering People to Unlock AI’s Full Potential at Work” (2025) — https://www.mckinsey.com/capabilities/tech-and-ai/our-insights/superagency-in-the-workplace-empowering-people-to-unlock-ais-full-potential-at-work
OECD/BCG, “Identifying and Overcoming Barriers to AI Adoption in Enterprises” (2025) — https://doi.org/10.1787/f9ef33c3-en
PwC, “Responsible AI Survey: From Policy to Practice” (2025) — https://www.pwc.com/us/en/tech-effect/ai-analytics/responsible-ai-survey.html
Tacho, L., “AI cuts onboarding time in half for new hires in the enterprise” (2025) — https://getdx.com/blog/ai-cuts-developer-onboarding-time-in-half

AI Delivery Decision Frameworks: Type 1, Type 2, DACI

Hugues Clouâtre — Tue, 26 May 2026 10:55:00 GMT

A bench mismatch, a scope gap, a stakeholder escalation: each gets treated as a hard blocker requiring structural intervention, when most warrant fast iteration instead. Four frameworks correct this: Type 1/Type 2 sets decision velocity; Eisenhower orders the queue; DACI assigns authority and prevents reversible decisions from stalling; PMBOK stakeholder management separates trust failure from delivery failure.

Contents

Why Are Staffing Mismatches Misclassified as Irreversible?
- Which Decision Types Are Most Often Misclassified?
What Is the Type 1/Type 2 Decision Lens?
- Delivery Examples of Each Type
How Does the Eisenhower Matrix Add Triage Ordering?
- Why Q2 Is the Real Leverage Point
- Immediate vs. Deferrable Actions Across Simultaneous Crises
Why Does Delivery Authority Collapse Without a DACI?
How to Distinguish a Stakeholder Failure from a Delivery Failure?
How Does AI Tooling Shift the Staffing Reversibility Boundary?
- How Does AI Tooling Reclassify Adjacent-Stack Decisions?
- How Do You Qualify the Boundary at Intake?
How Do the Four Frameworks Form a Single Decision Cadence?
- The Four-Layer Stack
What Operating Model Changes Prevent Delivery Failure Recurrence?
- What Three Structural Changes Close the Gaps?
- Follow-On Sequence per Crisis Type
References

Why Are Staffing Mismatches Misclassified as Irreversible?

When a client challenges team competence mid-engagement, the reflex is to swap resources, open a requisition, or restructure. PMI data across 4,455 project management practitioners shows communication, requirements, sponsorship, and alignment failures as leading causes of project underperformance, ahead of personnel gaps.

Triage requires four parties: the account executive (client relationship), the practice lead (bench and capability decisions), the resource manager (internal re-allocation and capacity), and the vendor CSM (Customer Success Manager) if a technology partner is co-funding the engagement. Convening the wrong subset produces a decision without the authority to execute it.

Which Decision Types Are Most Often Misclassified?

A resistant client is more often a symptom of unmet expectations than a verdict on team quality. A bench mismatch signals a lead-time gap, not a capability ceiling. A CEO-level escalation over team conflict is an interpersonal dynamic, not a personnel failure.

Bezos identified this pattern at organizational scale: the primary failure mode in large institutions is applying Type 1 process to reversible situations.

What Is the Type 1/Type 2 Decision Lens?

Type 1 decisions are consequential and largely irreversible (“one-way doors”), warranting great deliberation before committing. Type 2 decisions are reversible and cheap to undo (“two-way doors”), warranting speed and high-judgment individuals or small groups.

Delivery Examples of Each Type

Signing a milestone SOW with vague scope is Type 1: contractually binding and hard to reverse. Proposing a paid inception phase is Type 2: time-limited and reversible. Treating a two-way door as a one-way door stalls decisions.

**Table 1:** Type 1/Type 2 classification for three common delivery decisions.
Situation	Decision	Type	Rationale
Client questions team competence	Hold team; address trust directly	1	Replacing mid-engagement severs continuity and confirms the client’s narrative
Adjacent-stack mismatch, AI tooling available	Deploy with quality gate	2	Architectural reasoning transfers; syntax gap is addressable in the first sprint
CEO escalation over team conflict	Add interim senior lead	2	Addition is reversible; replacement is not

How Does the Eisenhower Matrix Add Triage Ordering?

Eisenhower determines the order of engagement; Type 1/Type 2 determines the decision velocity within each item.

**Table 2:** Eisenhower quadrant examples for delivery leaders.
Quadrant	Delivery example	Action
Q1: Urgent + Important	CEO escalation on a strategic account	Act now; deliberate Type 2 process
Q2: Not Urgent + Important	DACI clarity before SOW is signed; stakeholder relationship investment	Schedule; protect this time each week
Q3: Urgent + Not Important	Non-strategic sales pursuit routing request	Delegate immediately
Q4: Not Urgent + Not Important	Internal reporting that no stakeholder reads	Eliminate

Why Q2 Is the Real Leverage Point

In practice, many Q1 crises begin as neglected Q2 work, a framing consistent with Covey’s urgent-important model and with crisis-management literature emphasizing the cost of deferred preparation (Mitroff & Alpaslan, 2003): DACI not established before pursuit, stakeholder relationships not maintained, team capability not developed. These activities are never urgent, so they are chronically deferred.

Immediate vs. Deferrable Actions Across Simultaneous Crises

When multiple crises land at once, Eisenhower gives the sequence but not the split between stabilization and root-cause work. The immediate column stops the bleeding; the deferrable column addresses recurrence.

**Table 3:** Immediate vs. deferrable actions per crisis type (Eisenhower Q1/Q2 split).
Crisis type	Immediate	Deferrable
Client trust	Call client sponsor; present sprint completion rate and defect density; establish weekly client sync	Stakeholder map refresh; executive sync cadence; quarterly relationship review
Scope/contract	Hold SOW boundary as named Approver; deliver counter-proposal to Sales	Pre-pursuit DACI templates; milestone definition checklist
Team conflict	Convene mediated session; hold structural changes pending root-cause analysis	Psychological safety diagnostic; pairing rotation; retrospective redesign

Why Does Delivery Authority Collapse Without a DACI?

Without a DACI, no single role holds unambiguous decision rights. Decisions stall in silence across dotted-line relationships.

DACI assigns four roles to every significant decision: a Driver who owns moving the decision forward, a single Approver with final authority, Contributors who provide substantive input, and Informed parties notified afterward.

What Failure Mode Does DACI Prevent?

Its power is not taxonomic; it is a forcing function that requires organizations to name, in advance, who can say no. The Approver is the human-in-the-loop control point: the role that cannot be automated, delegated by silence, or assumed. Without that clarity, Sales commits to scope in a client meeting, Delivery identifies structural risks, and the client receives contradictory signals from the same firm. No individual owns the decision, so no individual acts. The tell: Sales and Delivery both believe they communicated clearly; the client received two different answers.

How Do You Exercise Authority Through Dotted-Line Relationships?

DACI works cleanly when authority is direct. When the Delivery Director has only dotted-line authority (informal influence without org-chart ownership), the Approver role depends on trust rather than structure. The mitigation: establish DACI expectations at engagement kickoff, before any crisis, so the dotted-line relationship carries decision-making weight when a scope dispute arrives. Silence is not agreement in dotted-line structures: the Delivery Director must confirm the Approver role in writing at engagement start.

Code Snippet 1: AGENTS.md delivery governance section. The Role block scopes agent authority; the Constraints block makes rules machine-checkable.

## Role
You assist the Delivery Director on scope, staffing, and client communication tasks.
You do not have authority to commit, approve, or communicate on behalf of Sales or the client.

## Constraints
- Scope changes to a signed SOW: flag for Approver sign-off before any client communication.
- Sales verbal commitments are not binding; require a written amendment before acting on them.
- Staffing proposals outside the AI capability frontier: classify Type 1 and flag for human review.
- Ambiguous decisions: output what information is missing; do not infer or assume.AGENTS.md

DACI in Practice: SOW Scope Disputes

When a client compresses a fixed-scope contract to a shorter timeline with vague milestones, the Delivery Director must be the named Approver on SOW structure. Without that delineation, Sales capitulates because no one has organizational standing to hold the boundary. Haleem et al. (2021) trace this failure to requirements uncertainty — missing, misinterpreted, or shifting stakeholder needs. The correct response is sharpening milestones before commitment, not compressing the schedule around them.

The distinction between T&M (Time and Materials) and milestone-based contracts is not cosmetic: T&M transfers schedule risk to the client; milestones transfer it to Delivery. Compressing T&M scope into a milestone structure without sharpening acceptance criteria turns a signed contract into a liability. The Delivery Director’s counter-proposal to Sales is not a refusal — it is a paid inception phase that sharpens milestones before risk transfers.

Code Snippet 2: DACI governance template for SOW milestone compression. Gatekeepers hold narrow, policy-bounded veto rights; silence within 48 hours defaults to sign-off.

decision: sow_milestone_compression
description: "Client requests timeline reduction on fixed-scope contract"

roles:
  driver: account_executive
  approver: delivery_director
  gatekeepers:
    delivery_operations:
      scope: bench capability and stack fit
      veto_right: true              # Blocks if skills gap cannot be closed 
    legal_compliance:
      scope: liability shift from T&M to milestone
      veto_right: true              # Blocks if indemnification limits breached 
  contributors:
    - lead_architect
    - engagement_manager
  informed:
    - practice_lead
    - engineering_bench

constraints:
  veto_sla_hours: 48                # Silence defaults to sign-off 
  escalation: delivery_director     # Unresolved veto escalates here
  decision_deadline_days: 3delivery/daci-sow-template.yaml

How to Distinguish a Stakeholder Failure from a Delivery Failure?

When a client says “your team is bad,” that is a claim, not a diagnosis. Genuine delivery underperformance and a stakeholder engagement breakdown produce nearly identical surface symptoms. Treating them as the same problem leads to the wrong intervention.

How Do You Separate Perception from Delivery Record?

PMBOK stakeholder management provides the diagnostic frame: a high-power, low-trust client who is resistant to decisions and consistently critical of the team is a textbook stakeholder engagement failure (per PMBOK’s power/interest grid) until the objective delivery record proves otherwise. Separate the client’s perception from the sprint completion rate, defect density, and milestone adherence. If the team is delivering against agreed commitments, the problem is trust, not performance.

How Does Psychological Safety Aid Team Recovery?

When a lead engineer requests reassignment and multiple engineers are in open conflict, this is a psychological safety breakdown. Replacing the team validates the client’s framing and signals the pattern will recur. The correct response is the Delivery Director engaging directly to stabilize the relationship and create conditions for recovery.

When a Lead Engineer Requests Reassignment

A reassignment request from a lead engineer is a Type 1/Type 2 decision with team-wide visibility. Granting it immediately signals that opting out under pressure is acceptable; denying it without addressing the cause risks losing the engineer and worsening morale. The decision hinges on whether the engineer’s distress is rooted in the client relationship, recoverable with Delivery Director intervention, or in the team dynamic, requiring mediation first. Hold the reassignment open while the Delivery Director stabilizes the client relationship directly. Psychological safety reduces turnover intent: a meta-analysis of 136 studies found a negative correlation between psychological safety and turnover intent (Frazier et al., 2017).

When multiple engineers are at odds, the cause is rarely uniform. The diagnostic split: interpersonal conflict resolves with mediation and pairing rotation; technical disagreement resolves with an architectural decision record and a designated technical authority; structural conflict (competing priorities, unclear ownership) resolves with DACI. Applying the wrong intervention to the wrong cause extends the crisis.

The first direct Delivery Director-to-client conversation should follow a structured agenda:

Acknowledge the frustration without conceding the diagnosis.
Present the objective delivery record: sprint completions, defect density, milestone adherence.
Establish a direct cadence: weekly sync with the Delivery Director attending.
Name one concrete commitment with a short timeline, visible before the next sync.

How Does AI Tooling Shift the Staffing Reversibility Boundary?

A client requires a stack the available bench has not worked in directly. Traditionally, this is treated as Type 1: the wrong profile risks delivery failure and reputational exposure that cannot be undone mid-engagement.

How Does AI Tooling Reclassify Adjacent-Stack Decisions?

AI-augmented delivery changes this calculus. Engineers on an adjacent stack carry the mental models that matter: async concurrency patterns, component-based UI (User Interface) architecture, REST (Representational State Transfer) API design, client-side state management. The gap is syntax and ecosystem conventions, not architectural reasoning, and that is what agentic engineering addresses: code generation, documentation injection, and automated quality gates. Peng et al. (2023) found that developers completed an HTTP server implementation task 55.8% faster with GitHub Copilot.

When live framework documentation is injected into agent context, unfamiliarity with a specific library significantly reduces as a delivery constraint, narrowing the bottleneck to quality review — a role a senior engineer performs in any language. Zhou et al. (2023) found that injecting API documentation at inference time improved code generation on unseen APIs by 52% relative (CodeT5 pass@1); Chen et al. (2025) measured 83-220% pass-rate improvements on unfamiliar Python libraries with documentation injection. Amazon migrated tens of thousands of production Java applications from Java 8 or 11 to Java 17, saving an estimated 4,500 developer-years of manual effort (AWS, 2024).

Figure 1: Adjacent-stack bench mismatches route to Type 1 (specialist required) or Type 2 (augment with AI tooling) based on gap type and a quality gate.

How Do You Qualify the Boundary at Intake?

For adjacent stacks, bench mismatch is now a Type 2 decision when agentic tooling is available. The qualification step belongs at engagement intake: assess which engineers have the architectural mental models that transfer and confirm tooling access is in place before committing the profile to the client. One constraint applies uniformly: for tasks outside the confirmed AI capability frontier, Dell’Acqua et al. (2023) found the BCG (Boston Consulting Group) cohort using AI performed 19% worse than those without it. Map the frontier before staffing, not after.

Code Snippet 3: SKILL.md for delivery decision triage. The description is a neutral trigger condition; the Rules block prevents inference beyond provided intake data.

---
name: delivery-decision-triage
description: "Classifies staffing and scope decisions as Type 1 or Type 2. Use when evaluating an adjacent-stack deployment, a scope change, or a delivery risk at intake."
---

## When to use
- Evaluating whether to deploy an engineer on an adjacent stack
- Assessing whether to approve a scope or timeline change
- Triaging a delivery risk under time pressure

## Rules
- Use only intake data provided; do not infer missing context.
- Architectural gap (domain knowledge missing): Type 1, escalate to Delivery Director.
- Syntactic gap with active AI tooling: Type 2, require a written quality gate before committing.
- Tasks outside the confirmed AI capability frontier: Type 1 regardless of tooling.
- Classification ambiguous after intake: output "insufficient data" and list what is missing.skills/delivery-decision-triage.md

How Do the Four Frameworks Form a Single Decision Cadence?

The four frameworks operate at different levels and answer distinct questions. Treating them as alternatives produces analysis paralysis; used as complementary layers, they produce a coherent cadence.

The Four-Layer Stack

**Table 4:** Four-framework decision cadence — what each layer answers and when it fires.
Framework	Question answered	When it fires
Eisenhower	What order do I act in?	On arrival of any new issue; set priority before acting
Type 1 / Type 2	How carefully should I decide?	Before committing to any action; governs deliberation depth
DACI	Who decides, who executes?	When authority is ambiguous or Sales and Delivery diverge
PMBOK Stakeholder	Whose buy-in do I need?	When a high-power party is not yet engaged or trust is low

What Operating Model Changes Prevent Delivery Failure Recurrence?

An AI operating model requires decision frameworks embedded at the pursuit stage, before any crisis arrives. Governance structures must be in place before the pressure that tests them.

All four failure modes share one structural root cause: decisions that should have been made upstream were deferred until pressure forced them.

Figure 2: Eisenhower sequences the queue, Type 1/Type 2 sets decision velocity, DACI assigns the Approver.

What Three Structural Changes Close the Gaps?

Three changes close those gaps:

Establish DACI before pursuit. Document the Approver role for SOW structure at pursuit, agree an escalation ladder between Sales and Delivery, and communicate it to the client before contract execution.
Protect Q2 capacity. Stakeholder investment, milestone sharpening, and team psychological safety are Q2 activities that Q1 crises crowd out; block Q2 time weekly.
Qualify the AI capability frontier at intake. Map which tasks fall within AI range, classify which staffing decisions are Type 2, and flag which require a specialist match tooling cannot substitute.

Follow-On Sequence per Crisis Type

The operating model changes are structural; the follow-on sequence is tactical. The phases below reflect practitioner convention, not empirical prescription: timelines compress or extend based on client risk tolerance and relationship history.

**Table 5:** Follow-on sequence per crisis type (practitioner heuristic).
Crisis type	Stabilize	Demonstrate	Decide
Client trust crisis	Delivery Director sync cadence established	Objective delivery scorecard shared with client	Client confirms team continues or escalation path named
Scope/contract dispute	Counter-proposal delivered to Sales	Sharpened milestones documented in writing	Signed SOW or explicit walk-away decision
Team conflict escalation	Mediated retrospective completed	Pairing rotation and lead coaching underway	Lead reassignment request resolved or closed

These frameworks do not eliminate hard conversations. They determine who has standing to have them, in what order, and with what authority. Most delivery failures trace to one deferred upstream decision: a DACI never written, a stakeholder relationship never invested in, a scope boundary never held (PMI, 2018). The delivery leader’s leverage is in Q2 work, not crisis response.

For a deeper look at how agentic tooling changes the architecture of delivery work itself, see Orchestrating AI Agents: A Practical Guide to Subagent Architecture.

References

AWS, “Amazon Q Developer Just Reached a $260 Million Dollar Milestone” (2024) — https://aws.amazon.com/blogs/devops/amazon-q-developer-just-reached-a-260-million-dollar-milestone/
Bezos, J., “2015 Letter to Shareholders” (2016) — https://s2.q4cdn.com/299287126/files/doc_financials/annual/2015-Letter-to-Shareholders.PDF
Chen, Y. et al., “When LLMs Meet API Documentation” (2025) — https://arxiv.org/abs/2503.15231
Covey, S.R., “The 7 Habits of Highly Effective People” (1989) — https://www.simonandschuster.com/books/The-7-Habits-of-Highly-Effective-People/Stephen-R-Covey/9781982137274
Dell’Acqua, F. et al., “Navigating the Jagged Technological Frontier” (2023) — https://doi.org/10.1287/orsc.2025.21838
Edmondson, A.C., “Psychological Safety and Learning Behavior in Work Teams” (1999) — https://doi.org/10.2307/2666999
Frazier, M.L. et al., “Psychological Safety: A Meta-Analytic Review and Extension” (2017) — https://doi.org/10.1111/peps.12183
Haleem, M. et al., “Cognitive Approach to Handle Requirements Uncertainty in Software Projects” (2021) — https://www.sciencedirect.com/science/article/pii/S2666307421000218
Mitroff, I.I. & Alpaslan, M.C., “Preparing for Evil” (2003) — https://www.researchgate.net/publication/10811426_Preparing_for_Evil
Peng, S. et al., “The Impact of AI on Developer Productivity: Evidence from GitHub Copilot” (2023) — https://doi.org/10.48550/arXiv.2302.06590
PMI, “Pulse of the Profession” (2018) — https://www.pmi.org/-/media/pmi/documents/public/pdf/learning/thought-leadership/pulse/pulse-of-the-profession-2018.pdf
Zhou, S. et al., “DocPrompting: Generating Code by Retrieving the Docs” (2023) — https://arxiv.org/abs/2207.05987

SRE for AI Agents: Error Budgets, Trust, and 90 Trials

Hugues Clouâtre — Thu, 11 Jun 2026 11:07:00 GMT

AI tooling budgets hit record highs. We ran 90 file-prediction trials to measure what an AI agent gets wrong before it touches production. The model predicted 1.7 files beyond the actual change set on average, even on a well-structured codebase. SRE is not ceremony. It is the empirical gate between velocity and blast radius.

Contents

Why Is AI Widening the Dev/Ops Gap?
- The Perception Gap: Dashboards vs. Practitioner Reality
How Did We Measure AI’s Scope Creep?
- Why Medium-Tier PRs Underperformed
What Does SRE Mean in a Regulated Enterprise?
- Error Budgets as Compliance Evidence
How Does SRE Act as AI’s Production Conscience?
- Decision Provenance and Error Budget Separation
- Blast Radius Containment and the Trust Ladder
Why Does Platform Maturity Gate AI Readiness?
- The Learning Time Deficit
Where Should You Start?
- The Four Actions in Order
References

Why Is AI Widening the Dev/Ops Gap?

Toil is the repetitive, manual operational work that scales with system load rather than adding lasting value. Catchpoint’s annual SRE reports tracked toil rising from 25% to 34% between 2024 and 2026, the first sustained increase in five years. Over the same period, 92% of developers report that AI tools increase the blast radius of bad code needing to be debugged (DevOps.com, 2025). Defensive pipeline architectures can close part of this gap, but they address the pipeline, not the production governance layer.

Three root causes keep surfacing in post-mortems:

AI babysitting: someone has to review generated runbooks and roll back missed-context remediations
Configuration drift: accelerating faster than humans can audit it
Validation overhead: compounding because every AI output needs a trust-but-verify pass before production

More frequent deploys, multiplied by more autonomous agents, against review capacity that has not scaled to match. METR found experienced developers took 19% longer with AI tools despite perceiving a 20% speedup (Becker et al., 2025).

The Perception Gap: Dashboards vs. Practitioner Reality

The perception gap makes this harder to fix. Directors reviewing dashboards see ticket counts drop and declare victory. Practitioners on the ground feel increased friction because toil shifted from “boring but predictable” to “novel and unpredictable.”

Figure 1: AI investment and measured toil both climbing, 2021-2026.

How Did We Measure AI’s Scope Creep?

We ran 90 file-prediction trials against tobymao/sqlglot, an MIT-licensed SQL transpiler with 9k+ stars: 30 merged PRs stratified across simple, medium, and complex tiers, 3 predictions each, using a single Claude Sonnet 4.6 Bedrock call with no agent loop or retrieval. Given a GitHub issue description and the repository file tree, the model predicted which files a human engineer modified. Scope hallucination counts files predicted beyond the human’s actual change set. For full data, see Supplementary Materials.

**Table 2:** Scope prediction results by complexity tier across 90 trials (30 PRs x 3 runs). F1 is harmonic mean of precision and recall.
Tier	Precision / Recall	F1	Scope Creep
Simple (1-2 files)	0.645 / 0.850	0.708	1.3 files
Medium (3-5 files)	0.540 / 0.585	0.552	2.2 files
Complex (6-15 files)	0.769 / 0.673	0.712	1.6 files

Why Medium-Tier PRs Underperformed

The non-monotonic curve is the headline finding: Jaccard similarity scored 0.60 for simple PRs, 0.41 for medium, and 0.58 for complex. Medium PRs are the hardest tier: too many candidate files to guess by elimination, yet not enough structural regularity to infer the change set from sqlglot’s dialect-file conventions. Complex PRs scored highest because sqlglot’s rigid directory structure makes multi-file change sets predictable from the dialect name alone. 12 of 30 PRs failed with a Jaccard score below 0.5, spread across all tiers; no tier is immune. In a shadow-mode deployment, every over-predicted file is a false positive the reviewer must filter before the change reaches production.

What Does SRE Mean in a Regulated Enterprise?

SRE is not DevOps with a different name. The distinction is structural. In regulated financial services, production reliability carries regulatory weight: OSFI’s B-13 guideline mandates technology risk management with board-level accountability, and the EU’s DORA regulation sets equivalent requirements for operational resilience across European financial services (European Parliament and Council, 2022).

Error Budgets as Compliance Evidence

SRE answers this with a reliability contract. Error budgets define how much unreliability a service can tolerate before feature work stops. SLOs (service level objectives) make reliability measurable rather than aspirational. Blameless postmortems treat incidents as system failures, not personnel failures. Google codified this framework in 2016 and enterprises have since adapted it, but in regulated environments the stakes include regulatory censure, not just customer churn.

Platform Engineering provides capability: the tools, the internal developer platform, the golden paths. SRE provides accountability: the error budgets, the incident response, the production governance. The question is whether that accountability holds when the agent making changes is not human.

How Does SRE Act as AI’s Production Conscience?

Deploying an AI agent is a reliability problem, not a monitoring problem. Monitoring tells you something broke; a reliability framework tells you how much breakage you can tolerate, who caused it, and whether to keep going.

Decision Provenance and Error Budget Separation

Decision provenance, the AI observability requirement that every agent action links to its inputs, reasoning, and authorization chain, goes beyond logging what an agent did. You need to trace why it made a choice, what context it consumed, and which prior decisions influenced the outcome. Without this, debugging an autonomous system is archaeology, not engineering. Under OSFI B-13 and DORA, an agent action without decision provenance is not just a debugging gap; it is a compliance liability.

Separate error budgets for AI-generated changes keep machine-authored deployments from hiding behind human baselines. If an AI agent burns through its error budget, its write permissions get revoked automatically, not the entire team’s. Our results showed 0.769 precision even on the best-performing tier, meaning roughly 1 in 4 predicted files was wrong. That error rate needs its own budget.

Code Snippet 1: Prometheus recording rules and burn-rate alert for AI-authored deployments. Separate SLIs per author type; 14.4x burn-rate threshold (Google, 2018).

groups:
  - name: sli.deploys
    rules:
      - record: sli:deploy_success:ratio1h
        expr: |
          sum(rate(deploy_success_total{author_type="ai"}[1h]))
          / sum(rate(deploys_total{author_type="ai"}[1h]))
        labels:
          author_type: ai
          slo_target: "0.995" # Stricter than human baseline of 0.990
      - record: sli:deploy_success:ratio1h
        expr: |
          sum(rate(deploy_success_total{author_type="human"}[1h]))
          / sum(rate(deploys_total{author_type="human"}[1h]))
        labels:
          author_type: human
          slo_target: "0.990"
      - alert: AIChangeErrorBudgetBurnRate
        expr: |
          (1 - sli:deploy_success:ratio1h{author_type="ai"})
          / (1 - 0.995) > 14.4
        for: 5m
        labels:
          severity: critical
          team: sresli-and-burn-rate.yaml

Production teams consistently trade agent capability for reliability, preferring narrower but predictable automation over broad but brittle autonomy (Pan et al., 2026). Separate error budgets formalize that trade-off.

Blast Radius Containment and the Trust Ladder

Blast radius containment means progressive rollout gates. No agent ships to 100% on day one. The trust ladder is a graduated set of AI guardrails where each rung grants broader blast radius only after the agent demonstrates reliability at the current level:

Shadow mode (read-only, no writes)
Supervised writes (human approval required)
Unsupervised writes (scoped to low-risk paths)
Full autonomy (reserved for proven, bounded tasks)

Our experiment is a proxy for what shadow mode catches. At the medium tier, where Jaccard dropped to 0.409, shadow mode would have flagged more than half the predicted change set as incorrect. The specific SLO threshold is yours to define; what matters is that it is explicit, measured, and tied to your error budget rather than a gut feeling.

Code Snippet 2: Kubernetes RBAC ClusterRoles for each trust ladder tier. Promotion from readonly to scoped-write to production-write is a ServiceAccount rebinding; demotion reverses it.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ai-agent-readonly
rules:
  - apiGroups: [""]
    resources: [pods, services, configmaps]
    verbs: [get, list, watch]
  - apiGroups: [apps]
    resources: [deployments, replicasets]
    verbs: [get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ai-agent-scoped-write
rules:
  - apiGroups: [""]
    resources: [pods, services, configmaps]
    verbs: [get, list, watch]
  - apiGroups: [apps]
    resources: [deployments]
    verbs: [get, list, watch, update, patch]
    resourceNames: [canary-payments]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ai-agent-production-write
rules:
  - apiGroups: [""]
    resources: [pods, services, configmaps]
    verbs: [get, list, watch, create, update, patch, delete]
  - apiGroups: [apps]
    resources: [deployments, replicasets]
    verbs: [get, list, watch, create, update, patch, delete]sre/trust-ladder-rbac.yaml

Accuracy alone cannot distinguish an agent that fails on a fixed subset of tasks from one that fails unpredictably at the same rate (Rabanser et al., 2026). Our 90 runs confirmed consistency (27 of 30 PRs showed zero variance across runs, 3 showed near-zero) but exposed robustness and safety gaps on complex refactoring tasks. Consistent failures are exactly what shadow mode is designed to catch: the model’s errors are systematic, not random, and a human reviewer can filter them. Early evidence supports this approach: STRATUS, a multi-agent SRE system operating under similar progressive constraints, achieved a 1.5x improvement over baselines in automated failure mitigation (Chen et al., 2025).

Why Does Platform Maturity Gate AI Readiness?

The 2025 DORA report is explicit: AI’s impact depends on the quality of the underlying organizational system. Bolt AI onto a fragile platform and you get faster fragility. An AI agent that auto-scales a misconfigured service does not fix the misconfiguration; it scales the blast radius. The AIOpsLab framework shows agent performance varies significantly with the quality of instrumented infrastructure underneath (Chen et al., 2025).

The maturity sequence matters. Build the IDP (internal developer platform) first, layer SRE practices including LLMOps telemetry for token consumption, latency, and decision traces on top, then introduce agentic AI. Skip a step and the agents inherit your tech debt at machine speed.

The Learning Time Deficit

Only 6% of SREs have dedicated, protected learning time (Catchpoint, 2026). You cannot build an SRE practice when the people staffing it have no time to learn the discipline. Concretely, 10% protected time means one half-day per week where an SRE studies agent failure modes, reviews postmortems from other teams, or shadow-tests a new observability tool without on-call interruptions. The organizations with the lowest toil trends treat learning hours like error budgets: protected, measured, and non-negotiable.

**Table 3:** Platform + SRE maturity levels and what each unlocks.
Maturity Level	Platform State	SRE State	AI Readiness
Foundation	Manual provisioning	Reactive ops, no SLOs	Not ready
Standardized	Self-service IDP	SLOs defined, error budgets	Read-only agents
Measured	Golden paths adopted	Toil tracked, burn-rate alerts	Shadow mode agents
Optimized	Platform-as-product	Blameless culture, SLO-driven	Supervised write agents
Autonomous	Full self-service	Proactive reliability	Agentic AI with guardrails

Read-only agents need SLOs because without a defined “good,” the agent cannot distinguish signal from noise. Supervised write agents need blameless culture because humans must feel safe overriding the machine.

Where Should You Start?

Enforce the prerequisites before enabling agentic AI on any service:

Code Snippet 3: OPA policy gate enforcing the minimum bar before any service receives agentic AI write access. Shadow period and decision provenance are the two gates most commonly skipped in practice.

package sre.ai.readiness

import rego.v1

default allow_agentic_ai := false

allow_agentic_ai if {
    input.slo_defined
    input.error_budget_policy
    input.shadow_period_days >= 30
    input.decision_provenance      
    input.rollback_automated
    input.toil_measured
}sre-readiness-check.rego

The Four Actions in Order

Four actions, in order:

Audit your toil budget. Measure actual toil against perceived toil. If practitioners report higher friction while dashboards show fewer tickets, you have shifted toil rather than eliminated it.
Define SRE boundaries. One team owns the IDP. Another owns the error budgets. Overlap is where accountability dies.
Split error budgets by author type. Human-authored and AI-authored deployments have different failure profiles. Track them independently and revoke AI write access when the budget burns too fast.
Protect learning time. Budget 10% of engineering hours for skill development or accept compounding operational risk.

Start with the toil audit, the only prerequisite you can measure without instrumentation already in place. Measure first. Then automate. Each rung you skip is a gap in your audit trail and a risk your board will eventually ask about.

References

Becker et al., “Evidence on the Impact of Generative AI on Software Development” (2025) — https://arxiv.org/abs/2507.09089
Catchpoint, “SRE Report 2026” (2026) — https://www.catchpoint.com/learn/sre-report-2026
Chen et al., “AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds” (2025) — https://arxiv.org/abs/2501.06706
Chen et al., “STRATUS: A Multi-agent System for Autonomous Reliability Engineering of Modern Clouds” (2025) — https://arxiv.org/abs/2506.02009
Clouatre, H., “SRE Shadow-Mode File Prediction” (2026) — https://doi.org/10.5281/zenodo.20645992 — https://github.com/clouatre-labs/sre-shadow-replay
DevOps.com, “Survey: AI Tools are Increasing Amount of Bad Code Needing to be Fixed” (2025) — https://devops.com/survey-ai-tools-are-increasing-amount-of-bad-code-needing-to-be-fixed-2/
DORA (Google Cloud’s DevOps Research and Assessment), “2025 State of AI-assisted Software Development” (2025) — https://dora.dev/research/2025/dora-report/
European Parliament and Council, “Digital Operational Resilience Act (DORA)” (2022) — https://www.digital-operational-resilience-act.com/
Google, “Site Reliability Engineering” (2016) — https://sre.google/sre-book/table-of-contents/
Jimenez, C. E. et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” (ICLR 2024) — https://arxiv.org/abs/2310.06770
OSFI, “Technology and Cyber Risk Management Guideline B-13” (2022) — https://www.osfi-bsif.gc.ca/en/guidance/guidance-library/technology-cyber-risk-management
Pan et al., “Measuring Agents in Production” (2026) — https://arxiv.org/abs/2512.04123
Rabanser et al., “Towards a Science of AI Agent Reliability” (2026) — https://arxiv.org/abs/2602.16666

What a Null Result Taught Us About AI Agent Evaluation

Hugues Clouâtre — Wed, 17 Jun 2026 11:04:00 GMT

A Google Research paper demonstrates that repeating the entire user prompt verbatim can lift accuracy by up to 76 percentage points at zero output cost. No chain-of-thought overhead. No reasoning budget. Just send the same instruction twice.

We ran 20 parallel agents across two experiments: 10 per experiment, 5 control vs. 5 treatment, blind-scored against a pre-registered rubric.

We found nothing. The nothing is the finding.

Contents

What Did the Paper Claim?
- Why Our Agent Seemed Like a Good Candidate
- Why This Matters for Engineering Teams
How Did We Design the Test?
What Happened in the FastMCP Refactor Test?
Did a Stricter Methodology Change the Result?
What Infrastructure Confound Did We Miss?
Why Did Both Experiments Hit 100%?
- Where the Boundary Falls
What Did We Learn About AI Evaluation Design?
When Should You Use Prompt Repetition?
- What Transfers to Your Team
Did Prompt Repetition Change Anything Else?
References

What Did the Paper Claim?

A 2025 paper by Leviathan et al. at Google Research proposes a simple technique: repeat the entire user prompt once, verbatim, before sending to the model.

The mechanism is structural, not empirical. Decoder-only transformers use causal masking: each token attends only to tokens before it. In a single-pass prompt, early tokens never see later context. Repeating the prompt creates a second copy where every token attends to the full instruction during prefill. This reduces the positional attention decay documented as the “lost in the middle” phenomenon (Liu et al., 2023). This is a fundamental limitation of the decoder-only architecture, not a quirk of specific benchmarks. Among decoder-only models, a 675B-parameter Mixture-of-Experts frontier model and a 3B-active-parameter small language model (SLM) (NVIDIA, 2025) share it equally. Bidirectional architectures, including diffusion language models such as Inception Mercury 2, attend to the full sequence in a single pass and do not exhibit this limitation.

Code Snippet 1: Causal masking creates an asymmetry where early tokens cannot attend to later context. Repeating the prompt gives the second copy full visibility over the first.

Standard prompt (single pass):
  Token 1  sees: [Token 1]
  Token 5  sees: [Token 1, 2, 3, 4, 5]
  Token 50 sees: [Token 1, 2, ..., 50]
  --> Early tokens are blind to later context

Repeated prompt (two copies):
  Token 51 sees: [Token 1, 2, ..., 50, 51]
  Token 55 sees: [Token 1, 2, ..., 50, 51, 52, 53, 54, 55]
  --> Every token in the second copy attends to the full first copy
  --> Full context available during prefill, zero generation costcausal-masking.txt

The reported gains:

Gemini 2.0 Flash-Lite on NameIndex: 21.33% to 97.33% accuracy
GSM8K and MMLU-Pro gains across Gemini 2.0 Flash, GPT-4o, Claude 3.7 Sonnet, DeepSeek V3, and others
Input tokens double; output tokens unchanged in fixed-format benchmarks (no latency increase, unlike chain-of-thought)

The paper positions this as a Pareto improvement over reasoning-heavy approaches: same output budget, better accuracy.

Why Our Agent Seemed Like a Good Candidate

Our Scout delegate, the research agent in our subagent architecture (full recipe), runs on claude-haiku-4-5 at temperature 0.5 with extended thinking off. Haiku 4.5 is structurally a non-reasoning model (extended thinking is opt-in, not default), making it precisely the class of LLM the paper’s title targets.

The paper tested Claude 3 Haiku alongside six other models; its strongest gains came from Gemini 2.0 Flash-Lite and GPT-4o-mini. We tested Claude 4.5 Haiku, a different model generation. Anthropic does not publish architectural details for either model. Whether the technique transfers across generations is an open question this experiment cannot answer, because our ceiling effects prevented any treatment from showing lift.

Why This Matters for Engineering Teams

Teams adopt AI techniques from papers without field-testing them first. BCG reports that 50% of companies are stagnating with AI (BCG, 2025), partly because they ship optimizations without measuring baselines. Shipping an unvalidated prompt change to production would cost more: doubled input tokens on every request, with no accuracy gain to show for it. As we covered in observability for AI agents, optimizing without measuring before and after is flying blind.

How Did We Design the Test?

Both experiments shared the same core structure: 10 parallel async Scout delegates, split 5 control vs. 5 treatment, scored blind against a pre-registered rubric. For detailed methodology and raw data, see Supplementary Materials.

Code Snippet 2: Shared delegate configuration. All 10 runs use the same model, temperature, and extensions.

# Shared config across all 10 delegates
model: claude-haiku-4-5
temperature: 0.5
extensions:
  - developer
  - context7
  - brave_search
output: "scout-run-{{ id }}.json"delegate-config.yaml

Control group: standard Scout instructions (~3,805 characters, instructions x1).

Treatment group: instructions repeated verbatim (~7,633 characters, instructions x2), mimicking the paper’s <QUERY><QUERY> pattern applied to the agent’s system prompt.

The orchestrator spawned all 10 delegates simultaneously via Goose’s background task system and handed off structured JSON. A separate blind-scoring delegate received only the output files (no group labels) and scored each against the rubric. Group assignments were sealed in a label map before scoring began.

Figure 1: Blind evaluation pipeline. Group labels are stripped before scoring to prevent bias.

What Happened in the FastMCP Refactor Test?

Target: FastMCP session ID refactor in math-mcp-learning-server. Open and unimplemented at the time of the experiment.

Rubric: 6 binary criteria, pre-registered before any runs were examined.

9 of 10 delegates produced valid output. control-1 ran 93 messages and wrote no output file. Session log analysis confirmed the file-write instruction appeared only at the end of the delegate prompt, and the model drifted past it. This is a delegate authoring flaw with a known fix: bookend critical instructions at the start and end.

Across the 9 valid runs, 5 of 6 criteria scored 100% in both groups. The only variance was C5 (must-not constraint violations): control 5.50/6, treatment 5.80/6, delta +0.30. The treatment scored marginally higher, but at n=4 vs n=5 with binary outcomes, Fisher’s exact test is degenerate (p = 1.0). Full per-criterion scores are in the raw data.

The task was too easy. The rubric could not discriminate. We needed a harder target.

Did a Stricter Methodology Change the Result?

Target: aptu#737, a tree-sitter AST (Abstract Syntax Tree)-based security scanner evaluation. Harder task, requiring synthesis from source code rather than retrieval from issue text. Unimplemented when tested.

Rubric: 7 binary criteria. C5, C6, and C7 required the delegate to read and reason about actual source code, not just summarize the issue. Pre-registered before any runs began.

Methodology improvements over Experiment 1:

Blinded file naming from the start (scout-run-01.json through scout-run-10.json with sealed label-map.json)
Mann-Whitney U test pre-specified (two-tailed, alpha = 0.05)
Wall-clock latency recorded per delegate

Results:

**Table 1:** Experiment 2 results. Zero variance in either group. Mann-Whitney U = 12.5, p = 1.0 (degenerate: complete ties, test cannot be evaluated).
Group	Score	Wall-clock median
Control (x1)	7/7 all runs	6m 21s
Treatment (x2)	7/7 all runs	7m 29s
Delta	0	+17.8%

Every Scout, in every run, in both groups, scored 7/7. Even C5, C6, and C7, the synthesis criteria we specifically designed to require source code reasoning, hit 100% across the board.

The 17.8% latency difference is in the expected direction (longer prompt, longer prefill), which is consistent with the paper’s Anthropic-specific latency caveat. At scale, that delta compounds: doubled tokens cost money, and the added prefill time costs throughput across every agent invocation. But n=5 cannot support any inference here, and the finding is further confounded by an infrastructure issue we discovered afterward.

The scores told us nothing. The session logs told us something we did not expect.

What Infrastructure Confound Did We Miss?

Post-hoc session log analysis revealed a confound present in both experiments.

Goose enforces a hard cap of 5 concurrent background delegates. When all 10 delegates were spawned simultaneously, runs 06-10 hit the cap and were queued into a second batch after runs 01-05 completed.

The resulting batch structure was unbalanced:

**Table 2:** The 5-delegate concurrency cap split 10 simultaneous spawns into two unbalanced batches. Exact run assignments are in the raw data.
Batch	Runs	Control	Treatment	Condition
1 (runs 01-05)	5	C1, C2, C3	T1, T2	Resource contested
2 (runs 06-10)	5	C4, C5	T3, T4, T5	No contention

Treatment delegates landed disproportionately in the less-contested second batch, making any latency comparison between groups uninterpretable. Accuracy was unaffected (ceiling effects dominated regardless), but the exposure is worth naming: pre-registration does not protect against runtime infrastructure behavior you did not know existed.

The confound matters for latency. But the bigger question is why accuracy showed zero variance in the first place.

Why Did Both Experiments Hit 100%?

Two experiments, two rubrics designed to be harder than the last, two 100% results.

This is itself a finding. A well-designed Scout delegate on a well-scoped engineering issue is already operating above the baseline accuracy threshold where prompt repetition shows lift. The paper’s largest gains came from synthetic positional tasks, NameIndex (Leviathan et al., 2025), where the answer is a name buried in a list. Real engineering issues, even unimplemented ones, give the agent structured context, code references, and acceptance criteria. The agent finds what it needs without help from the prefill geometry.

This is the core finding: prompt repetition solves an attention problem that well-scoped engineering tasks do not have. The technique’s value is real, but the paper’s benchmarks do not cover agentic engineering tasks. Our experiments tested that boundary. When the agent already has structured context pointing it to the right code, repeating the instruction adds input tokens without adding accuracy signal. Understanding where SLMs succeed and fail on their own is not academic: hybrid architectures like SMART (Kim et al., 2025) use SLMs as the primary reasoning engine, with LLMs intervening only at critical junctures. Every prompt-level optimization that improves standalone SLM accuracy reduces how often the expensive backstop fires.

For teams evaluating prompt techniques at scale, the implication is financial: doubling input tokens across every agent invocation is a measurable cost increase. If your agents already converge correctly on well-scoped tasks, that spend returns nothing. Embracing negative results as a research practice (Berger et al., 2024) prevents exactly this kind of waste: publication bias toward positive results means the null findings that would have saved you the experiment often go unpublished. The full methodology and statistical analysis from both experiments are documented in the accompanying paper (Clouatre, 2026).

Where the Boundary Falls

The gap is between task types, not between models:

Positional retrieval tasks (NameIndex, needle-in-haystack): high positional attention decay, repetition helps
Structured engineering tasks (scoped issues with code context): low positional decay, Scout already converges correctly

**Table 3:** Comparison of experimental conditions. The paper’s gains concentrate on positional retrieval tasks; our structured engineering tasks hit ceiling effects before any treatment could show lift.
Dimension	Paper (Leviathan et al.)	Experiment 1	Experiment 2
Task type	Standard + custom retrieval (MMLU-Pro, NameIndex, others)	Issue analysis (FastMCP refactor)	Source code synthesis (AST scanner)
Model	Gemini 2.0 Flash-Lite, Claude 3 Haiku, 5 others	Claude 4.5 Haiku	Claude 4.5 Haiku
Sample size	McNemar test on full benchmark datasets (7 benchmarks, 7 models)	n=4 vs n=5 (1 dropped)	n=5 vs n=5
Accuracy delta	47/70 pairs improved, 0 regressed; +76pp on NameIndex (Flash-Lite)	+0.30 (noise)	0.00 (ceiling)
Confounds	None reported	Delegate authoring failure	5-delegate batch cap

Designing a rubric that discriminates between good and very good on the second category is harder than it looks. Both of ours failed. The criteria require synthesis and judgment under genuine ambiguity, not retrieval from a well-scoped document.

What Did We Learn About AI Evaluation Design?

Rubric Design Is Harder Than Experiment Design

We iterated twice and hit the ceiling both times. A 7-point rubric with “source code synthesis” criteria is not automatically harder. It depends on whether the task actually creates ambiguity the agent must resolve. Ours did not.

A practical calibration target: if your scoring delegate can answer any criterion by reading the issue alone (without running the code), the criterion will not discriminate.

Infrastructure Behavior Is a Confounder

The 5-delegate cap is undocumented. It is enforced as a hard rejection in source (GOOSE_MAX_BACKGROUND_TASKS defaults to 5), with no queuing or retry. Excess delegates are dropped, not deferred. It silently split our groups into unbalanced batches. This category of confound (runtime resource limits, queue behavior, model routing) is endemic to agent systems and invisible without structured logging.

Future experiments: spawn delegates in explicit batches of 5 with documented batch assignments. Record session IDs. Treat infrastructure state as a variable, not background noise.

Delegate Authoring Has a Turn-Length Problem

Long sessions drift from instructions that appear only once. The control-1 failure (93 messages, no output) demonstrated the fix: bookend critical actions at both the start and end of delegate prompts. This class of failure is predictable and preventable, but only if you treat delegate prompt structure as part of your experimental design.

The blind scoring infrastructure proved its value here. Each run produced a structured justification the scorer generated without knowing group assignment:

Code Snippet 3: Blind scorer output for a single run. Each criterion includes a justification generated without knowledge of group assignment.

{
  "run_id": "run-01",
  "C1": 1, "C2": 1, "C3": 1, "C4": 1,
  "C5": 1, "C6": 1, "C7": 1, "total": 7,
  "justifications": {
    "C1": "Issues #735/#736 explicitly identified as regex limitation.",
    "C5": "Backward compatibility addressed via hybrid approach.",
    "C7": "Synthesis connects tree-sitter AST parsing to existing rules."
  }
}scorer-output.json

When Should You Use Prompt Repetition?

The null result is not a failure of the paper. Prompt repetition won 47 out of 70 benchmark-model combinations with zero losses (Leviathan et al., 2025). The technique works. The question is where.

The paper’s gains concentrate on benchmarks with positional retrieval components: NameIndex, MiddleMatch, options-first multiple choice. Tasks where the answer depends on information placement in the context window. The paper also notes a neutral-to-slight effect with reasoning prompts (5 wins, 1 loss, 22 neutral with step-by-step). Reasoning appears to compensate for the same attention decay that repetition addresses.

The industry trend is not exclusively toward reasoning models. Capable SLMs are gaining ground. NVIDIA’s Nemotron 3 Nano (NVIDIA, 2025) activates 3 billion of its 30 billion parameters per token, delivering 3.3x the throughput of Qwen3-30B on a single H200, designed explicitly for multi-agent systems at scale. RLP (Hatamizadeh et al., 2025) embeds reinforcement learning into pretraining itself, lifting math and science accuracy by 19% on a 1.7B-parameter model without post-training reasoning. These models are non-reasoning by default. The causal masking limitation that prompt repetition addresses is structural to the decoder-only architecture all of them share. Their users are also the most cost-sensitive to doubled input tokens. Every token matters when you are optimizing for throughput at the edge.

Our null result came from the other side of that boundary: structured engineering tasks where the agent already has scoped context, code references, and acceptance criteria. The ceiling was in the task, not the technique.

Figure 2: When to use prompt repetition. Three decision points, two outcomes.

What Transfers to Your Team

Three things that transfer directly to any team evaluating AI agent behavior:

Baseline accuracy determines whether any prompt technique has room to work. Measure it before testing an optimization.
Infrastructure constraints are confounder candidates. Audit your delegate system’s limits before attributing latency or throughput differences to treatment variables.
Rubric discrimination is the bottleneck. Two rubrics, two ceiling effects. If your scoring criteria can be satisfied by reading the issue description alone, the rubric will not discriminate.

Did Prompt Repetition Change Anything Else?

One observation worth noting: treatment agents in both experiments used fewer output tokens and messages to reach the same scores. This is consistent with what the attention mechanism would predict; a richer prefill context may reduce exploratory turns in an agentic loop, though the mechanism is not established. The original paper’s benchmarks (MMLU, GSM8K) produce fixed-format answers where output length does not vary, making this effect invisible. Agentic workloads, where the model decides how many turns to take, may be where the efficiency signal surfaces.

The economics are also different than single-turn benchmarks suggest: in a multi-turn session, the doubled prompt adds single-digit overhead to accumulated input, not 100%. The growing conversation history dominates each API call. In our data, treatment agents used 13.1% fewer input tokens and 15.4% fewer output tokens despite the longer prompt. Each avoided turn eliminates an entire context window from the running total. The effect is confounded and too small to draw conclusions, but it is a pattern worth investigating with a discriminating rubric.

The evaluation methodology described in this post is the subject of Canadian patent application CA 3315358, filed Jun 17, 2026 (CIPO).

References

BCG, “AI Adoption Puzzle: Why Usage Is Up But Impact Is Not” (2025) — https://www.bcg.com/publications/2025/ai-adoption-puzzle-why-usage-up-impact-not
Berger et al., “Position: Embracing Negative Results in Machine Learning” (2024) — https://arxiv.org/abs/2406.03980
Clouatre, H., “Orchestrating AI Agents: A Subagent Architecture for Code” (2025) — https://clouatre.ca/posts/orchestrating-ai-agents-subagent-architecture/
Clouatre, H., “Ceiling Effects and Convergence: Null Results for Instruction Repetition in LLM-Agent Pipelines” (2026) — https://doi.org/10.5281/zenodo.20039271
Clouatre, H., “Prompt Repetition Experiments: Supplementary Materials and Dataset” (2026) — https://doi.org/10.5281/zenodo.19696593 — https://github.com/clouatre-labs/prompt-repetition-experiments
Hatamizadeh, A. et al., “RLP: Reinforcement as a Pretraining Objective” (2025) — https://arxiv.org/abs/2510.01265
Kim, Y. et al., “Guiding Reasoning in Small Language Models with LLM Assistance” (2025) — https://arxiv.org/abs/2504.09923
Leviathan, Y., Kalman, M., and Matias, Y., “Prompt Repetition Improves Non-Reasoning LLMs” (2025) — https://arxiv.org/abs/2512.14982
Liu et al., “Lost in the Middle: How Language Models Use Long Contexts” (2023) — https://arxiv.org/abs/2307.03172
NVIDIA, “Nemotron 3: Efficient and Open Intelligence” (2025) — https://arxiv.org/abs/2512.20856

Why Your AI Agent Failed in Production

Hugues Clouâtre — Tue, 16 Jun 2026 19:10:00 GMT

Your AI agent just approved a $50,000 invoice for office supplies. A legitimate vendor. The PO number matches. But the quantity is wrong by a factor of 10. By the time finance catches it, you’ve already paid, the goods already shipped, and you’re stuck negotiating a return.

The agent’s logs show “decision: approved” but nothing about why it ignored the quantity anomaly that a human would have caught instantly. Without proper instrumentation, root cause analysis stretches from minutes to days. This is what happens when observability is treated as “nice to have” instead of foundational infrastructure.

This post covers the production architecture, the vendor-neutral stack, and why you need to instrument before deployment, not after the first failure.

Contents

Why Should Observability Be Foundational Infrastructure?
- The Observer Effect Paradox
What Is Decision Provenance and Why Does Compliance Require It?
How Do Silent Integration Failures Kill AI Agents?
Why Do Token Costs Spiral Out of Control?
- Alerting on Token Budgets
What Does a Vendor-Neutral Observability Stack Look Like?
- How the Components Connect
- Why Trace-Metric Correlation Matters
What Is the ROI and How Do You Start?
Are You Ready for Production?
References

Why Should Observability Be Foundational Infrastructure?

When your agent fails in production, you need to answer three questions immediately: What decision did it make? What data did it use? How much did it cost? But monitoring only answers questions you already knew to ask. The failures that damage production most, a model quietly conserving tokens by skipping reasoning steps, an upstream API silently returning empty results, a prompt change that fixes one behavior while degrading another, never trigger an alert. They show up as patterns across thousands of traces, invisible to any dashboard you configured in advance.

The Observer Effect Paradox

Instrumentation changes what you measure. Synchronous logging to external systems adds latency to every LLM call. In multi-agent systems, this can trigger timeout-based retries where observability causes the failures it detects.

OpenTelemetry’s BatchSpanProcessor solves this by queuing spans in memory and exporting in batches, minimizing per-request overhead.

What Is Decision Provenance and Why Does Compliance Require It?

How do you prove your AI agent made the right decision six months ago when a regulator asks? Logging outputs without reasoning fails every major compliance framework.

Why Every Framework Requires Decision Trails

Every major compliance framework mandates reconstructible reasoning.

SOC 2 Type II: audit trails of system access and user activity. The gen_ai.conversation.id attribute ties every decision to a user and timestamp.
GDPR Article 30: records of processing activities. Structured logs with trace IDs link inputs to outputs.
HIPAA: audit controls for ePHI access. Span attributes capture what data the agent accessed.
PCI DSS 4.0.1 Requirement 10: tracking cardholder data access with automated log reviews. Prometheus metrics enable real-time anomaly detection.

Structured Logging with Correlation IDs

The fix links every decision to its inputs.

Code Snippet 1: OpenTelemetry structured logging captures decision provenance with trace IDs, span attributes, and correlation across the entire request lifecycle.

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
import logging

tracer = trace.get_tracer(__name__)
logger = logging.getLogger(__name__)

def make_decision(invoice_data, retrieved_context):
    with tracer.start_as_current_span("make_decision") as span:
        span.set_attribute("invoice.id", invoice_data["id"])
        span.set_attribute("invoice.amount", invoice_data["amount"])
        span.set_attribute("context.sources", len(retrieved_context))
        
        decision = analyze(invoice_data, retrieved_context)
        confidence = calculate_confidence(decision)
        
        span.set_attribute("decision.result", decision["action"])
        span.set_attribute("decision.confidence", confidence)
        
        logger.info(
            "Decision made",
            extra={
                "trace_id": format(span.get_span_context().trace_id, "032x"),
                "invoice_id": invoice_data["id"],
                "decision": decision["action"],
                "confidence": confidence,
                "context_count": len(retrieved_context)
            }
        )
        
        return decisionagent/decision_logger.py

This gives you a complete audit trail: trace ID links the decision to all upstream data retrievals, span attributes capture the decision logic, and structured logs provide queryable records. When the regulator asks “why did you approve invoice #12345?”, you can show exactly what data the agent saw and how it weighted each factor.

Tracking Tool Calls with GenAI Semantic Conventions

Multi-agent systems make dozens of tool calls per decision. OpenTelemetry’s GenAI semantic conventions provide standard attributes for tracking these interactions:

Code Snippet 2: GenAI semantic conventions enable cross-platform analysis across LangChain, LlamaIndex, and custom agents.

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def execute_tool_call(tool_name, arguments, conversation_id):
    with tracer.start_as_current_span("execute_tool") as span:
        # Standard GenAI attributes
        span.set_attribute("gen_ai.operation.name", "execute_tool")
        span.set_attribute("gen_ai.tool.name", tool_name)
        span.set_attribute("gen_ai.conversation.id", conversation_id)
        span.set_attribute("gen_ai.tool.call.arguments", str(arguments))
        
        result = call_tool(tool_name, arguments)
        
        span.set_attribute("gen_ai.tool.call.result", str(result))
        
        return resultagent/tool_tracking.py

Standard attributes like gen_ai.tool.name let you answer operational questions across your entire stack: “Which tools fail most often?” or “Which conversations require the most tool calls?” When you swap frameworks, your dashboards still work.

How Do Silent Integration Failures Kill AI Agents?

Your AI agent calls a legacy API that returns HTTP 200 with an empty result set. The agent interprets “no data” as “no problem” and proceeds. But the API actually failed silently because the database connection pool was exhausted. By the time you notice, you’ve processed 500 transactions with incomplete data.

AI agents don’t fail loudly. They fail gracefully, hiding problems until they cascade. You need distributed tracing that correlates agent decisions with integration health across every dependency.

Code Snippet 3: Distributed tracing propagates correlation IDs and flags suspicious patterns like empty responses from healthy endpoints.

from opentelemetry import trace, propagate
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer(__name__)

def retrieve_from_legacy_api(query):
    with tracer.start_as_current_span("legacy_api_call") as span:
        span.set_attribute("api.endpoint", "/legacy/search")
        span.set_attribute("query", query)
        
        headers = {}
        propagate.inject(headers)  # Inject trace context 
        
        response = requests.get(
            "https://legacy.example.com/search",
            params={"q": query},
            headers=headers
        )
        
        span.set_attribute("http.status_code", response.status_code)
        span.set_attribute("response.size", len(response.content))
        
        if response.status_code == 200 and len(response.json()) == 0:
            span.set_status(Status(StatusCode.ERROR, "Empty result set"))
            span.add_event("Suspicious empty response from healthy endpoint")
        
        return response.json()agent/trace_integration.py

Correlation ID propagation (line 12) and explicit error marking for suspicious patterns (lines 23-25) are what matter. When you see a spike in “empty result set” errors correlated with database saturation metrics, you know the integration is degraded even though HTTP status codes look fine. The harder failure is invisible: when an upstream API returns an error, some models fabricate a plausible answer rather than retrying. Token costs stay flat, latency looks normal, and your users receive confident, invented responses. Span attributes on both the tool call result and the subsequent LLM generation, correlated by trace ID, are the only way to detect it.

Why Do Token Costs Spiral Out of Control?

Your agent works in testing. Then production traffic hits and your LLM bill explodes. AI costs are surging 36% year-over-year, yet only half of organizations can confidently evaluate ROI (CloudZero, 2025). Without per-operation cost tracking, you can’t identify which workflows are burning money.

Consider a Claude 4.5 Sonnet deployment: input tokens cost $3/million, output tokens cost $15/million. A single complex query might use 50K input tokens and 4K output tokens, costing $0.21. At 10,000 queries per day, that’s $2,100 daily, or $63,000 monthly, just for one workflow. If your agent retries on failures or chains multiple calls, costs multiply fast.

Code Snippet 4: Prometheus metrics track token usage with labels for model, operation, and user tier, enabling real-time cost monitoring and per-operation granularity to prevent budget overruns.

from prometheus_client import Counter, Histogram

# Token counter with model and operation labels
tokens_total = Counter(
    'ai_tokens_total',
    'Total tokens consumed',
    ['model', 'operation', 'user_tier']
)

# Latency histogram with cost correlation
request_duration = Histogram(
    'ai_request_duration_seconds',
    'Request duration',
    ['operation'],
    buckets=[0.1, 0.5, 1.0, 2.0, 5.0, 10.0, float('inf')]
)

def process_query(query, user_tier):
    with request_duration.labels(operation='query').time():
        embedding = embed(query)
        tokens_total.labels(
            model='text-embedding-3-small',
            operation='embed',
            user_tier=user_tier 
        ).inc(len(query.split()))
        
        results = vector_search(embedding)
        
        response = generate_response(results)
        tokens_total.labels(
            model='claude-4.5-sonnet',
            operation='generate',
            user_tier=user_tier 
        ).inc(response['usage']['total_tokens'])
        
        return responseagent/metrics.py

Alerting on Token Budgets

Labels let you slice cost by model, operation, and user tier. When free-tier token usage spikes on expensive models, you can throttle, switch to cheaper models, or convert users to paid tiers before costs spiral.

Code Snippet 5: Prometheus alerting rule triggers when any user tier exceeds 1,000 tokens per second sustained over 5 minutes, enabling proactive cost control before budget overruns.

groups:
- name: ai-cost-alerts
  rules:
  - alert: TokenBudgetExceeded
    expr: sum(rate(ai_tokens_total[5m])) by (user_tier) > 1000
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Token budget exceeded for {{ $labels.user_tier }}"
      description: "{{ $labels.user_tier }} tier consuming {{ $value }} tokens/sec"ai-alerts.yml

What Does a Vendor-Neutral Observability Stack Look Like?

Enterprise platforms like Datadog and Splunk offer polished, integrated experiences. For teams prioritizing cloud-native portability, OpenTelemetry handles instrumentation, Prometheus stores metrics, and Grafana visualizes everything. Zero licensing cost, no vendor lock-in, and production-proven.

Already invested in an enterprise platform? OpenTelemetry collectors export directly to these platforms, preserving full trace context and semantic attributes. You can adopt incrementally without disrupting existing dashboards, gaining richer observability now and portability for future migrations.

Figure 1: OpenTelemetry + Prometheus + Grafana stack

How the Components Connect

Your agent emits traces, metrics, and logs via OpenTelemetry SDKs. The OpenTelemetry Collector receives, processes, and routes telemetry to backends. Prometheus scrapes metrics and stores time-series data. Grafana queries Prometheus for metrics, Tempo for traces, and Loki for logs, correlating them in unified dashboards.

Why Trace-Metric Correlation Matters

When a user reports “the agent is slow”, start in Grafana, filter metrics by user ID, see elevated p95 latency, drill down to the request, and find the bottleneck in 30 seconds. Without correlation, you’re grepping logs for hours.

Figure 2: Trace ID propagation with per-operation latency breakdown

Every request gets a correlation ID that propagates through RAG retrieval, API calls, and decision logic. When you need to audit a decision, query by trace ID to reconstruct the entire flow: what data was retrieved, which APIs were called, response times, and token consumption.

What Is the ROI and How Do You Start?

Think of AI observability as a hierarchy: logging lets you see that something happened, monitoring alerts you to known problems in real time, and analytics surfaces the failures you did not know to look for. Most teams stop at monitoring and call it done. Scott Clark, founder of Distributional, calls this a “Maslow’s hierarchy of observability,” and the analytics layer is precisely the one most production failures hide in. The vendor-neutral stack below covers all three.

Setup cost is roughly 40-80 hours of engineering time, with payback in 1-3 months. Forrester TEI studies on comparable observability investments show 165-201% ROI (Forrester, 2020-2022). This stack delivers comparable returns with full portability.

**Table 1:** AI Observability ROI
Without	With
$5K-$20K/month untracked token spend	Per-request cost attribution
4-8 hours debugging per incident	30 minutes with trace correlation
$20K-$50K manual audit reconstruction	Query-ready decision logs

Start small: instrument one critical path (the highest-risk decision your agent makes) with decision provenance logging. Add integration health tracing for your most fragile API dependency. Implement cost tracking for your most expensive model. Expand based on what breaks. This is the same incremental approach I described in my AI agents ROI post: start with 5% of workflows, prove value, then scale.

Decision provenance, integration health, and cost runaway are not edge cases. They cause production AI failures. Fix them before deployment, not after the invoice arrives.

Are You Ready for Production?

Before your next AI deployment, verify these capabilities:

Decision provenance: High-risk workflows log inputs, reasoning, and outputs with trace IDs using OpenTelemetry and structured logging
Integration health: Distributed tracing covers legacy APIs and third-party services, with alerts on silent failures like empty responses from healthy endpoints
Cost attribution: Token usage tracked per model, operation, and user tier, with budget alerts via Prometheus metrics
Audit reconstruction: Any decision from the past 12 months can be fully reconstructed in under 30 minutes
Behavioral drift: You have a mechanism to detect when agent behavior shifts across deployments, not just latency or cost, but output patterns and tool call sequences

If you can’t confirm all five, you’re not ready for production.

For the SRE framework that operationalizes decision provenance with error budgets and trust ladders, see SRE for AI Agents: Error Budgets, Trust, and 90 Trials.

References

AICPA, “2017 Trust Services Criteria (With Revised Points of Focus - 2022)” (2022) - https://www.aicpa-cima.com/resources/download/2017-trust-services-criteria-with-revised-points-of-focus-2022
Clark, Scott, “How to Find the Agent Failures Your Evals Miss”, TWiML AI Podcast (2025) - https://twimlai.com/podcast/twimlai/how-find-agent-failures-your-evals-miss
CloudZero, “The State of AI Costs in 2025” (2025) - https://www.cloudzero.com/state-of-ai-costs/
European Union, “GDPR Article 30: Records of Processing Activities” (2018) - https://gdpr-info.eu/art-30-gdpr/
Forrester Consulting, “The Total Economic Impact of Chronosphere” (2022) - https://chronosphere.io/forrester-total-economic-impact-chronosphere/
Forrester Consulting, “The Total Economic Impact of Microsoft Sentinel” (2020) - https://tei.forrester.com/go/microsoft/microsoft_sentinel/
Kiteworks, “HIPAA Audit Logs: Complete Requirements for Healthcare Compliance in 2025” (2025) - https://www.kiteworks.com/hipaa-compliance/hipaa-audit-log-requirements/
OpenTelemetry, “Semantic Conventions for Generative AI Systems” — https://opentelemetry.io/docs/specs/semconv/gen-ai/
OpenTelemetry, “Tracing SDK Specification” — https://opentelemetry.io/docs/specs/otel/trace/sdk/
PCI Security Standards Council, “PCI DSS v4.0.1” (2024) - https://docs-prv.pcisecuritystandards.org/PCI%20DSS/Standard/PCI-DSS-v4_0_1.pdf

RAG for Legacy Systems: 7,432 Pages to 3s Answers

Hugues Clouâtre — Tue, 16 Jun 2026 18:19:00 GMT

Your legacy system documentation is 20 years old, 7,432 pages, and locked in PDFs. Manual search takes 15-30 minutes per query. We made it queryable in 170 seconds. Query response time: 3-5 seconds. ROI break-even: one day.

This isn’t a prototype. It’s Retrieval-Augmented Generation (RAG) on Amazon Bedrock, a system that retrieves relevant documentation and uses an LLM to generate answers without retraining models. Validated across four LLM families with 480 measurements. The implementation indexes 20,679 chunks and delivers sub-5-second responses with model-agnostic reranking. Overhead: 27.2ms ± 4.6ms regardless of which LLM you use.

Yes, 7,432 pages fit in any search index. But ranked results aren’t answers.

Here’s the production architecture, the multi-model validation data, and why you can switch providers without re-tuning.

Contents

Why RAG, Not Fine-Tuning?
How Does RAG Turn PDFs Into Answers?
Does Reranking Work Across Different Models?
What Are the Real Performance Numbers?
What’s the ROI Without Modernization?
When Does RAG Fail?
How Do You Migrate from Prototype to Production?
What Should You Do Next?
References

Why RAG, Not Fine-Tuning?

Fine-tuning trains a model on your docs. It bakes knowledge into weights (making provenance verification difficult), requires retraining for every update, and costs $1.32-6.24 per run on A100 GPUs (Thunder Compute, 2025). RAG costs $0 setup with local embeddings, $0.0011 per query on Bedrock, updates in 2 seconds, and keeps sources verifiable.

For legacy systems, choose RAG for operational factors, not economics. Documentation evolves as reverse-engineering uncovers new system behaviors; fine-tuning would require retraining each time. The deciding factors: instant updates (2 seconds vs retraining), source citations for compliance, and simpler maintenance.

A common objection: modern LLMs have million-token context windows, so why not load all 7,432 pages and skip retrieval entirely? Two reasons. First, relevance ranking still matters: injecting the full corpus degrades answer quality because the LLM attends poorly to relevant passages buried in noise. Second, compliance requires provenance, knowing not just what the answer is, but which document and page it came from. A context dump cannot produce a citable source. Retrieval can.

How Does RAG Turn PDFs Into Answers?

The Ingestion Pipeline

The pipeline has six stages:

Extract - Pull text from PDFs using PyMuPDF (44 pages/second)
Transform - Convert to Markdown with heading detection
Chunk - Split at headings (1,000-char limit, 200-char overlap)
Embed - Generate vectors with all-MiniLM-L6-v2 (local, free)
Index - Store in ChromaDB vector database
Retrieve - Hybrid search (BM25 + vector) with FlashRank reranking

Code Snippet 1: PyMuPDF extracts text and converts to Markdown with heading detection, processing 44 pages/second.

import fitz  # pymupdf

doc = fitz.open(pdf_path)
for page_num in range(len(doc)):
    page = doc[page_num]
    text = page.get_text()
    
    for line in text.split("\n"):
        # Detect chapter headings (e.g., "Chapter 1. Title")
        if line.startswith("Chapter ") and ". " in line:
            cleaned_lines.append(f"\n## {line}\n")
        # Detect section headings
        elif len(line) < 80 and line[0].isupper():
            cleaned_lines.append(f"\n### {line}\n")
doc.close()src/ingest.py

The pipeline converts PDFs to Markdown before chunking, preserving chapter and section hierarchy. Chunks split at heading boundaries instead of mid-paragraph, keeping related content together. Storing the parent section per chunk allows the retriever to expand context upward during reranking, useful when an answer spans a subsection and its parent chapter.

Hybrid Retrieval: BM25 + Vector Search

Why not just Elasticsearch or Ctrl+F? Pure keyword search fails when you search “memory error” but the 2005 docs say “data file cache exhaustion.” Pure vector search misses exact terms like “port 5432.” Dense retrieval (ChromaDB) handles conceptual queries; sparse (BM25) ensures exact references for compliance and explainability. Legacy terminology drift makes both necessary. Hybrid retrieval solves ranking. The LLM solves synthesis: combining fragments from multiple documents into an actionable answer. Reciprocal Rank Fusion (RRF) consistently outperforms single-method search (Mandikal & Mooney, 2024).

Code Snippet 2: RRF formula combines keyword and semantic search scores with k=60 constant.

# RRF combines BM25 + vector scores
doc_scores: dict[str, tuple[Document, float]] = {}

for rank, idx in enumerate(bm25_top_indices[:retrieve_k]):
    doc = chunks[idx]
    doc_id = doc.metadata.get("source", "") + str(hash(doc.page_content[:100]))
    rrf_score = 1 / (rank + 60)  # RRF with k=60
    if doc_id in doc_scores:
        doc_scores[doc_id] = (doc, doc_scores[doc_id][1] + rrf_score)
    else:
        doc_scores[doc_id] = (doc, rrf_score)src/rag.py

Hybrid retrieval returns 16 candidate chunks. A cross-encoder model (FlashRank) scores each query-document pair and returns the top 8. This fixes the precision problem: high recall from hybrid search, high precision from reranking.

Example: Error Lookup in 3.4 Seconds

User query: “What is error 1006030 and how do I fix it?”

Generated answer:

Error 1006030: “Failed to bring a data file page into cache. Data file cache is too small.”

Cause: Essbase cannot store the data file page in the data file cache.

Solution: Increase the data file cache size. After fixing, check for database corruption (Error Message Reference, p. 126).

Timing: 3.4s total (retrieval: 80ms, reranking: 31ms, generation: 3.3s)

Retrieved from: Error Message Reference v11.1.1 (ranked 3rd of 8 after reranking)

The system retrieved error 1006030 from the Error Message Reference (ranked 3rd of 8 after reranking) and synthesized an actionable answer. Manual search would require opening the 1,200-page Error Message Reference PDF and using Ctrl+F. FlashRank reranking adds 31ms overhead for a 6-8% accuracy gain (George, 2025).

Figure 1: RAG pipeline with hybrid retrieval and reranking (FlashRank adds 31ms overhead for precision gain).

Local Embeddings and Model-Agnostic Design

Why local embeddings? Cost, simplicity, and performance. Cloud embedding APIs charge $0.10-0.50 per million tokens. Local models are free, require no API keys, and embed 1,000 chunks in under 10 seconds on CPU. The all-MiniLM-L6-v2 model is 80 MB and runs without GPU acceleration. FlashRank follows the same logic: local cross-encoder, no API call, no per-request cost.

The architecture is model-agnostic by design. We use Amazon Bedrock, but the same pipeline works with Azure OpenAI, Google Vertex AI, or local models.

Does Reranking Work Across Different Models?

We tested four LLM families across two providers (Amazon Bedrock, OpenRouter) to validate portability. Mean latency: 27.2ms ± 4.6ms across 480 measurements, with no statistically significant difference (ANOVA p=0.34). Cross-provider variance was only 4.1ms.

**Table 1:** Latency is consistent across models and providers (480 measurements, ANOVA p=0.34).
Model	Family	Latency	Provider
Claude Haiku 4.5	Anthropic	+31.3ms	Amazon Bedrock
Mistral Devstral-2512	Mistral	+32.5ms	OpenRouter
Llama 3.3 Instruct	Meta	+24.1ms	OpenRouter
Qwen 2.5 Coder	Alibaba	+25.1ms	OpenRouter

The latency is dominated by FlashRank’s cross-encoder, not the LLM. This means you implement once and switch providers without re-tuning.

Code Snippet 3: FlashRank reranks 16 candidates in 31ms using cross-encoder scoring.

from flashrank import Ranker, RerankRequest

def _rerank(self, query: str, docs: list[Document]) -> list[Document]:
    passages = [
        {"id": i, "text": doc.page_content, "meta": doc.metadata}
        for i, doc in enumerate(docs)
    ]
    
    rerank_request = RerankRequest(query=query, passages=passages)
    results = self.ranker.rerank(rerank_request)
    
    return [docs[result["id"]] for result in results[:RERANK_TOP_N]]src/rag.py

Reranking is infrastructure, not model-specific configuration. Cross-encoder reranking (FlashRank) is the right starting point; for higher-stakes queries, an LLM-as-judge reranker scores query-document pairs using semantic reasoning rather than similarity, at higher cost but better precision on complex domain questions.

What Are the Real Performance Numbers?

With model-agnostic reranking validated, here are the production metrics.

We indexed 7,432 pages in 170 seconds. First-time setup includes PDF extraction (120s), chunking (20s), embedding (25s), and indexing (5s). Cached runs skip extraction and take 2.2 seconds. Query response time averages 3-5 seconds: retrieval (80ms), LLM generation (4s), overhead (200ms).

Cost per query is $0.01-0.05 on Amazon Bedrock. Input tokens (context from retrieved chunks) cost $0.25 per million. Output tokens (LLM answer) cost $1.25 per million. A typical query uses 2,000 input tokens and 500 output tokens, totaling $0.0011.

**Table 2:** Performance metrics across two production RAG systems (System A handles technical docs, System B processes meeting notes).
Metric	System A (Docs)	System B (Notes)
Documents	14 PDFs (7,432 pages)	94 markdown files
Chunks	20,679	Auto-chunked
First run	170s	40s
Cached run	2.2s	2s
Query time	3-5s	3-5s
Cost/query	$0.01-0.05	$0.01-0.05

Reranking adds 31ms to retrieval time. That’s a 65% increase in retrieval latency but only 0.3% of total query time. Users don’t notice 31ms in a 9-second end-to-end response. The 6-8% accuracy improvement compounds with hybrid retrieval’s gains over single-method search, making the overhead negligible compared to the final quality benefit. For detailed methodology and raw data, see Supplementary Materials.

What’s the ROI Without Modernization?

Manual search through 7,432 pages takes 15-30 minutes (median: 25 min). You open PDFs, use Ctrl+F, read context, cross-reference sections. RAG reduces this to 3-5 seconds.

Assume 10 queries per day during a 6-month migration project. Labor cost: $100/hour (mid-market technical consultant). Time saved: 25 minutes per query. Success rate: 87% (28 of 32 evaluation queries returned useful results; 4 required human review).

Daily savings: 10 queries × 25 min × ($100/hr ÷ 60) × 87% success rate = $362/day

Setup cost: 170 seconds of compute time plus $0 for local embeddings. Query cost: $0.01-0.05 on Amazon Bedrock. Break-even happens in one day.

When Does RAG Fail?

RAG fails on multi-step reasoning, ambiguous questions, and knowledge not in the docs. We’ve seen three failure modes in production.

Hallucination

The LLM invents answers not in the retrieved chunks. Mitigation: constrain responses to retrieved context only, and surface source citations with document name and page number for every answer. In compliance contexts, citations are not a UX nicety; they are an audit trail. A user who can see “Error Message Reference v11.1.1, p. 126” can verify, escalate, or dispute the answer. A context-window dump cannot produce that chain of custody.

Context Overflow

Complex queries need more context than fits in the LLM’s window. Mitigation: break queries into sub-questions, use query expansion for domain terms, implement multi-hop retrieval for connected concepts.

Drift

Two kinds of drift affect production RAG. Data drift: documentation changes but embeddings don’t update; mitigate with hash-based cache invalidation for PDFs and timestamp-based re-indexing for markdown. Model drift: generation changes between model versions (or silent provider-side updates) alter answer tone and structure, requiring prompt recalibration. Maintain a regression test set of known queries and expected answers; run it on every model or prompt change.

Corpus Limitations

Not all failures are system failures. The evaluation revealed three corpus-related issues:

Corpus gap: Knowledge doesn’t exist (e.g., specific error codes not documented). The system correctly responds “I don’t know.”
Scattered information: Knowledge exists but spread across sections, making synthesis incomplete.
Query formulation: Symptom-based queries (“out of memory errors”) outperform code-based queries (“error 1012001”) when exact codes aren’t indexed.

These are honest limitations, not RAG failures. The mitigation is corpus expansion, not system tuning.

What Is the Overall Failure Rate?

12.5% of queries (4 of 32) need human review, primarily for multi-step reasoning or ambiguous questions. The alternative is searching 7,432 pages manually. RAG handles the straightforward cases autonomously, while experts focus on edge cases.

**Table 3:** 32 scored queries across 4 categories (n=8 each), 98.1% ground truth accuracy, 0% false positive rate. Success rate includes partial matches. (methodology).
Query Category	Success Rate	Common Failure Mode	Mitigation
Error lookup	62%	Exact code not in corpus	Symptom-based queries; corpus expansion
Conceptual	100%	Rare; scattered information	Query expansion with domain terms
Procedural	100%	None observed (n=8)	Query expansion with command names
Multi-hop	88%	Knowledge scattered or missing	Corpus expansion; honest “not found”

The operational metric that matters in compliance settings is not internal test accuracy; it is the expert adjustment rate: the percentage of answers a subject-matter expert had to correct before use. Every point of improvement here maps directly to labor cost avoided and liability reduced.

How Do You Migrate from Prototype to Production?

We started on OpenRouter’s free tier. Model: Devstral-2512. Cost: $0. Limits: rate-limited, no compliance guarantees. We validated quality with 20-30 test queries.

Migration to Amazon Bedrock took under 30 minutes. Code changes: swap dependencies (langchain-openai to langchain-aws), replace ChatOpenAI with ChatBedrock, update authentication to use AWS credentials instead of API keys. Benefits: no rate limits, SOC 2 compliance, governance controls, better answer quality from Claude Haiku 4.5.

The migration path: start small with one document set and one use case. Validate quality with test queries comparing RAG answers to ground truth from source documents. Measure adoption by tracking query volume and user feedback. Iterate by adding more docs, tuning chunking strategy, and improving retrieval.

Figure 2: Migration path from free tier validation to enterprise production (iterate on quality before investing in infrastructure).

A common alternative at this stage is an AI agent that searches a file system directly. Agents work when the corpus is small and well-structured. When document volume grows, query specificity demands ranked retrieval, or answers require a citable source, a proper retrieval pipeline outperforms file-system search. The compliance argument is unambiguous: agents browsing files produce no provenance record.

What Should You Do Next?

Identify high-value document sets. Look for onboarding materials, compliance docs, or migration guides. Estimate ROI using queries per day, time saved per query, and hourly labor cost. If the math works, start with a free tier.

Use OpenRouter or local models for validation. Run 20-30 test queries. Compare RAG answers to ground truth from source documents. Measure accuracy, check for hallucinations, verify source citations. If quality is acceptable, invest in enterprise infrastructure.

Amazon Bedrock and Azure OpenAI offer compliance, governance, and better models. Cost is $0.01-0.05 per query. For 100 queries per day, that’s $1-5 daily or $30-150 monthly. Compare that to $9,000 in labor savings.

The decision framework: RAG wins when documentation changes frequently, source citations matter for compliance, or you need operational agility. Fine-tuning wins when knowledge is stable, you need specialized behavior beyond retrieval, or query volume is extreme (thousands per day) with strict latency requirements.

For legacy systems, RAG delivers ROI without modernization. No need to rewrite docs, migrate databases, or retrain staff. Layer RAG over existing PDFs and get 3-second answers to 20-year-old questions.

For broader integration patterns and ROI frameworks, see AI Agents in Legacy Systems: ROI Without Modernization.

References

Braintrust, “RAG Evaluation Metrics: How to Evaluate Your RAG Pipeline” (2025) — https://www.braintrust.dev/articles/rag-evaluation-metrics
Clouatre, H., “RAG Reranking Benchmarks: Supplementary Materials” (2026) — https://doi.org/10.5281/zenodo.19056880 — https://github.com/clouatre-labs/rag-reranking-benchmarks
de Luis Balaguer et al., “RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture” (2024) — https://arxiv.org/abs/2401.08406
Dettmers et al., “QLoRA: Efficient Finetuning of Quantized LLMs” (2023) — https://arxiv.org/abs/2305.14314
Gan et al., “Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey” (2025) — https://arxiv.org/abs/2504.14891
George, Sherine, “Enhancing Retrieval-Augmented Generation with Two-Stage Retrieval: FlashRank Reranking and Query Expansion” (2025) — https://arxiv.org/abs/2601.03258
LangChain Documentation, “Contextual Compression and Reranking” (2025) — https://python.langchain.com/docs/how_to/contextual_compression/
Mandikal & Mooney, “Sparse Meets Dense: A Hybrid Approach to Enhance Scientific Document Retrieval” (2024) — https://arxiv.org/abs/2401.04055
Oche et al., “A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions” (2025) — https://arxiv.org/abs/2507.18910
Thunder Compute, “AI GPU Rental Market Trends December 2025: Complete Industry Analysis” (2025) — https://www.thundercompute.com/blog/ai-gpu-rental-market-trends

AI Agents in Legacy Systems: ROI Without Modernization

Hugues Clouâtre — Sat, 20 Jun 2026 00:00:00 GMT

You run a company with SAP, mainframe, or AS400 systems that work but won’t win awards. The board wants AI. Your team wants modernization budgets. You’re stuck in the middle.

Every AI agent case study assumes clean APIs, cloud-native apps, and real-time data. Your world is batch jobs, COBOL, and integration layers built in 2003. The conventional answer is “modernize first, then AI.” That’s a 2-5 year, $5M-$50M bet before you prove a single dollar of AI value.

This post shows where AI agents make economic sense on top of legacy systems, how to measure ROI without enterprise-wide transformation, and which integrations work when your data lives in places LLMs have never heard of. You’ll walk away with a decision framework for identifying agent use cases that pay back in quarters, not years.

Contents

Why Legacy Systems Became the #1 AI Adoption Obstacle
The Reverse Modernization Strategy: Layer AI First, Upgrade Later
- Why This Works
- When Reverse Modernization Doesn’t Apply
How Do AI Agents Actually Integrate with Legacy Systems?
Why Observability Infrastructure Is Non-Negotiable
What ROI Can You Actually Expect?
Why Do 40% of AI Agent Projects Still Fail?
How to Start: A Practical Implementation Framework
Why Should Your Board Fund This Now?
References

Why Legacy Systems Became the #1 AI Adoption Obstacle

Legacy systems top the list of AI adoption obstacles, but the conventional fix is worse than the problem. Traditional modernization projects require multi-year timelines and eight-figure budgets before proving a single dollar of AI value. No wonder 40% of agentic AI projects will be canceled by 2027 (Gartner, 2025) due to escalating costs and unclear business value.

The real bottleneck isn’t legacy systems. It’s the false choice between “modernize everything” and “do nothing.” You need integration patterns that work with what you have.

The Reverse Modernization Strategy: Layer AI First, Upgrade Later

Layer AI agents over existing systems first. Capture ROI in months, then fund selective modernization. Prove value before investing in infrastructure.

Allianz’s Project Nemo reduced food spoilage claims processing from four or more days to hours (Allianz, 2025), an 80% reduction, using seven specialized agents with a human making the final payout decision. The business case was proven before committing to deeper modernization.

Why This Works

Reverse modernization works because legacy systems keep running (no disruption), ROI arrives in 3-6 months (30-80% productivity gains), and you start small: one workflow, one team, one agent.

Modernization is a binary bet. Agent layering is incremental: prove value, fund upgrades, repeat.

When Reverse Modernization Doesn’t Apply

Three scenarios require modernization first. End-of-life systems without vendor support expose you to compliance violations and security breaches (Cyber Snowden, 2026). Agent integration can’t fix missing security patches. Regulatory mandates that explicitly require infrastructure upgrades (e.g., PCI-DSS 4.0, GDPR data residency) make layering non-compliant. Systems scheduled for decommissioning within 12 months don’t justify integration investment. In these cases, accelerate modernization or sunset the system entirely.

For everything else, reverse modernization applies.

Figure 1: Reverse modernization flow (agents first, infrastructure later)

How Do AI Agents Actually Integrate with Legacy Systems?

But integration is where most projects fail. Agents need access to data and business logic buried in legacy systems. You have three options. Each has tradeoffs.

API Mediation Layer

Build a facade that abstracts legacy complexity. Agents interact with clean, modern interfaces while the mediation layer handles authentication, data translation (EBCDIC to UTF-8, fixed-width to JSON), and error handling. When the legacy system changes, you update the facade, not the agents. You also get a single point for logging, monitoring, and compliance audits.

Code Snippet 1: FastAPI facade translates COBOL fixed-width records to validated JSON.

from fastapi import FastAPI
from pydantic import BaseModel

class Customer(BaseModel):  # Modern JSON schema
    id: str
    name: str
    balance: float

@app.get("/customers/{customer_id}", response_model=Customer)
async def get_customer(customer_id: str) -> Customer:
    raw = legacy_client.call("CUSTINQ", customer_id.ljust(10))
    return Customer(
        id=raw[0:10].strip(),
        name=raw[10:40].encode("cp037").decode("utf-8"),  # EBCDIC to UTF-8  
        balance=int(raw[40:52]) / 100,  # Packed decimal
    )mediation/legacy_facade.py

Event-Driven Architecture

Legacy systems publish state changes through Dapr, which supports Kafka, Azure Event Hub, and others. Agents subscribe and react in near real-time. This pattern scales better than API mediation: the system pushes updates when they matter instead of agents polling constantly. Dapr’s abstraction avoids vendor lock-in.

The tradeoff: you need to instrument the legacy system to publish events, which isn’t trivial if the system is old and undocumented.

Code Snippet 2: Dapr subscriber receives legacy system events via CloudEvents (swap Kafka/RabbitMQ/Azure without code changes).

from dapr.ext.grpc import App
from cloudevents.sdk.event import v1
import json

app = App()

@app.subscribe(pubsub_name="legacy-events", topic="orders")
def handle_order(event: v1.Event) -> None:
    data = json.loads(event.Data())  # CloudEvents envelope  
    # Agent processes order without polling legacy system
    agent.process_order(data["order_id"], data["customer_id"])

app.run(6002)events/order_subscriber.py

Model Context Protocol (MCP)

Anthropic’s open standard for agent-to-data connections. You write one MCP server for your legacy system, and any agent can use it. No custom integration code for each agent. This matters when coordinating multiple agents, a problem I’ve written about in orchestrating multiple AI agents with subagent architecture.

Code Snippet 3: FastMCP 3.0 server exposes legacy data to any MCP-compatible agent (one server, many agents).

from fastmcp import FastMCP  # FastMCP 3.0  

mcp = FastMCP("Legacy ERP")

@mcp.tool
def query_customer(customer_id: str) -> str:
    """Query customer from mainframe. Any MCP-compatible agent can call this."""
    result = mainframe_client.execute(f"SELECT * FROM CUSTMAST WHERE ID='{customer_id}'")
    return json.dumps(result)

if __name__ == "__main__":
    mcp.run(transport="http", port=8000)  # Remote agents connect via HTTPmcp/legacy_server.py

Which Pattern Should You Choose?

The patterns above work differently from platform solutions. You build lightweight facades around legacy systems using open tools (FastAPI, Dapr, MCP). Implementation takes weeks, not quarters. You prove value before committing to platforms.

**Table 1:** Integration pattern selection guide
Pattern	Best When	Timeline
API Mediation	Stable APIs, 1-2 agents, tight control needed	4-8 weeks
Event-Driven	1,000+ transactions/hour, sub-second response	8-12 weeks
MCP	3+ agents, standardization priority	6-12 weeks

When to Consider Platform-Level Solutions

Enterprise platforms like Palantir Foundry take a different approach: containerizing legacy code itself through Foundry Container Engine (FCE). This “lift and shift” strategy runs your COBOL or Fortran logic in a modern environment (Palantir, 2023) without rewriting it.

The tradeoff is vendor commitment. Palantir engagements typically start at $1M+/year and require dedicated integration teams. For organizations with complex, multi-system landscapes and enterprise budgets, this can make sense.

These approaches complement each other. Demonstrate ROI with integration patterns, then justify platform investments for deeper modernization. Deloitte calls this “data-first ERP modernization” (Deloitte, 2025).

Figure 2: Three integration patterns for legacy systems

Why Observability Infrastructure Is Non-Negotiable

Whatever integration pattern you choose, log everything. Every integration call. Every agent decision. Every error. This isn’t optional.

Why Log Everything?

Compliance: Auditors ask “why did your agent approve this transaction?” You need logs showing the decision path.
Debugging: When an agent fails, trace which integration call failed, what data it received, and why it made the wrong decision.
Developer autonomy: Teams that can observe system behavior can configure and tune agents without waiting for platform intervention, a prerequisite for scaling agent ownership.

Track three categories of metrics.

Integration Health Metrics

Monitor API latency (p50, p95, p99), error rates by type, and timeout frequency—metrics that align with OpenTelemetry GenAI conventions.

Agent Performance Metrics

These require deliberate instrumentation design. Task completion rate needs a definition of “complete” per task type (e.g., “ticket resolved without escalation”). User satisfaction comes from thumbs up/down on responses, escalation rate, and support ticket correlation. Decision accuracy is the hardest to measure, see below.

How Do You Measure Decision Accuracy?

Ground truth is often available. The question is where to find it.

For RAG systems, use categorized query benchmarks with validation subsets. Test accuracy by query type (conceptual, procedural, error lookup, multi-hop) since each fails differently.

For approval workflows, compare agent decisions against eventual outcomes. Was the approved invoice paid? Was the flagged transaction actually fraudulent? The business process itself provides ground truth.

When ground truth is unavailable, sample decisions for human or AI-assisted review. The question is: how many?

Sample 300-400 decisions monthly. For most systems this achieves +/-5% margin of error at 95% confidence (Cochran, 1977). For systems under 500 decisions/month, review all or accept wider uncertainty.

Business Impact Metrics

Calculate these monthly against pre-deployment baselines. These are not real-time dashboard metrics:

**Table 2:** Business impact calculation formulas
Metric	Formula
Response time reduction	Agent-handled avg vs. pre-deployment baseline
Throughput increase	Tickets/hour after vs. before deployment
Cost savings	(Hours saved x labor cost) - (API costs + infrastructure)

These metrics prove ROI and guide your next investments. For a complete observability implementation guide, see closing the AI observability gap.

What ROI Can You Actually Expect?

The numbers are compelling. Let me walk through real examples.

Bank of America’s Erica reduced IT service desk calls by 50% (Bank of America, 2025) across 213,000 employees.

BCG reports AI can reduce core insurance modernization costs by 30-50% (BCG, 2026). Agents pay for themselves, then fund the upgrades.

**Table 3:** ROI examples across integration patterns (note: API Mediation dominates early wins due to faster implementation)
Company	Pattern	Metric	Result
Bank of America	API Mediation	IT Service Desk Calls	50% reduction
Allianz (Project Nemo)	Event-Driven	Claims Processing Time	80% reduction (days to hours)
BCG research (cross-industry)	Any	Modernization cost reduction	30-50% lower operational costs

Why Do 40% of AI Agent Projects Still Fail?

Projects fail when teams skip fundamentals.

Unclear Business Value. Gartner predicts over 40% of agentic AI projects will be canceled by 2027 due to escalating costs, unclear value, and inadequate risk controls. Launching with vague goals like “improve productivity” makes it impossible to measure success. Define exact metrics before development: “Reduce invoice processing time from 8 days to 2 days while maintaining 99.5% accuracy” is measurable; “handle invoices better” is not.

Security Vulnerabilities. Prompt injection attacks are ranked #1 in OWASP 2025 Top 10 for LLMs (OWASP, 2025). Treat agents as privileged service accounts with these controls:

Tool allowlisting (no arbitrary network/file access)
Schema validation on tool inputs/outputs
Output sanitization (no untrusted content forwarded)
Secrets isolation (no secrets in prompts; short-lived tokens)
Rate limiting + anomaly detection
Approval gates for high-impact actions
Audit logs (immutable, centralized)

A compromised agent can make thousands of requests per minute. For deeper coverage, see AI supply chain security risks.

Governance Retrofitting. Adding compliance controls after deployment requires painful redesigns. Plan audit trails, role-based access, and compliance testing from the start. Treat governance as a living alignment task: agent standards are evolving and your controls must track them, not just satisfy requirements at launch.

How to Start: A Practical Implementation Framework

Step 1: Set Specific Targets. Pick one workflow with high volume, predictable, with success metrics, and low regulatory risk. Budget onboarding time: the agent tooling learning curve is steep and compounds with each new capability your team adopts.

Step 2: Audit Data Quality. Check for duplicates, format inconsistencies, missing values, and access permissions. Fix the top three issues. Aim for 80% clean data, not perfection.

Step 3: Choose Your Integration Pattern. API mediation for stable APIs and 1-2 agents (4-8 weeks). Event-driven for 1,000+ transactions/hour (8-12 weeks). MCP for 3+ agents or standardization priority (6-12 weeks).

Step 4: Build Observability from Day One. Track integration health (latency, error rates), agent performance (completion rate, accuracy), and business impact (your target KPI). Set alerts for error rates above 5%, latency p95 spikes, and approval override surges.

Step 5: Start with Single-Agent Workflows. Run your first agent in shadow mode for 2-4 weeks (longer for complex workflows). Compare agent decisions against human decisions. Exit criteria: Switch to production when accuracy exceeds 95% (adjust based on cost-of-failure analysis). Production gate: Error rate below 5% and manual override rate trending down. Expansion gate: KPI sustained for 4 weeks with no Sev-1 incidents.

Step 6: Fund Modernization with Agent ROI. Track which legacy systems create the most integration friction. If agents generate measurable savings, reinvest a defined share into infrastructure upgrades. McKinsey finds that top-performing modernizers direct at least one-third of their technology budget to change activities such as modernization and AI platforms (McKinsey, 2026). At this point the challenge shifts: coordinating agents across business lines is a strategic concern, not just a technical one.

Why Should Your Board Fund This Now?

ROI before major infrastructure investment: your board sees results next quarter, not in three years.

Competitive advantage: while competitors wait for budgets, you’re winning deals with faster response times, higher throughput, and lower support costs. Risk mitigation: agents layer over existing systems with no business disruption. Capital One is actively scaling agents to serve millions of customers in production (Shetty, TWiML #765, 2026), a signal that the enterprise agent lifecycle has moved from experimentation to fleet-scale operation.

Explore how subagent architectures can orchestrate multiple AI agents without coordination complexity.

References

Allianz, “When the Storm Clears, So Should the Claim Queue” (2025) — https://www.allianz.com/en/mediacenter/news/articles/251103-when-the-storm-clears-so-should-the-claim-queue.html
Bank of America, “A Decade of AI Innovation: Erica Surpasses Milestones” (2025) — https://newsroom.bankofamerica.com/content/newsroom/press-releases/2025/08/a-decade-of-ai-innovation—bofa-s-virtual-assistant-erica-surpas.html
BCG, “Agentic AI Power Core Insurance AI Modernization” (2026) — https://www.bcg.com/publications/2026/agentic-ai-power-core-insurance-ai-modernization
Cochran, W.G., Sampling Techniques, 3rd ed. (1977) — https://en.wikipedia.org/wiki/Sample_size_determination
Cyber Snowden, “Difference Between End of Life and Legacy Cyber Security” (2026) — https://cybersnowden.com/difference-between-end-of-life-and-legacy-cyber-security/
Deloitte, “Deloitte & Palantir: Driving Value in Enterprise Operations” (2025) — https://www.deloitte.com/us/en/Industries/government-public/perspectives/deloitte-palantir-collaboration.html
Gartner, “Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027” (2025) — https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
McKinsey, “Recalibrating CIO Technology Budgets for the AI Era” (2026) — https://www.mckinsey.com/capabilities/mckinsey-technology/our-insights/recalibrating-technology-budgets-for-the-ai-era
OpenTelemetry, “Semantic Conventions for Generative AI” (2024) — https://opentelemetry.io/docs/specs/semconv/gen-ai/
OWASP, “LLM01:2025 Prompt Injection” (2025) — https://genai.owasp.org/llmrisk/llm01-prompt-injection/
Palantir, “Safely Modernize Legacy Systems with Palantir Foundry Container Engine (FCE)” (2023) — https://blog.palantir.com/safely-modernize-legacy-systems-with-palantir-foundry-container-engine-fce-d8900464da7c
Shetty, R., “How Capital One Delivers Multi-Agent Systems,” TWiML AI Podcast #765 (2026) — https://twimlai.com/podcast/twimlai/how-capital-one-delivers-multi-agent-systems

AI Supply Chain Attacks: New Vectors in Dependencies

Hugues Clouâtre — Wed, 11 Feb 2026 20:45:00 GMT

A CI pipeline trusts 400 packages. Last week, one of them laid off 75% of its engineering team. Two years ago, another nearly shipped a backdoor to every major Linux distribution. Attackers are now registering package names that only exist because an AI hallucinated them.

Three incidents. Three attack vectors. One common thread: AI is reshaping software supply chain risk faster than most security programs can adapt.

Most organizations scan for CVEs and maintain an SBOM. Few monitor for maintainer burnout, AI-driven revenue collapse, or packages that only exist because an LLM invented them. The following framework closes those gaps.

Figure 1: Three AI-driven attack vectors targeting a software dependency chain.

Contents

What Are the Three Attack Vectors?
Why Does This Matter at Enterprise Scale?
How Does the Two-Tier Ecosystem Create Different Risks?
- Tier 1: Foundation-Backed Projects
- Tier 2: Indie and VC-Backed Projects
How Should Organizations Assess AI Exposure Risk?
How Does Vibe Coding Multiply Supply Chain Risk?
How Can Organizations Defend Against AI Supply Chain Attacks?
- Extend the Toolchain
- Sponsor Strategically
How Will Relicensing Reshape the Stack?
- Before the Next Sprint
References

What Are the Three Attack Vectors?

AI does not just accelerate existing supply chain risks. It creates new ones.

Vector 1: Maintainer Collapse (AI-Accelerated)

On January 6, 2026, Adam Wathan attributed Tailwind’s layoffs directly to AI’s brutal impact on their business. Documentation traffic dropped 40%. Revenue collapsed 80%. Yet Tailwind CSS downloads keep climbing.

The mechanism is simple: developers ask Copilot for a Tailwind grid layout. The AI generates it. No documentation visit. No discovery of Tailwind UI. No conversion. The developer gets value. The maintainer gets nothing.

This is not a failing product. It is a failing business model, and not unique to Tailwind. Any project monetizing through documentation traffic faces the same exposure.

Meanwhile, curl maintainer Daniel Stenberg scrapped the project’s bug bounty program on January 21, 2026, citing “intact mental health.” Twenty AI-generated vulnerability reports flooded HackerOne in January alone. None identified actual vulnerabilities. Researchers paste code into LLMs, submit the hallucinated analysis, then loop follow-up questions through the same models. The result: maintainers spend hours triaging garbage instead of shipping code.

Different mechanism, same outcome. Two forces compete: AI lowers the cost of writing software (beneficial for maintainers), but it also diverts users away from the direct engagement that funds maintainers (unsustainable for business models). Koren et al. (2026) model this formally and show the demand-diversion channel dominates. Stack Overflow activity has declined roughly 25% since ChatGPT’s launch, following the same pattern as Tailwind.

In March 2024, Microsoft engineer Andres Freund discovered a backdoor in XZ Utils days before it would have shipped to most Linux distributions. CVE-2024-3094 scored a perfect 10.0. The backdoor enabled remote code execution through SSH on affected systems.

The attack took two years. A contributor using the name “Jia Tan” gained the sole maintainer’s trust through legitimate contributions, then inserted malicious code. The maintainer was burned out, working alone, grateful for help.

This pattern repeats. The event-stream incident in 2018 followed the same playbook: abandoned maintainer transfers control, attacker inserts cryptocurrency-stealing code. XZ Utils proved the technique works at infrastructure scale.

Two years later, the technique has evolved. Attackers now use LLMs to maintain technically helpful, perfectly patient personas over months, bypassing the “vibe check” that once caught human bad actors.

Vector 3: Slopsquatting (AI-Native)

This attack vector did not exist before LLMs. Slopsquatting exploits models that confidently recommend nonexistent packages. One in five AI suggestions points to a package that was never published (Spracklen et al., 2025).

The attack flow:

Researchers run popular LLMs and collect hallucinated package names
Attackers register those names on npm, PyPI, or RubyGems with malicious payloads
Developers install AI-suggested packages without validation
Malicious code executes

Unlike typosquatting, attackers do not need to guess which names developers might mistype. The model identifies exactly which fake packages to register. Names like “aws-helper-sdk” and “fastapi-middleware” appear in AI-generated code but never existed until attackers registered them.

The defense is straightforward: verify that packages existed before the commit date. This check catches AI-hallucinated packages that attackers registered after the model suggested them:

Code Snippet 1: Detect slopsquatting by checking package age. Conceptual implementation only.

import requests
from datetime import datetime

def check_package_age(name: str, commit_date: str) -> bool:
    resp = requests.get(f"https://registry.npmjs.org/{name}")
    if resp.status_code == 404:
        return False  # Package does not exist
    pkg = resp.json()
    published = datetime.fromisoformat(pkg['time']['created'].replace('Z', '+00:00'))
    return published < datetime.fromisoformat(commit_date)scripts/validate_deps.py

Why Does This Matter at Enterprise Scale?

Sonatype’s 2024 State of the Software Supply Chain report documented 512,847 malicious packages discovered between November 2023 and November 2024. A 156% year-over-year increase. The Verizon 2025 DBIR found that 30% of breaches now involve third-party components, double the previous year.

Log4Shell proved how fast these risks materialize. According to Wiz and EY, 93% of enterprise cloud environments were affected. Four years later, 13% of Log4j downloads from Maven Central still contain the vulnerable version, roughly 40 million vulnerable downloads per year.

These vectors do not operate in isolation. They cascade. The same “software-begets-software” feedback loop that drove OSS growth now amplifies contraction: fewer maintainers produce fewer packages, which reduces ecosystem quality, which further weakens incentives to share. At 70% vibe coding adoption, engagement-based monetization drops roughly 70%, but OSS entry can only sustain an 11% decline before projects start disappearing. That 59-percentage-point gap is the crisis window.

Monitoring the right signals matters more than counting CVEs.

How Does the Two-Tier Ecosystem Create Different Risks?

Not all dependencies face equal exposure. A two-tier structure is emerging.

Tier 1: Foundation-Backed Projects

CNCF, Apache, and Linux Foundation projects operate differently. Kubernetes maintainers are typically employed by member companies. Corporate membership dues fund development. Governance structures distribute responsibility.

These projects face sustainability challenges: burnout, security maintenance burdens, contributor fatigue. Foundation backing buffers against documentation-traffic collapse, but not full immunity. Corporate membership dues often correlate with ecosystem health; if the broader ecosystem contracts, so does corporate willingness to fund.

Tier 2: Indie and VC-Backed Projects

Tailwind, Bun (pre-acquisition), curl, and thousands of smaller projects depend on sponsorships, consulting revenue, or VC runway. Many have single maintainers. The bus factor is often one.

These projects underpin the modern web. They are also most exposed to all three attack vectors.

Most enterprise stacks span both tiers. Kubernetes (Tier 1) might orchestrate containers running applications built with Tailwind (Tier 2). The risk profiles differ, and monitoring should reflect that.

How Should Organizations Assess AI Exposure Risk?

Traditional dependency scanning catches CVEs. It does not catch maintainer burnout, revenue collapse, or AI-hallucinated packages. Additional signals are needed.

The 5-Signal AI Exposure Audit

Funding model: Corporate-backed or sponsorship-dependent? Check GitHub Sponsors, Open Collective, or company backing. Sponsorship-dependent projects face higher AI exposure.
Contributor count: Bus factor greater than three? XZ Utils had one active contributor. Look at commit history and active contributors over the past 12 months.
Governance: Foundation membership or solo maintainer? CNCF and Apache projects have succession plans. Solo projects often do not.
AI exposure score: Docs-driven monetization (high exposure) or infrastructure utility (lower exposure)? UI libraries and developer tools face higher risk than compression utilities or parsers.
Recent signals: Layoffs, acquisition talks, burnout posts? Monitor project blogs, maintainer social media, and GitHub discussions.

Decision Thresholds

The following framework maps signals to response levels:

**Table 1:** Decision thresholds for dependency risk response.
Risk Level	Signals	Action
Monitor	1-2 signals, Tier 1 project	Quarterly review
Watch	2-3 signals, any tier	Monthly review, identify alternatives
Mitigate	3+ signals, Tier 2 project	Sponsor, fork, or migrate
Critical	Active incidents (layoffs, security events)	Immediate review, contingency plan

Enterprise Application

Consider a fintech platform with 400 npm dependencies. Traditional scanning surfaces CVEs. The AI exposure audit surfaces different risks:

**Table 2:** AI exposure audit across dependency categories.
Dependency Type	Count	High AI Exposure	Action
UI frameworks	12	4	Review monetization models
Build tools	8	2	Track bus factor, burnout signals
Infrastructure	45	1	Lower priority
Utility libraries	335	23	Automate monitoring

The 23 high-exposure utility libraries are not all equal. Prioritize by criticality: is it in the authentication path? The payment flow? The deployment pipeline?

How Does Vibe Coding Multiply Supply Chain Risk?

Andrej Karpathy coined “vibe coding” in February 2025: developers who “fully give in to the vibes” and let AI generate entire applications. The practice is accelerating, and it compounds every vector above.

Slopsquatting is the visible risk. The less visible one: license laundering. AI training data includes GPL and AGPL (copyleft) code. When developers accept AI-generated output without review, they risk shipping restrictive-licensed code as proprietary. The provenance is untraceable. Tools like FOSSA and Snyk License Compliance can scan for license violations in CI, but they catch dependencies, not AI-generated source code. Manual review remains essential.

Code Snippet 4: FOSSA scans dependencies for GPL/AGPL violations and fails the build on policy breach.

jobs:
  license-scan:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v6
      - name: Scan licenses with FOSSA
        uses: fossas/fossa-action@v1
        with:
          api-key: ${{ secrets.FOSSA_API_KEY }}
          run-tests: true.github/workflows/license-check.yml

AI-assisted commits warrant validation: flagging new dependencies, verifying packages existed before the commit date (Code Snippet 1), and checking maintainer history. For teams using AI agents extensively, subagent architectures can dedicate a validation agent to check every dependency before acceptance.

How Can Organizations Defend Against AI Supply Chain Attacks?

Extend the Toolchain

Existing tools catch CVEs. Tools that catch sustainability risks fill the gap:

OpenSSF Scorecard (scorecard.dev) scores projects on maintainer activity, security practices, and bus factor
deps.dev provides dependency graphs with contributor data
Socket.dev detects supply chain attacks including slopsquatting patterns

Scorecard and Socket integrate directly into AI-augmented CI/CD pipelines via GitHub Actions, flagging risky dependencies before merge.

For a quick CLI check:

Code Snippet 2: OpenSSF Scorecard CLI checks a repository’s security health score (0-10).

# Check a project's health score (0-10)
scorecard --repo=tailwindlabs/tailwindcss

To enforce this in CI, integrate Scorecard into a GitHub Actions workflow. The workflow below fails the build if a dependency scores below 7 out of 10, preventing risky dependencies from entering production:

Code Snippet 3: GitHub Actions workflow to enforce minimum dependency health score of 7/10 in CI.

- name: Run OpenSSF Scorecard
  uses: ossf/scorecard-action@v2
  with:
    results_file: scorecard.json

- name: Block on low score
  run: |
    jq -e '.score >= 7' scorecard.json || exit 1.github/workflows/scorecard.yml

Tooling catches symptoms. Sponsorship addresses the cause. If a Tier 2 project in a critical path shows stress signals, $500/month establishes a relationship with the maintainer and early warning on sustainability issues.

But individual sponsorship is a stopgap, not a solution. The systemic fix requires platform-level change. One proposed model: AI coding platforms already track which packages they import, so they could redistribute subscription revenue to maintainers based on attributed usage, a “Spotify for open source.” The infrastructure exists; the coordination does not. Until it does, direct sponsorship remains the best lever CTOs have.

The companies that sponsored Log4j before Log4Shell had maintainer relationships when the crisis hit. The ones that did not scrambled with everyone else.

How Will Relicensing Reshape the Stack?

The OSS ecosystem is adapting. Bun’s acquisition by Anthropic shows one path: AI companies absorbing critical infrastructure. More acquisitions are likely.

Experimental protocols like x402 attempt to let AI agents pay for resource access. Still early, but architecturally sound.

The two-tier ecosystem is crystallizing. Foundation-backed projects will weather it better. Indie projects will face consolidation pressure, through acquisition, abandonment, new business models, or relicensing. HashiCorp moved Terraform to BSL in 2023. Redis followed months later. Sentry, MariaDB, Elastic. The pattern is clear. When sponsorships fail and AI erodes documentation revenue, restrictive licenses become the survival strategy. For enterprises, this means dependencies assumed to be permissively licensed may not stay that way.

Before the Next Sprint

Run OpenSSF Scorecard on the top 10 dependencies
Check bus factor on Tier 2 critical-path projects
Add slopsquatting validation to CI (Code Snippet 1)

AI adoption is accelerating. So are the attacks exploiting it. The question is not whether a supply chain has risk, but whether teams that see it first have the advantage.

References

CrowdStrike, “CVE-2024-3094 and XZ Upstream Supply Chain Attack” (2024) - https://www.crowdstrike.com/en-us/blog/cve-2024-3094-xz-upstream-supply-chain-attack/
Goodin, Dan, “Overrun with AI slop, cURL scraps bug bounties” (Ars Technica, 2026) - https://arstechnica.com/security/2026/01/overrun-with-ai-slop-curl-scraps-bug-bounties-to-ensure-intact-mental-health/
Infosecurity Magazine, “Vibe Coding: A Hidden Security Risk of the AI Era” (2025) - https://www.infosecurity-magazine.com/opinions/vibe-coding-security-risk-ai/
Karpathy, Andrej, “vibe coding” (X, 2025) - https://x.com/karpathy/status/1886192184808149383
Koren, Miklos et al., “Vibe Coding Kills Open Source” (arXiv, 2026) - https://arxiv.org/abs/2601.15494
Snyk, “Slopsquatting: New AI Hallucination Threats” (2025) - https://snyk.io/articles/slopsquatting-mitigation-strategies/
Sonatype, “10th Annual State of the Software Supply Chain” (2024) - https://www.sonatype.com/state-of-the-software-supply-chain/introduction
Spracklen, Joseph et al., “We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs” (arXiv, 2025) - https://arxiv.org/abs/2406.10279
Sumner, Jarred, “Bun is joining Anthropic” (2025) - https://bun.sh/blog/bun-joins-anthropic
Verizon, “2025 Data Breach Investigations Report” - https://www.verizon.com/business/resources/reports/dbir/
Wathan, Adam, “GitHub comment on Tailwind layoffs” (2026) - https://github.com/tailwindlabs/tailwindcss.com/pull/2388#issuecomment-3717222957

Orchestrating AI Agents: A Subagent Architecture

Hugues Clouâtre — Wed, 17 Jun 2026 11:04:00 GMT

Single-agent AI coding hits a ceiling. Context windows fill up. Role confusion creeps in. Output quality degrades. The solution: multiple specialized agents handle distinct phases with structured handoffs.

Basic code assistants show roughly 10% productivity gains. But companies pairing AI with end-to-end process transformation report 25-30% improvements (Bain, 2025). The difference isn’t the model. It’s the architecture, specifically how you engineer the context each agent receives.

Anthropic’s research on multi-agent systems confirms what we observe: architecture matters more than model choice. Their finding that “token usage explains 80% of the variance” reflects the impact of isolation: focused context rather than accumulated conversation history.

This post documents a production workflow using Goose (Agentic AI Foundation). The same architecture runs in Claude Code via a skill file. Specialized agents, each optimized for its phase: research, planning, building, and validation.

Contents

Why Do Single-Agent AI Coding Workflows Hit a Ceiling?
How Does Subagent Architecture Solve Context Problems?
How Does Model Selection Affect Cost and Quality?
How Do Subagents Communicate?
Where Should Human Judgment Stay in AI Workflows?
What Results Does This Produce?
- Design Targets
When Does This Work (and When Doesn’t It)?
Takeaways
References

Why Do Single-Agent AI Coding Workflows Hit a Ceiling?

A single AI model handling an entire coding task accumulates context with every interaction. By implementation time, the model carries baggage from analysis, research, and planning phases. This stems from three core problems.

Context Rot

Long conversations consume token budgets. The model forgets early instructions or weighs recent context too heavily. Chroma Research calls this context rot: performance degrades consistently as input tokens increase, even on simple tasks, and worsens for multi-step reasoning like coding (Chroma Research, 2025). On-demand retrieval adds another failure mode: agents miss context 56% of the time because they don’t recognize when to fetch it (Gao, 2026).

Role Confusion

A model asked to analyze, plan, implement, and validate lacks clear boundaries. It starts implementing during planning. It skips validation steps. Outputs blur together.

Accumulated Errors

Mistakes in early phases propagate. A misunderstanding in analysis leads to a flawed plan. A flawed plan leads to incorrect implementation. Fixing requires starting over.

How Does Subagent Architecture Solve Context Problems?

The fix: spawn specialized subagents for each phase. An orchestrator handles high-level coordination and synthesis. Subagents handle execution with fresh context.

Figure 1: Core subagent workflow. SCOUT and GUARD handle RESEARCH sequentially; ORCHESTRATOR synthesizes both into a PLAN. Each agent starts with fresh context, isolated from prior phases. SETUP and COMMIT/PR phases omitted for clarity.

The orchestrator (Claude Sonnet 4.6) handles PLAN only. RESEARCH is split between a SCOUT agent (Haiku, exploratory) and a GUARD agent (Haiku, adversarial) that stress-tests SCOUT’s proposals before the orchestrator synthesizes both into a plan. SCOUT operates on structured codebase inputs (file trees, function indexes, call graphs), where output volume matters more than reasoning range. GUARD’s task is lower-entropy still: finding flaws in a concrete proposal rather than producing one. Both fit Haiku’s profile. After plan completion, the orchestrator spawns a BUILD subagent (Claude Sonnet 4.6) that receives only the plan, not accumulated history. The builder writes code, runs tests, then hands off to a CHECK subagent (Haiku) for validation.

Each subagent starts with clean context. The builder knows what to build, not how we decided to build it. The validator knows what was built, not what alternatives we considered. This is context engineering in practice: deciding what information enters each context window, in what form, and when, distinct from prompt engineering, which only governs phrasing. Smaller, structured inputs produce deterministic behavior: the same plan JSON produces the same output. That reproducibility is what drives a high PR acceptance rate.

How Does Model Selection Affect Cost and Quality?

Different phases need different capabilities. Planning requires reasoning. Building requires speed and instruction-following. Validation requires balanced judgment.

**Table 1:** Model selection by agent. Sonnet handles open-ended reasoning; Haiku handles constrained execution. Temperature reflects task openness: higher for exploration, lower for verification.
Agent	Model	Temperature	Role
ORCHESTRATOR	Sonnet	0.3	Plan and agent coordination
SCOUT	Haiku	0.5	Exploratory research and code analysis
GUARD	Haiku	0.1	Adversarial risk review
BUILD	Sonnet	0.2	Precise instruction-following coding
CHECK	Haiku	0.1	Compliance and security validation

How Does Model Routing Reduce Cost?

Building involves the most token-heavy work: reading files, writing code, running tests. Routing this volume to cheaper models cuts costs significantly.

**Table 2:** Anthropic API pricing, March 2026. Standard rates (prompts under 200K tokens). Three agents on Haiku, two on Sonnet.
Model	Input	Output	Agent
Opus	$5/MTok	$25/MTok	Not used in default flow
Sonnet	$3/MTok	$15/MTok	ORCHESTRATOR + BUILD
Haiku	$1/MTok	$5/MTok	SCOUT + GUARD + CHECK

Research on multi-agent LLM systems shows up to 94% cost reduction through model cascading (Gandhi et al., 2025). This architecture targets 50-60% savings: three of five agents run on Haiku ($1/$5 per MTok), two on Sonnet ($3/$15 per MTok). SCOUT, GUARD, and CHECK run on Haiku and together account for 61% of pipeline sessions at one-third the per-token cost of Sonnet. BUILD runs on Sonnet despite being the single highest-volume phase by tokens (39% of pipeline total): the retry reduction from 77% to 14% offsets the model premium. Versus Opus, the saving is 5x.

Smaller models with focused context outperform a single large model carrying full session history. Anthropic’s multi-agent research showed a 90.2% performance gain over single-agent Opus by distributing work across Sonnet subagents with isolated context windows (Anthropic Engineering, 2025). Cost drops; quality improves.

Beyond cost, fresh context enables tasks that fail with single agents. A 12-file refactor that exhausts a single model’s context window succeeds when each subagent starts clean.

Which Alternative Models Pass Quality Gates?

Each run was scored on an 8-point binary rubric (file identification, dependency verification, architectural tradeoff, pattern analysis, taint-tracking gap, non-obvious synthesis, JSON schema compliance). QP is a composite efficiency metric: cost per run divided by score times reliability. Our model-comparison experiments tested eight candidates as SCOUT delegates: only MiniMax M2.5 ($0.019/QP) and Kimi K2.5 ($0.040/QP) passed against a Haiku baseline of $0.150/QP. Mistral Small 4 ($0.002/QP) narrowly failed on a floor score violation; DeepSeek V3.2 failed all gates; Mercury-2, the cheapest candidate at $0.001/QP, failed on score (Clouatre, 2026).

How Does Project Context Reach Subagents?

Recipes define the workflow, but subagents also need project context: build commands, conventions, file structure. That’s where AGENTS.md comes in, a portable markdown file that provides the baseline knowledge every subagent inherits. Goose, Cursor, Codex, and 40+ other tools read it natively. In Vercel’s evals (Gao, 2026), an AGENTS.md file achieved a 100% pass rate on build, lint, and test tasks where skills-based approaches maxed out at 79%.

Think of it as CSS for agents: global rules cascade into every project, project-specific rules override where needed. The orchestrator and every subagent it spawns inherit both layers without explicit prompting.

Code Snippet 1: Global and project-level AGENTS.md files. The builder subagent from Table 1 inherits both layers: it knows to GPG-sign commits (global) and use cargo-deny (project) without either appearing in the handoff JSON.

## Commits
- Conventional commits, GPG signed and DCO sign-off
- Feature branches only, PRs for everything
- Never merge without explicit user request

## Security
- Treat all repositories as public
- No secrets, API keys, credentials, or PII~/.config/goose/AGENTS.md

## Stack
Rust 2024 + Tokio + Clap (derive) + Octocrab

## Project-Specific Patterns
- Apache-2.0 license with SPDX headers
- cargo-deny for dependency auditsaptu/AGENTS.md

How Do Subagents Communicate?

Subagents communicate through JSON files in $WORKTREE/.handoff/. Each session uses an isolated git worktree, so handoff files are scoped to that execution context. This creates an explicit contract between phases.

**Table 5:** Handoff file chain. Each file is scoped to one phase transition.
File	From	To
`01a-research-scout.json`	SCOUT	GUARD
`01b-research-guard.json`	GUARD	ORCHESTRATOR
`02-plan.json`	ORCHESTRATOR	BUILD
`03-build.json`	BUILD	CHECK
`04-validation.json`	CHECK	BUILD (on failure)

GUARD confirms SCOUT’s file list, scores each approach by safety, and flags risks SCOUT didn’t surface. The orchestrator synthesizes both into a plan only after GUARD clears it.

Code Snippet 3: GUARD handoff file. Three things SCOUT didn’t produce: a risk with concrete mitigation, safety scores with rationale, and a test gap.

{
  "scout_verification": {"accurate": true, "corrections": []},
  "risks": [
    {
      "function": "try_with_fallback",
      "risk": "Async extraction requires careful borrowing",
      "severity": "medium",
      "mitigation": "Extract only sync setup; keep async operation in caller"
    }
  ],
  "safety_ranking": [
    {"id": "A", "score": 95, "rationale": "Minimal scope, aligns with existing patterns"},
    {"id": "B", "score": 70, "rationale": "Struct over-engineered for current scope"},
    {"id": "C", "score": 55, "rationale": "try_fold() pattern; highest maintenance risk"}
  ],
  "guard_test_gaps": [
    "try_with_fallback: add test for fallback chain with real failures"
  ]
}.handoff/01b-research-guard.json

The plan file contains everything the builder needs:

Code Snippet 4: Plan handoff file with structured task definition for the builder subagent.

{
  "overview": "Remove 4 dead render_with_context methods",
  "files": [
    {"path": "src/output/triage.rs", "action": "modify"},
    {"path": "src/output/history.rs", "action": "modify"},
    {"path": "src/output/bulk.rs", "action": "modify"},
    {"path": "src/output/create.rs", "action": "modify"}
  ],
  "steps": [
    "Remove render_with_context impl blocks from each file",
    "Remove #[allow(dead_code)] annotations",
    "Remove unused imports",
    "Run cargo fmt && cargo clippy && cargo test"
  ],
  "risks": ["None - confirmed dead code"]
}.handoff/02-plan.json

The validator reads both 02-plan.json and 03-build.json to verify implementation matches requirements. It writes structured feedback to 04-validation.json:

Code Snippet 5: Validation handoff file with actionable feedback for the builder to address.

{
  "verdict": "FAIL",
  "checks": [
    {"name": "Remove #[allow(dead_code)] annotations", "status": "FAIL",
     "notes": "Annotations still present in history.rs:145, bulk.rs:31, create.rs:63"}
  ],
  "issues": ["Plan required removing annotations, but these are still present"],
  "next_steps": "Fix issue: Remove the three annotations, then re-validate"
}.handoff/04-validation.json

The builder reads this feedback, fixes the specific issues, and triggers another CHECK cycle until validation passes.

Why files instead of memory? Three reasons:

Auditable. Every decision is recorded. Debug failures by reading the handoff chain.
Resumable. Handoff files checkpoint state. Resume any session from where it stopped.
Debuggable. Failed validations include exact locations and actionable next steps.

Both Goose and Claude Code resume sessions natively, but they restore full conversation history, reintroducing the context rot this architecture avoids. Resuming here means picking up the plan and build results with fresh agent context, not replaying an accumulated chat.

Figure 2: BUILD-CHECK feedback loop. Validation verdict routes to PR-ready state on PASS, user gate on PASS WITH NOTES, or retry loop on FAIL with actionable next steps.

Where Should Human Judgment Stay in AI Workflows?

The quality of the input determines the ceiling for everything that follows. A well-scoped GitHub issue with clear acceptance criteria and explicit constraints, written with AI assistance but reviewed by a human, is the best prompt you can give this workflow. Imprecise requirements produce technically correct but misaligned results. A high-quality issue or spec is the single biggest lever for PR acceptance rate.

The workflow runs autonomously: RESEARCH, PLAN, BUILD, and CHECK all auto-proceed. The one recommended human touchpoint at the end is the PR review itself. The code has your name on it.

Phases (all auto-proceed):

SETUP: Initialize context and gather requirements
RESEARCH: SCOUT and GUARD run sequentially; the orchestrator synthesizes their findings autonomously
PLAN: Design solution based on research synthesis
BUILD: Execute the approved plan
CHECK: Validate and loop back to BUILD on failure, then push to PR

What Results Does This Produce?

This architecture powers development across multiple projects. Three examples from aptu:

**Table 3:** Representative PRs using subagent architecture. All passed CI, all merged without rework.
PR	Scope	Files Changed
#272	Consolidate 4 clients → 1 generic	9 files
#256	Add Groq + Cerebras providers	9 files
#244	Extract shared AiProvider trait	9 files

The validation phase caught issues the builder missed. In PR #272, the CHECK subagent identified a missing trait bound that would have failed compilation. The builder fixed it on the retry loop. No human intervention required.

The CHECK phase consistently catches issues before they reach a PR: wrong serde attributes, uncommitted builds, scope creep, CLI interface drift, and formatting violations. The workflow now operates at production scale across dozens of repositories, with thousands of pull requests merged. Representative public benchmarks: aptu reached PR #1,325 and aptu-coder reached PR #1,038, both built entirely with this pipeline.

Design Targets

Enterprise platforms reach the same conclusions at scale. Sid Pardeshi, CTO of Blitzy, described on the TWiML AI podcast (2026) a system that writes millions of lines of code autonomously. His diagnosis of the root constraint is identical: the effective context window has been stuck at 80-120K tokens for two years, regardless of headline sizes. Their solution replaces the single orchestrator with a database as the coordination layer, enabling tens of thousands of parallel agents. The architectural principle is the same one this workflow applies: isolate context, specialize roles, and pass structured state between agents rather than accumulating a single conversation.

Research on multi-agent frameworks for code generation shows they consistently outperform single-model systems (Raghavan & Mallick, 2025).

**Table 4:** Isolated context, specialized roles, structured state over accumulated conversation.
Metric	Single-Agent	Multi-Agent
Output quality over session	Degrades as context fills	Stable (each agent starts fresh)
Model strategy	Generic model	Specialized per role
Estimated cost reduction	Baseline	50-60% via model routing
Human interventions	Throughout	Issue scoping + PR review
Audit trail	Conversation history	Structured JSON handoff chain

When Does This Work (and When Doesn’t It)?

Works well for:

Multi-file refactors where context isolation prevents confusion
Feature additions following established patterns
Complex changes requiring distinct planning and execution
Teams wanting audit trails (handoff files document decisions)

Less effective for:

Simple one-file fixes (overhead exceeds benefit)
Legacy systems without clear patterns (builder lacks context)
Exploratory work where plans change during implementation

See AI agents in legacy environments for integration patterns that work when your data lives in mainframes and AS400 systems.

Takeaways

Separate reasoning from execution. Use capable models for planning, fast models for building.
Fresh context beats accumulated context. Subagents start clean. They follow instructions without historical baggage.
Structured handoffs create audit trails. JSON files document what was planned, built, and validated.
Quality input, autonomous execution. A well-scoped issue is the highest-leverage human contribution.

The full workflow is available on GitHub Gist as both a Goose recipe (YAML) and a Claude Code skill (Markdown). It builds on patterns from AI-Assisted Development: The Accountability Layer.

The agentic architecture described in this post is the subject of Canadian patent application CA 3315347, filed Jun 16, 2026 (CIPO).

References

Anthropic Engineering, “How we built our multi-agent research system” (2025) — https://www.anthropic.com/engineering/multi-agent-research-system
Bain & Company, “From Pilots to Payoff: Generative AI in Software Development” (2025) — https://www.bain.com/insights/from-pilots-to-payoff-generative-ai-in-software-development-technology-report-2025/
Chroma Research, “Context Rot: How Increasing Input Tokens Impacts LLM Performance” (2025) — https://research.trychroma.com/context-rot
Gao, Jude, “AGENTS.md outperforms skills in our agent evals” (2026) — https://vercel.com/blog/agents-md-outperforms-skills-in-our-agent-evals
Clouatre, H., “LLM Agent Experiments: Model Comparison for SCOUT Delegates” (2026) — https://doi.org/10.5281/zenodo.19056876
Gandhi et al., “BudgetMLAgent: A Cost-Effective LLM Multi-Agent System” (2025) — https://arxiv.org/abs/2411.07464
Pardeshi, Sid, “Agent Swarms, Knowledge Graphs, and Autonomous Software Development” (2026) — https://twimlai.com/podcast/twimlai/agent-swarms-knowledge-graphs-autonomous-software-development
Raghavan & Mallick, “MOSAIC: Multi-agent Orchestration for Task-Intelligent Scientific Coding” (2025) — https://arxiv.org/abs/2510.08804

AI-Augmented CI/CD: Shift Left Security Without Risk

Hugues Clouâtre — Mon, 30 Mar 2026 16:16:00 GMT

Code reviews are a bottleneck. Engineering teams lose measurable velocity waiting for feedback. This delay compounds when security vulnerabilities escalate: defects caught late cost orders of magnitude more to fix than those caught at design time.

AI in CI/CD augments human review by analyzing code patterns and tool outputs before human reviewers see the changes.

Contents

What Is the Real Cost of Manual Code Review?
How Does AI Integration Work Without Prompt Injection Risk?
How Do Tier 2 and Tier 3 Balance Speed Against Security?
How Does Uncontrolled AI Analysis Become a Security Risk?
What Outcomes Does AI-Augmented CI/CD Deliver?
How Do You Get Started With AI-Augmented CI/CD?
References

What Is the Real Cost of Manual Code Review?

The Review Bottleneck

Development velocity correlates with code review latency. Code review bottlenecks are well-documented across engineering teams. Feedback loops stretch from hours to days while developers context-switch or wait on reviewers. Research from Forsgren et al. (2024) shows context-switching during code review significantly reduces developer productivity and satisfaction.

GitHub’s 2024 Octoverse reports median time from PR open to first review is 4 hours in large organizations, 22 hours in enterprises.

Traditional CI/CD pipelines run automated linters and security scanners, generate reports, then stop. A human reads the output, interprets it, decides if it matters, and either approves or comments. This handoff creates velocity bottlenecks. Eight-hour review windows delay production deployments. Critical insights get buried in noise. Studies confirm developers fear review delays will slow delivery, even though they recognize reviews’ long-term quality benefits (Santos et al., 2024). The cost of this wait scales with engineer compensation.

The Security Cost Multiplier

Security defects amplify this cost multiplier. Boehm & Basili (2001) document that the cost multiplier is phase-dependent, rising from single-digit factors at design time to two or more orders of magnitude at production; Tassey (2002) corroborates these findings at the systems level. The expenses compound: rework costs, deployment delays, and potential security incidents each add to the total as the defect progresses through the pipeline.

Shift-left automation detects issues before a PR merges, before human review begins. AI analyzes linter output, security scan results, and code patterns in seconds. Developers receive immediate feedback, iterate faster, and ship with higher confidence.

The Prompt Injection Risk

Raw AI analysis of code diffs introduces a critical vulnerability: prompt injection. If a CI/CD pipeline feeds user-submitted code directly to an AI model, an attacker can craft a PR with embedded instructions that manipulate the AI’s behavior. The AI might approve malicious code, disable security checks, or expose sensitive information. This is not theoretical. It represents a live attack surface in every AI-augmented system.

Defensive architecture mitigates this risk. The AI analyzes tool output (structured, deterministic results from linters, security scanners, and static analysis) rather than untrusted input directly. The pipeline sequence: linter runs first, generates JSON, AI summarizes the findings, human approves. This removes the direct injection vector, though structured output can still carry adversarially crafted content via file paths or error message text (OWASP, 2025).

Threat models vary by repository type. A private repository with a trusted five-person team tolerates different risk than open-source projects accepting external contributors. Three security tiers match different threat models while maintaining analysis speed.

How Does AI Integration Work Without Prompt Injection Risk?

Tier 1 eliminates prompt injection risk. The linter runs first, produces JSON output, and the AI analyzes only that structured data. The AI never sees the raw code, never processes user input, and never runs in the context of potentially malicious diffs. Using an independent review instance, one without prior context from code generation, is the recommended architectural pattern; a session that wrote the code carries implicit context that can suppress contradictory findings during self-review.

Code Snippet 1: In Tier 1, AI analyzes only JSON output from the linter, never raw code. Full example

name: AI Analysis - Maximum Security
on: [pull_request]

permissions:
  contents: read

jobs:
  analyze:
    runs-on: ubuntu-24.04
    steps:
      - name: Checkout
        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
      
      - name: Lint Code
        run: pipx run ruff check --output-format=json . > lint.json || exit 0
      
      - name: Setup Goose
        uses: clouatre-labs/setup-goose-action@35f35c3a8f08aa333486693114938ec643bf8310  # v1.0.7
      
      - name: AI Analysis
        env:
          GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
        run: |
          echo "Summarize these linting issues:" > prompt.txt
          cat lint.json >> prompt.txt
          # Only structured tool output appended. Never raw source code.
          goose run --instructions prompt.txt --no-session --quiet > analysis.md
      
      - name: Upload Analysis
        uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f  # v7.0.0
        with:
          name: ai-analysis
          path: analysis.mdtier1-maximum-security.yml

Pinning actions to a commit SHA rather than a floating tag is a supply chain control, not a style preference. In March 2026, the Trivy ecosystem supply chain was briefly compromised (CVE-2026-33634): an attacker force-pushed 76 version tags in aquasecurity/trivy-action to point to credential-stealing code, and every workflow using a floating tag silently executed the malicious version. A SHA reference is immutable, so tag mutation has no effect. The maintenance burden is minimal: Renovate and Dependabot both open automated PRs to bump pinned SHAs when upstream releases a new version, so staying current requires only a one-click merge.

The AI sees only JSON. No code, no comments, no user input. Attack surface is minimal, but file paths and error strings in JSON output remain a residual indirect-injection vector (OWASP, 2025). This pattern applies to public repositories, open-source projects, and any system where external contributors submit PRs.

Figure 1: Tier 1 defensive pattern. AI analyzes tool output, never sees raw code. Immune to prompt injection.

How Do Tier 2 and Tier 3 Balance Speed Against Security?

Tier 2 provides additional context (file paths, change stats, commit metadata) without exposing raw code. This represents a middle ground: more insight than Tier 1, lower risk than Tier 3.

Code Snippet 2: In Tier 2, AI sees file scope and metadata, but not code diffs. Full example

name: AI Analysis - Balanced Security
on: [pull_request]

permissions:
  contents: read

jobs:
  analyze:
    runs-on: ubuntu-24.04
    steps:
      - name: Checkout
        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
      
      - name: Get Changed Files
        id: files
        run: |
          git diff --name-only origin/main...HEAD > files.txt
          wc -l files.txt >> summary.txt
      
      - name: Setup Goose
        uses: clouatre-labs/setup-goose-action@35f35c3a8f08aa333486693114938ec643bf8310  # v1.0.7
      
      - name: AI Analysis
        env:
          GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
        run: |
          echo "Review these file changes:" > prompt.txt
          cat files.txt summary.txt >> prompt.txt
          # File names and stats. Not the actual code content.
          goose run --instructions prompt.txt --no-session --quiet > analysis.md
      
      - name: Upload Analysis
        uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f  # v7.0.0
        with:
          name: ai-analysis
          path: analysis.mdtier2-balanced-security.yml

The AI sees file-level patterns but not line-by-line changes. Injection risk is low but non-zero: an attacker could craft filenames or commit messages to manipulate analysis. This tier applies to private repositories with trusted contributors.

Tier 3 applies to small, trusted teams where analysis speed outweighs defense-in-depth requirements. The AI sees full code diffs. Injection risk exists but is controlled through human approval gates.

Code Snippet 3: In Tier 3, AI sees full diffs for subtle patterns. Full example

name: AI Analysis - Advanced Patterns
on: [pull_request]

permissions:
  contents: read

jobs:
  analyze:
    runs-on: ubuntu-24.04
    steps:
      - name: Checkout
        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
      
      - name: Get Full Diff
        run: git diff origin/main...HEAD > changes.diff
      
      - name: Setup Goose
        uses: clouatre-labs/setup-goose-action@35f35c3a8f08aa333486693114938ec643bf8310  # v1.0.7
      
      - name: AI Analysis
        env:
          GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
        run: |
          echo "Deeply analyze these code changes:" > prompt.txt
          cat changes.diff >> prompt.txt
          # Complete code diffs for maximum context and detail
          goose run --instructions prompt.txt --no-session --quiet > analysis.md
      
      - name: Upload Analysis
        uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f  # v7.0.0
        with:
          name: ai-analysis
          path: analysis.mdtier3-advanced-patterns.yml

Each tier trades visibility for security. Tier 1 eliminates injection risk by sacrificing some context. Tier 2 accepts low risk for moderate context. Tier 3 prioritizes insight over security and is typically used sparingly.

Tier selection depends on three factors:

Repository access model (external contributors vs internal team)
Required AI context (tool output vs full diffs)
Risk tolerance (injection risk vs deeper analysis)

Figure 2: Three security tiers. Selection depends on threat model and team trust level.

**Table 1:** Tier comparison: security, risk, and recommended use.
Tier	Input	Injection Risk	Approval Gate	Recommended For
1	Tool output (JSON)	None	Human reviews artifact	Public repos, OSS, any external contributors
2	File stats + metadata	Low	Human pre-approval	Private repos, internal teams
3	Full code diff	Controlled	Optional	Tiny trusted teams only

The decision framework is simple: start at Tier 1. Measure deployment velocity, security posture, and developer satisfaction. Only move to Tier 2 or 3 if team consensus is that the additional AI context outweighs the injection risk. Most teams never need to leave Tier 1.

How Does Uncontrolled AI Analysis Become a Security Risk?

The naive approach feeds AI the code diff directly and allows it to comment on the PR. This is fast, appears intelligent, and creates an injection surface. The improved approach layers security tiers on top, providing a decision framework that matches the threat model.

Figure 3: Evolution from uncontrolled AI analysis to risk-managed tiers.

The shift is architectural, not just operational. The evolution moves from “AI sees everything and decides” to “AI sees what’s safe and humans decide what matters.” This distinction enables both security and speed improvements.

**Table 2:** Architectural comparison: uncontrolled vs. managed AI analysis.
Dimension	Uncontrolled AI	Managed AI (Tiered)
Input to AI	Full code diffs, commit messages, PR context	Structured tool output (Tier 1) or metadata (Tier 2)
Decision Authority	AI suggests, human reviews comments	AI analyzes, human approves before action
Injection Surface	High (user-submitted code in prompt)	Minimal to none (tool output only in Tier 1)
Applicable Context	Small trusted teams only	All team sizes and trust models

What Outcomes Does AI-Augmented CI/CD Deliver?

Human first-review latency runs 4–22 hours in large organizations (Graphite, “State of Code Review 2024”); in our implementation, AI analysis completes in 2–7 seconds, a reduction of three to four orders of magnitude compared to that baseline. Developers iterate faster because they receive feedback immediately. CI/CD pipelines do not stall waiting for human review availability.

AI flags issues during windows when human attention degrades: late-night reviews, context-switching mid-sprint. Linting issues get flagged automatically. Security tool outputs get analyzed for severity and context. Fewer critical issues reach production because they are caught earlier in the workflow.

For broader observability patterns in AI agent workflows, including legacy system integration, see AI agents in legacy systems.

Developer satisfaction increases when velocity and quality both improve. Engineers are not blocked by the review process. They receive comprehensive feedback without waiting. They trust the pipeline because it combines deterministic tools with AI insight and human judgment.

The expected directional outcomes, higher deployment frequency, lower mean time to resolution, fewer security incidents, follow directly from the mechanisms described above. Baseline measurement before integration is the only reliable way to confirm these trends in a given environment.

How Do You Get Started With AI-Augmented CI/CD?

Start With Tier 1

Start with Tier 1. It provides maximum security with zero prompt injection risk. The example workflow demonstrates the complete pattern. As of v1.0.7, the action verifies the Goose binary against its SLSA provenance attestation before installation, and the project holds an OpenSSF Best Practices Silver badge, a level reached by fewer than 1% of open-source projects. For AWS-native environments, setup-kiro-action offers SIGV4 authentication without API keys in secrets.

Tier selection depends on threat model. External contributors and public repositories warrant Tier 1. Internal teams with trusted code may benefit from Tier 2 or Tier 3 context. The key is matching exposure level to trust level.

Measure Before You Integrate

Baseline measurement establishes the starting point: current review latency, deployment frequency, and security incident rate. A two-week measurement period provides sufficient data for comparison. After AI integration, the same metrics reveal impact.

The human gate remains essential throughout. AI generates artifacts for review, not merge approvals. Engineers validate recommendations before acting. This preserves accountability while accelerating feedback cycles.

Tune for Signal Quality

Explicit review criteria improve signal quality. Define which issue categories the AI should report (bugs, security vulnerabilities, API misuse) and which to skip (minor style preferences or project-local conventions). Vague instructions like “be thorough” produce high false-positive rates that erode developer trust across all finding categories.

On re-runs after new commits, pass prior findings in context and instruct the AI to report only new or still-unaddressed issues. This prevents duplicate comments from accumulating on long-lived PRs. For large PRs spanning many files, split the review into a per-file local analysis pass followed by a separate cross-file integration pass. Reviewing everything in a single prompt overloads context and produces contradictory findings (Anthropic, “Best Practices for Claude Code”, 2026).

Give the AI Project Context

An AGENTS.md file at the repository root is the idiomatic mechanism for providing project-level context (testing standards, review criteria, fixture conventions) to CI-invoked AI without modifying prompts per workflow. Whether those gains materialise at the expected magnitude depends on how precisely the review criteria and context files are configured. The infrastructure exists; the constraint is configuration discipline.

For observability patterns in AI agent workflows, see AI Observability Gaps.

References

Anthropic, “Best Practices for Claude Code” (2026) - https://code.claude.com/docs/en/best-practices
Boehm & Basili, “Software Defect Reduction Top 10 List” (2001) - https://www.cs.umd.edu/projects/SoftEng/ESEG/papers/82.78.pdf
Forsgren et al., “DevEx in Action: A study of its tangible impacts” (2024) - https://dl.acm.org/doi/10.1145/3639443
Graphite, “State of Code Review 2024” - https://static.graphite.dev/Graphite_State_of_code_review_2024.pdf
GitHub Advisory Database, “Trivy ecosystem supply chain was briefly compromised” CVE-2026-33634 (2026) - https://github.com/advisories/GHSA-69fq-xp46-6x23
OWASP LLM Top 10 (2025 edition), Prompt Injection LLM01 - https://genai.owasp.org/llmrisk/llm01-prompt-injection/
Santos et al., “Modern code review in practice: A developer-centric study” (2024) - https://www.sciencedirect.com/science/article/pii/S0164121224003327
Tassey, G., “The Economic Impacts of Inadequate Infrastructure for Software Testing,” NIST (2002) - https://www.nist.gov/system/files/documents/director/planning/report02-3.pdf

AI-Assisted Development: The Accountability Layer

Hugues Clouâtre — Sat, 30 May 2026 07:34:00 GMT

A 2025 analysis found security flaws in 45% of AI-generated code outputs when context was underspecified (Zigler, 2025). AI accelerated the part of development that was never the bottleneck.

Implementation was already fast relative to debugging, architecture, and review. AI compressed it further. The slow parts, the ones requiring genuine domain understanding, got harder when code volume grew without a corresponding rise in qualified human attention. The result is not a productivity gain but a redistribution that benefits teams with strong review discipline and harms those without it.

The structural response is not better prompting. It is a governance layer that enforces human decision points where judgment matters: at authoring, at commit, and at ship. This post describes that layer in concrete terms.

Contents

Why Does AI-Assisted Development Create an Accountability Gap?
- The Volume Problem
- The Synthesis Degradation Finding
How Does AI Shift Time From Implementation to Judgment?
How Do Structured Workflows Enforce Human Decision Points?
- The Five-Phase Recipe
- Hard Blocks vs. Guidelines
How Do You Maintain Accountability When AI Writes the Code?
What Do Measured Time Savings Actually Look Like?
- CI Modernization (PR #52)
- Matrix Operations Feature (PR #109)
When Does This Approach Work — and When Does It Not?
- Security-Sensitive Tasks
- The Candidate Generation Model
What Does This Mean for Technical Leaders?
- The Perception Gap Risk
- Three Conditions for Success
References

Why Does AI-Assisted Development Create an Accountability Gap?

Software developers spend roughly 11% of their time coding; the rest is distributed across debugging, architecture, reviews, meetings, and operational tasks (Kumar et al., 2025). Coding was never the binding constraint.

The Volume Problem

When AI tools eliminate implementation friction, code volume increases. More pull requests, more changed lines, more design decisions embedded in generated output. But the human capacity for careful review does not scale with code volume. A reviewer evaluating 200 lines of handwritten code may face 800 lines of AI-generated output covering the same feature, with subtler assumptions embedded in the structure.

The Synthesis Degradation Finding

CHI 2025 research on AI-assisted development found that synthesis tasks, the core of architecture and design, show the steepest degradation under cognitive offloading (Shukla, Bui, Parsons et al., 2025). Reviewers who routinely approve AI output without deep engagement gradually lose the ability to catch what they once would have caught.

The accountability gap is structural: code is now generated faster than human judgment can validate it. Without explicit controls, the natural equilibrium is faster shipping with lower comprehension per merged line.

How Does AI Shift Time From Implementation to Judgment?

The Controlled Trial Baseline

The original case for AI coding assistance rested on controlled experiments. A randomized controlled trial with 95 developers found task completion approximately 55.8% faster with AI assistance (Peng et al., 2023). That result held for greenfield, well-specified programming tasks. Later research complicates it.

Where the Research Diverges

METR’s 2025 study found that experienced developers working on real tasks with AI tools took 19% longer while perceiving a 20% speedup (Becker et al., 2025). The gap between perceived and measured performance is itself informative: the feeling of productivity increased while actual throughput declined. That is the metacognitive failure mode that governance controls are designed to interrupt.

**Table 1:** Time allocation shift under AI-assisted development. Implementation shrinks; review and judgment must grow to hold quality constant.
Activity	Traditional	AI-Assisted	Direction
Implementation	High (bottleneck)	Low (AI-generated)	Shrinks
Debugging own code	High	Lower (less handwritten code)	Shrinks
Review and judgment	Low (limited by implementation time)	Must grow to match volume	Must grow
Architecture and design	Fragmented	Consolidated	Grows
Cognitive synthesis	Frequent, shallow	Fewer, deeper sessions required	Deepens

Metacognition as the Hidden Variable

Prather et al. (2024) found that metacognitive skill, not experience level, determines whether a developer benefits or is harmed by AI assistance. The skill in question is the ability to accurately assess what you understand and what you do not. Developers with strong metacognition catch AI errors because they notice when generated code does something they cannot explain. Developers without it ship the errors because the code looked plausible.

Karpathy’s distinction between vibe coding and agentic engineering (Karpathy, 2025) maps directly onto this: vibe coding raises the floor for simple tasks and lowers the ceiling for complex ones. Agentic engineering, with explicit spec design, diff review, eval design, security oversight, and quality judgment, preserves the ceiling. The difference is whether human judgment is structurally required or merely available.

How Do Structured Workflows Enforce Human Decision Points?

The recipe workflow used in this practice formalizes the judgment gates that distinguish agentic engineering from vibe coding. A recipe is a YAML workflow definition that codifies process: AI handles analysis, research, and implementation; the human approves direction at mandatory STOP points before any code is written.

The Five-Phase Recipe

Each phase requires explicit human approval before the next begins:

ANALYZE: Understand the codebase, identify problem scope and affected files
RESEARCH: Explore 2-3 solution approaches with trade-offs; human selects direction
PLAN: Detailed implementation plan reviewed before any code is written
IMPLEMENT: Code, tests, documentation; AI executes against approved plan
PREPARE: PR creation, branch verification, push; human approves before opening

Code Snippet 1: GATE pattern from production recipe. AI presents constrained options; human selects direction before any code is written.

## Phase 1: RESEARCH

Understand scope and constraints:
- Read issue/PR description, linked discussions
- Identify affected files with `rg` and `analyze`
- Note CI requirements, test patterns, coding standards

### GATE: Research Summary  

**STOP - Present to user:**
- Problem statement (1-2 sentences)
- Affected files and scope
- Constraints discovered (CI, tests, dependencies)
- 2-3 possible approaches with trade-offs

**ASK:** "Which approach do you prefer?"~/.config/goose/recipes/goose-coder.yaml

This pattern ensures that architectural direction is always a human decision, not an inference from incomplete context. The full recipe is available at goose-coder.yaml on GitHub Gist.

Hard Blocks vs. Guidelines

Governance controls operate at three layers. AGENTS.md is the policy declaration: it instructs the agent on commit conventions, identity requirements, and policy boundaries before any code is written. Local git hooks enforce the same contract at commit time.

**Table 2:** Three-layer governance enforcement stack
Layer	Mechanism	Controls
Policy declaration	AGENTS.md	Commit conventions, identity requirements, policy boundaries
Commit-time	git hooks	Conventional commits, DCO sign-off, protected branch block
Repository	Branch rulesets, code owner review	GPG signing, SLSA provenance, OpenSSF certification

Repository controls enforce the rules again at the server: branch rulesets, required code owner review, GPG signing, and provenance attestation. All three layers are intentionally aligned: a well-configured agent should never trip a hook. REPO-STANDARDS.md documents the full pipeline.

Code Snippet 2: Global commit-msg hook enforces conventional commits and DCO. Hard blocks prevent non-compliant commits from reaching review.

# Conventional commit format
CONVENTIONAL_REGEX='^(feat|fix|docs|...)(\([a-z0-9_-]+\))?(!)?: .{1,100}$'

if ! echo "$COMMIT_MSG" | grep -qE "$CONVENTIONAL_REGEX"; then
    echo "BLOCKED: Commit message must follow conventional format"
    exit 1
fi

# DCO required
if ! grep -q "^Signed-off-by:" "$COMMIT_MSG_FILE"; then
    echo "BLOCKED: Missing DCO (Signed-off-by)"
    exit 1
fi~/.githooks/commit-msg

How Do You Maintain Accountability When AI Writes the Code?

When any portion of code is AI-generated, the accountability question sharpens. The human who submits and certifies the code bears full responsibility for what it does, regardless of how much of it was generated. Making that accountability explicit requires controls that a policy document alone cannot provide. AI_POLICY.md formalizes four of them.

DORA 2025 confirms that AI acts as an organizational capability amplifier: the greatest return on investment accrues to teams with strong review discipline and platform engineering foundations, not to teams that simply adopt the tools (DORA / Google Cloud, 2025). The governance layer described here is precisely those foundations.

Figure 1: Governance chain. Each step adds a verifiable artifact; the chain is only as strong as the named human at the review gate.

Authorship and Attribution

Industry practice varies: the Linux kernel requires Assisted-by disclosure; Claude Code adds Co-Authored-By: Claude by default. The position here is accountability over attribution. DCO (Developer Certificate of Origin) sign-off is a responsibility certification, not an originality one: the committer certifies they have the right to submit the change and understand its contents. That certification is only honest if the reviewer has engaged with the code. As AI generates more of it, the reviewer role shifts from syntax-checker to spec-verifier and security judge: a higher-accountability function, not a diminishing one.

Commit Integrity

All commits are GPG-signed. A GPG signature ties the commit cryptographically to a verified identity, making it impossible to silently alter commit history or impersonate a contributor. Combined with DCO, every commit carries a named, verified human who certified its contents. These two controls together mean the audit trail is tamper-evident: you can verify not just what changed but who vouched for it.

Review Accountability

Every non-trivial change requires a named human reviewer. The PR checklist includes an explicit attestation: the reviewer has read every line and can explain it. This standard matters specifically because AI-generated code can be syntactically correct and pass all tests while containing subtle assumptions the reviewer would catch if they engaged deeply. Approving on the basis of CI green alone is the failure mode the attestation is designed to prevent.

Build Provenance and Certification

SLSA (Supply-chain Levels for Software Artifacts, a build integrity framework) Level 3 build provenance establishes a verifiable chain from source to artifact. Every build produces a signed attestation of what was compiled, from which commit, with which toolchain. OpenSSF (Open Source Security Foundation) Best Practices certification documents that these controls are maintained continuously.

These controls are not compliance theater. They are the structural response to the accountability gap: when code is generated faster than it can be understood, the governance layer is the only verified signal that a qualified human evaluated what ships.

What Do Measured Time Savings Actually Look Like?

Two production examples provide concrete data. Both are single-author measurements on a controlled codebase without a comparison group. They are illustrative, not generalizable. These gains held because the governance controls in the previous section ensured reviewer quality was sufficient to catch AI errors; without that condition, the same speed would produce a different outcome.

CI Modernization (PR #52)

math-mcp-learning-server had no CI workflow. The judgment call: build from scratch or adapt patterns from a similar project. AI identified Ruff, uv, and pytest-cov as the right stack. Review covered the risk assessment, tooling fit, and zero-regression confirmation. Result: approximately 20 minutes versus an estimated 3-4 hours, with CI runtime at 5 seconds and 67 tests passing at 83% coverage. Source: PR #52.

Matrix Operations Feature (PR #109)

Five matrix operation tools with NumPy integration. The judgment call: implement incrementally or batch with shared validation patterns. AI identified the common infrastructure needs: dimension validation, ToolError handling, DoS prevention via size limits. Review covered API design, error handling conventions, and security boundaries. Result: 2 minutes from PR creation to merge, 5 tools, 21 tests, 395 lines. Source: PR #109.

**Table 3:** Measured time savings from two production examples (single-author, no control group)
Task	AI-Assisted	Traditional Estimate	Savings
CI modernization (PR #52)	~20 min	3-4 hours	~90%
Matrix operations, 5 tools (PR #109)	2 min	1-2 hours	~95%

At 10 infrastructure tasks per month, savings at this rate recover approximately 60 hours per year per engineer. That estimate depends entirely on reviewer quality being sufficient to catch AI errors; the governance controls in the previous section are what make that condition hold.

GitHub / Kalliamvakou (2024) found a 26% overall productivity increase across 4,867 developers, with 60-75% reporting increased job fulfillment. These aggregate figures align with the production examples directionally. The governance layer is what separates an individual positive experience from a reproducible organizational outcome.

When Does This Approach Work — and When Does It Not?

**Table 4:** Task type fit for AI-assisted development, based on production experience
Task Type	Fit	Evidence
CI/DevOps automation	High	20 min vs 3-4 hrs (PR #52)
Feature implementation with established patterns	High	2 min for 5 tools (PR #109)
Boilerplate and scaffolding	High	Common pattern in both PRs
Legacy code: analysis and documentation	High	aptu-coder: structured agent access without full context load
Greenfield architecture	Medium	More judgment gates needed
Security-sensitive code	Low	Context underspecification risk
Regex and parsing logic	Low	Subtle bugs compound
Legacy code: generation and modification	Low	Hallucination risk without grounding context

Security-Sensitive Tasks

The security-sensitive category warrants elaboration. The 45% security flaw rate from Veracode (reported in Zigler, 2025) applies specifically when context is underspecified: incomplete specifications, ambiguous threat models, or missing documentation of invariants. It is not a universal finding about all AI-generated code. Well-specified codebases with clear security requirements produce substantially better results. Reviewer attestation and SLSA provenance are the mechanism for verifying that the higher-quality path was actually taken.

The Candidate Generation Model

The critical success factor is consistent across task types: the human evaluating AI proposals must have sufficient expertise to recognize errors. DORA 2025 confirms that AI amplifies existing organizational capability rather than substituting for it (DORA / Google Cloud, 2025). A team without strong review culture will see AI increase their defect rate, not decrease it.

Backlund (2024) frames this as a candidate generation problem: in large-scale projects it is infeasible to thoroughly research every decision, but more candidates increase the likelihood that the ideal solution is among them. AI excels at candidate generation. Human judgment determines which candidate ships.

What Does This Mean for Technical Leaders?

The Perception Gap Risk

That perception gap is an organizational risk, not just an individual one. Teams adopting AI tools without governance controls may be degrading engineering throughput while believing they are improving it.

Implementation was already a small fraction of developer time (Kumar et al., 2025). AI compresses it further. The question is whether the time freed is reinvested in the judgment tasks that determine output quality.

Three Conditions for Success

The governance layer described here works when three conditions hold: reviewer expertise is sufficient to evaluate AI proposals, the workflow enforces human approval at structural decision points, and cryptographic controls make accountability non-repudiable.

DORA 2025 is direct on this: teams with strong engineering foundations see the gains; teams without them see increased complexity, larger pull requests, and architectural drift (DORA / Google Cloud, 2025). The governance layer is what creates the foundation.

This is Part 1 of a two-post series. Part 2 scales these controls across multi-agent workflows: Orchestrating AI Agents: Subagent Architecture.

The practical test is simple: audit your last ten pull requests and ask how many reviewers could explain every line of AI-generated code before approving. That gap between “CI passed” and “I understand this” is where the accountability layer lives.

References

Backlund, Emil, “A Cost-Based Decision Framework for Software Engineers” (2024) — https://www.emilbacklund.com/p/a-cost-based-decision-framework-for
Becker et al. / METR, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity” (2025) — https://arxiv.org/abs/2507.09089
DORA / Google Cloud, “State of AI-assisted Software Development 2025” (2025) — https://dora.dev/dora-report-2025
GitHub / Kalliamvakou, “Research: quantifying GitHub Copilot’s impact on developer productivity and happiness” (2024) — https://github.blog/news-insights/research/research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/
Karpathy, Andrej, “Software Is Changing (Again)” (2025) — https://x.com/karpathy/status/1886192184808149082
Kumar et al., “Time Warp: The Gap Between Developers’ Ideal vs Actual Workweeks in an AI-Driven Era” (2025) — https://arxiv.org/abs/2502.15287
Peng et al., “The Impact of AI on Developer Productivity: Evidence from GitHub Copilot” (2023) — https://arxiv.org/abs/2302.06590
Prather et al., “The Effects of GitHub Copilot on Computing Students’ Programming Effectiveness, Efficiency, and Processes in Brownfield Programming Tasks” (2024) — https://arxiv.org/abs/2506.10051
Shukla, Bui, Parsons et al., “De-skilling, Cognitive Offloading, and Misplaced Responsibilities: Potential Ironies of AI-Assisted Design” (2025) — https://doi.org/10.1145/3706599.3719931
Zigler, Andrew, “Mise en Place for Agentic Coding: Deliberate Preparation as Context Engineering Methodology” (2025) — https://arxiv.org/abs/2605.05400

AI-Governed DNS Migration Without Maintenance Windows

Hugues Clouâtre — Sat, 30 May 2026 10:25:00 GMT

DNS migrations fail in production because teams discover errors after the nameserver switch, not before it. The operational question is not whether AI can execute the migration, but whether you can validate the entire target state against live Cloudflare nameservers before committing. That pre-validation step, not the AI, is what makes business-hours execution safe.

The migration described here moved 20+ DNS records, three CI/CD workflows, and a static hosting platform from Amazon Route53 and GitHub Pages to Cloudflare Workers in two hours, during business hours, with zero downtime. The AI assistant (Goose with Claude Sonnet) is the implementation detail. The governance architecture, pre-validation pipeline, and the auditable artifacts it produced are the thesis.

Human error causes 66-80% of infrastructure outages (Uptime Institute, 2025). The operational solution is not to be more careful; it is to eliminate the manual steps that create the error surface in the first place.

Contents

Why Do DNS Migrations Still Require Weekend Windows?
What Does Programmatic Pre-Validation Actually Look Like?
- How Does the Pre-Validation Window Work?
- What Does a Clean Diff Confirm?
How Does the Approval-Gate Workflow Distribute Risk?
- What Does Each Gate Surface?
- How Does This Meet DORA Elite Performer Criteria?
What Did the Migration Produce as Artifacts?
How Do the Numbers Compare to a Manual Migration?
- Where Does the 88% Deploy Reduction Compound?
When Does This Pattern Apply Beyond DNS?
- Which Infrastructure Patterns Generalize?
- When Does This Pattern Not Apply?
Takeaways
References

Why Do DNS Migrations Still Require Weekend Windows?

Weekend deployment windows are a risk-management response to an information problem: teams do not know whether their target configuration is correct until they switch nameservers and observe production traffic. At that point, any error is already in production: a missing MX record, a transposed SPF value, an incorrect CNAME priority. Rollback means re-pointing nameservers and waiting for propagation, typically 15-60 minutes with modern low-TTL configurations, but with real exposure in the interim.

The weekend window does not reduce the probability of error; it reduces the business cost of the error by shifting it to off-peak hours. That is a reasonable mitigation for a manual process, but it is not a governance solution. It accepts the failure mode and schedules around it.

The compounding problem is knowledge concentration. DNS record formats, TTL semantics, DNSSEC interactions, and cloud provider API specifics are tribal knowledge. When the engineer who last touched Route53 is unavailable, the migration either waits or proceeds with higher risk. Downtime costs Global 2000 companies $400B annually (Splunk/Oxford Economics, 2024), and a significant fraction of that cost comes from configuration errors that a programmatic diff would have caught before execution.

What Does Programmatic Pre-Validation Actually Look Like?

How Does the Pre-Validation Window Work?

The core mechanism is querying the target nameservers directly before the nameserver switch. Cloudflare assigns nameservers to a zone the moment the zone is created; those nameservers resolve the zone’s records independently of where the domain’s registrar is pointing. This creates a window in which you can validate the entire target DNS state against authoritative Cloudflare resolvers while production traffic still flows through the old nameservers.

Cloudflare’s nameserver infrastructure serves as the reference implementation: see Cloudflare’s full setup documentation for the zone creation mechanics that enable pre-validation.

What Does a Clean Diff Confirm?

Code Snippet 1: Pre-validation against Cloudflare nameservers before switching (zero output from diff confirms zero-risk parity)

# Verify records match before switching nameservers
dig @nameserver1.cloudflare.com clouatre.ca MX +short
# Output: 1 aspmx.l.google.com. (matches Route53)
diff <(aws route53 list-resource-record-sets) <(curl cloudflare-api)
# Output: (empty = 100% match, zero risk)scripts/validate-cloudflare-dns.sh

The diff producing zero output is the safety gate. Every MX record (5 records for Google Workspace), every TXT record (SPF, DKIM, DMARC), every CNAME, and every A record was verified against Cloudflare’s authoritative nameservers before the registrar switch was initiated. The validation report confirmed 100% parity across all 20+ records.

This approach externalizes domain knowledge into executable code. The validation script encodes what “correct” looks like; it does not rely on an engineer remembering every record type and priority.

How Does the Approval-Gate Workflow Distribute Risk?

What Does Each Gate Surface?

The migration ran as a five-phase recipe workflow: Analyze, Research, Plan, Implement, Prepare. Each phase produced a structured artifact and stopped at an explicit approval gate before proceeding. This is not autonomous execution; it is AI-assisted execution with human governance at every decision point.

Figure 1: Migration workflow with five phases and two human approval gates. Each gate produces a reviewable artifact before the next phase begins.

The first gate, after the Analyze and Research phases, surfaced the full infrastructure inventory: 20+ Route53 records, 15 classified as critical (email, Google Workspace, SSL validation) and 5 as obsolete. DNSSEC verification returned negative, confirming no migration blocker. A CTO reviewing this gate sees the full blast radius before any change is made.

The second gate, after the Plan phase, presented the complete Cloudflare zone configuration and the CI/CD migration design for approval before any API calls were made to the target platform. The Cloudflare API token (the one manual step in the entire workflow, approximately two minutes to create) was the only credential that could not be automated from the outside.

How Does This Meet DORA Elite Performer Criteria?

The result is a change process that satisfies the DORA 2024 elite performer profile: on-demand deployment with change lead time under one hour, achieved not by removing governance but by encoding it into the workflow structure itself. For a broader treatment of how governance gates integrate with AI delivery pipelines, see Decision Frameworks for AI Delivery.

The before and after infrastructure states are captured below. The key architectural shift is consolidation: from three separate platforms (Squarespace registrar, Route53 DNS, GitHub Pages hosting) to a single Cloudflare control plane for DNS, hosting, and CDN.

Figure 2: Infrastructure before (Route53 + GitHub Pages) and after (unified Cloudflare Workers platform). DNS, hosting, and CDN under a single control plane.

What Did the Migration Produce as Artifacts?

The standard framing for AI-assisted migration focuses on what the AI did. The more useful framing for a technical leader is what the migration left behind as operational artifacts.

What Does a 38-Second Deploy Time Prove?

The CI/CD reconfiguration produced a GitHub Actions workflow that deploys to Cloudflare Workers via the official wrangler action. The 38-second deploy time is a falsifiable metric visible in every subsequent CI run.

Code Snippet 2: GitHub Actions deployment to Cloudflare Workers (88% faster than the GitHub Pages baseline)

# Cloudflare Workers deployment (38-second deploys)
- name: Deploy to Cloudflare Workers
  uses: cloudflare/wrangler-action@v3
  with:
    apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
    accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
    command: deploy dist --project-name=clouatre-ca.github/workflows/deploy.yml

How Does Every PR Become a Compliance Artifact?

The governance trail is more significant than the workflow file. Every phase produced a pull request containing the proposed change, the rationale, and a rollback procedure. The PR is the audit artifact: reviewable before merge, traceable after. This satisfies common compliance requirements for change management without a separate change management system.

Why Is the Runbook Executable Code, Not a Document?

The validation script is itself a reusable artifact. Parameterized for a different domain and a different source DNS provider, the same pre-validation pattern applies to any subsequent migration. The runbook is not a document describing what to do; it is executable code that enforces correctness before each step executes.

The preview deployment infrastructure, provisioned as part of the migration, means every subsequent branch gets an isolated preview URL with seven-day auto-cleanup. This was not a pre-existing capability; it was produced as a migration output.

How Do the Numbers Compare to a Manual Migration?

The comparison baseline is a manual DNS migration executed by a competent engineer: exporting records from Route53, manually importing to Cloudflare, testing each record type, monitoring propagation, and coordinating the registrar switch. The typical execution window is four to six hours of focused work, scheduled over a weekend.

**Table 1:** Before and after metrics for the complete migration (DNS + Hosting + CI/CD)
Metric	Before	After	Change
DNS Resolution	20-30ms	10-15ms	~50% faster global access
Deploy Time	5-8 min	38 sec	88% reduction
Platform Cost	Route53: $12/year	Cloudflare: Free	Cost-neutral
Preview Deployments	None	Per PR, 7-day cleanup	New capability
Migration Window	Weekend	2 hours, business hours	Risk profile eliminated

Where Does the 88% Deploy Reduction Compound?

The deploy time reduction compounds across every subsequent deployment. At five deploys per day, that is approximately 35 minutes of CI waiting time recovered daily. The more significant number for risk management is the migration window: two hours during business hours versus a weekend event with an on-call team standing by.

The cost-neutral outcome is notable. Cloudflare’s free tier covers the use case, so the migration produced capability improvements (preview deployments, faster resolution, consolidated control plane) without a budget line. The DNS resolution improvement, from 20-30ms to 10-15ms globally, is consistent with DNSPerf benchmark data on Cloudflare’s anycast network performance.

When Does This Pattern Apply Beyond DNS?

Which Infrastructure Patterns Generalize?

The pre-validation pattern generalizes to any infrastructure change where a target state can be queried before traffic is redirected to it. DNS is a clean example because Cloudflare nameservers are queryable independently of registrar pointing, but the structural pattern is broader.

The same governance architecture applies to CDN origin switching (validate the new origin responds correctly before updating the CDN routing rule), database migration cutover (validate read replica lag and schema parity before promoting), and API gateway routing changes (validate the new backend passes all contract tests before updating the routing table). In each case, the question is the same: can you assert that the target state is correct before making it the production state?

**Table 2:** Pre-validation pattern applied across infrastructure change types
Infrastructure Change	Pre-Validation Query	Assertion Condition
DNS migration	Authoritative nameserver dig against target zone	Record parity diff is empty
CDN origin switch	Health check against new origin backend	All contract tests pass
Database cutover	Read replica lag + schema diff	Lag below threshold, zero schema delta

The conditions under which this pattern is strictly better than alternatives (Terraform, Pulumi, manual execution) are worth stating precisely. Terraform and Pulumi produce excellent state management for greenfield infrastructure, but they require the source state to already be expressed as code. When the source state lives in a legacy cloud console with 20+ manually created records and no IaC history, the migration cost to get into a Terraform-managed state is itself a significant project. The approach described here treats the migration as a one-time governed event that produces an IaC artifact (the Cloudflare zone configuration) as output, without requiring IaC as a prerequisite.

When Does This Pattern Not Apply?

The pattern is not appropriate when the change cannot be pre-validated (some stateful database cutovers), when regulatory requirements mandate human execution of each step rather than scripted automation, or when the team lacks the domain knowledge to evaluate the AI-proposed configuration at each gate. The approval gates require a reviewer who can assess whether the proposed DNS records are correct, not just whether the workflow ran. For observability patterns that support gate reviewers in production, see Closing the AI Observability Gap.

Takeaways

Pre-validation is the safety mechanism, not the AI. Querying Cloudflare nameservers before the registrar switch converts an information problem into a boolean assertion. The AI accelerates the mechanics; the pre-validation architecture is what makes business-hours execution defensible.
Approval gates produce audit artifacts, not just checkpoints. Each gate generates a PR with rationale and rollback procedure. The compliance artifact is a byproduct of the governance structure, not a separate documentation effort.
The runbook is the deliverable. A parameterized, executable validation script that can be reused for any subsequent migration has more operational value than a completed migration. The infrastructure state is recoverable; an executable governance process is built, not bought.
Elite deployment performance follows from pre-validation confidence. The DORA 2024 finding that elite performers deploy on-demand with sub-hour change lead time is a consequence of investing in pre-deployment correctness guarantees, not a consequence of moving faster.

For a broader treatment of agentic workflow patterns and approval-gate architecture, see AI-Assisted Development: The Accountability Layer.

References

Cloudflare, “Change your nameservers (Full setup)” (2026) — https://developers.cloudflare.com/dns/zone-setups/full-setup/setup/
DNSPerf, “DNS Performance Benchmarks” (2026) — https://www.dnsperf.com/
DORA, “Accelerate State of DevOps Report” (2024) — https://dora.dev/research/2024/dora-report/
Splunk and Oxford Economics, “Downtime Costs Global 2000 Companies $400B Annually” (2024) — https://www.splunk.com/en_us/newsroom/press-releases/2024/conf24-splunk-report-shows-downtime-costs-global-2000-companies-400-billion-annually.html
Uptime Institute, “Annual Outage Analysis Report 2025” (2025) — https://uptimeinstitute.com/about-ui/press-releases/uptime-announces-annual-outage-analysis-report-2025

Hugues Clouâtre - AI & Platform Engineering

Open-Weight LLMs Reach the Structured Output Quality Ceiling

Table of contents

What Does It Mean for Structured-Output Quality to Hit a Ceiling?

How Was Experiment 9 Designed to Avoid Benchmark Gaming?

What Did the Blind Evaluation Reveal About Cost Versus Quality?

What Does a 95x Cost Spread Mean at Scale?

When Does Switching to Open-Weight Models Make Economic Sense?

Why Is C8 a Token Budget Problem, Not a Reasoning Problem?

What Does C8 Mean for Self-Hosted vs. Managed Inference?

How Wide Is the Remaining Gap Between Open and Closed Models?

What Are the Scope Boundaries?

When Do Closed Models Still Win?

What Does Apache 2.0 Licensing Change for Enterprise Procurement?

What Does Self-Hosting Provide for Audit Trails?

How Does EU AI Act Enforcement Shift the Data Residency Calculus?

Does Using a Third-Party API Create a Data Residency Risk?

Takeaways

References

AI Approval Gates: Engineering Oversight at Machine Speed

Table of contents

Why Are AI Agents Causing Production Incidents Without Any External Attacker?

Why Does Adding a Human Reviewer Sometimes Make Things Worse?

Why Debugging Skill Is the Specific Casualty

What Does DORA Say About External Approval Processes?

What Is Reversibility, and Why Does It Define the Gate Design?

Why Risk Category Fails as a Gate Axis

What Does the EU AI Act Require From This Design?

How Does a Three-Tier Reversibility Gate Work in Practice?

What Happens When an Action Is Mis-Tiered?

How Do You Prevent Reviewer Atrophy From Hollowing Out the Judgment Tier?

How Do You Measure Whether Your Gates Are Holding?

What Should Engineering Leaders Act On?

References

AI Adoption in Engineering: Breaking the 50% Plateau

Table of contents

Why Do Enterprise AI Programs Stall at 50%?

The Self-Reporting Trap

The Deployment-Before-Instrumentation Pattern

Who Is Actually Blocked, and Why?

Three Cohorts, Three Interventions

The Policy Document as Unlocker

What Does Purpose-Built Tooling Change?

Benchmark Methodology

Benchmark Results

How Do You Instrument Before You Deploy?

Five Numbers Before Day One

What Generic Monitoring Misses

What Operating Model Sustains Adoption?

Phase 1: Baseline and Segmentation

Phase 2: Friction Removal

Phase 3: Workflow Integration

Phase 4: Sustaining

Which AI Adoption Plays Waste Budget?

How Do You Measure Adoption, Not Just Activity?

The Five-Number Rule

The KPI Scorecard

What Should Engineering Leaders Do Next?

References

AI Delivery Decision Frameworks: Type 1, Type 2, DACI

Table of contents

Why Are Staffing Mismatches Misclassified as Irreversible?

Which Decision Types Are Most Often Misclassified?

What Is the Type 1/Type 2 Decision Lens?

Delivery Examples of Each Type

How Does the Eisenhower Matrix Add Triage Ordering?

Why Q2 Is the Real Leverage Point

Immediate vs. Deferrable Actions Across Simultaneous Crises

Why Does Delivery Authority Collapse Without a DACI?

What Failure Mode Does DACI Prevent?

How Do You Exercise Authority Through Dotted-Line Relationships?

DACI in Practice: SOW Scope Disputes

How to Distinguish a Stakeholder Failure from a Delivery Failure?

How Do You Separate Perception from Delivery Record?

How Does Psychological Safety Aid Team Recovery?

When a Lead Engineer Requests Reassignment

How Does AI Tooling Shift the Staffing Reversibility Boundary?

How Does AI Tooling Reclassify Adjacent-Stack Decisions?

How Do You Qualify the Boundary at Intake?

How Do the Four Frameworks Form a Single Decision Cadence?