HYPERNOVELTYINSTITUTE

Technical intelligence
for accelerating change.

Search the record

Home/Analysis

Lead analysis ยท October 7, 2026

The Agent Could Do the Task but Failed to Build the System

A preprint posted to arXiv on October 6, 2026, "Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?", tests a deceptively simple question: when an AI agent can already solve a task, can it convert that capability into a reusable, inexpensive artifact that handles the same task at industrial scale?

Conceptual illustration of one successful prototype beside a stalled scaled production line

That result was stark: across 60 runs covering 10 models and three large classification datasets, the conversion usually lost substantial performance or failed to finish.

This conversion is what the researchers call "bottling." They give an agent a large repetitive workload, up to 5.62 million documents in one task, along with a fixed ten-hour time budget, a weighted five-million-token cost ceiling, and full access to shell, Python, GPU, the internet, and the agent's own model APIs. The objective is to produce a reusable artifact that can process the full workload cheaply without calling the frontier model again for every item. The comparison baseline is zero-shot performance on a 1,000-example sample, the same capability the agent demonstrably possesses before bottling begins.

Forty-eight of the 60 bottled runs fell below the zero-shot 95% confidence interval for that model on that task. Thirty-one fell below a stronger baseline: a naive small-model distillation approach that uses no agentic infrastructure at all. Only 6 of 60 bottled runs recovered at least 95% of zero-shot performance. Twenty-one runs exhausted their budget; eight failed to produce the required output file.

The benchmark used three datasets, MAVE (4.77 million product-attribute pairs), ESCI (2.62 million query-product pairs), and RAID (5.62 million documents), run on a single A100 40GB GPU with 8 CPU cores and 64GB RAM. That is broad enough to expose a recurring engineering problem, though far too narrow to establish how agents will perform across production work in general.

By contrast, there are cases where bottling worked economically. On the MAVE task, Opus 5 produced a bottled artifact scoring mean F1 of .461 against a zero-shot .483, while cutting estimated cost from roughly $27,914 to about $31.40, approximately 889 times cheaper. On ESCI, Opus 5 bottled at macro-F1 .499 versus zero-shot .609, cutting projected cost from $17,273 to $26.28, roughly 657 times cheaper while retaining about 82% of the zero-shot score. Those tradeoffs may be acceptable in some production environments. They were unusual within this benchmark.

The result exposes a gap between two distinct competencies: answering a question correctly when prompted, and building a maintainable process that answers the question cheaply and reliably across millions of instances. A successful demo establishes that a model can answer the task in front of it. This benchmark asks whether an agent can turn that ability into a process. Organizations can mistake a high-scoring demo for operational leverage and discover the gap only after committing a large workload and budget to a system that produces no output file.

Verification bottleneck

Demonstrating that an agent scores well on a sample does not tell a buyer whether that agent can build a system that scores comparably at scale, finishes within a cost envelope, and delivers a product with known failure modes. The 8 runs that produced no output file represent a total loss of the time and budget invested, with no partial artifact to salvage. Buyers need held-out quality evaluation on the full workload distribution rather than sample results; completion guarantees with specified fallback behavior; full cost accounting that includes retry and repair cycles rather than projected inference costs alone; and explicit ownership of maintenance when the artifact degrades under distributional shift or API changes. The paper evaluates classification artifacts in a controlled harness. Production pipelines subject to security review, governance requirements, and ongoing monitoring are outside the benchmark's scope and remain entirely the operator's problem.

Opportunities

The bottling framing gives teams a concrete design target that capability benchmarks do not supply. Rather than asking whether a model can solve a task, teams can now ask whether the agent can solve the engineering problem of building a robust classifier for that task, and instrument for that difference before committing a workload.

The economic cases where bottling succeeded despite quality loss suggest a real optimization frontier. For tasks where 80 to 95 percent of zero-shot quality is acceptable and direct-inference costs are prohibitive, bottled artifacts may be useful under today's benchmark conditions. Identifying which workloads fall into that zone is a tractable problem with a payoff.

The benchmark's harness also provides a replication target. Three tasks and two bottled runs per condition are a thin base for general conclusions. Independent replication across more task types, looser hardware constraints, and longer time budgets could widen or narrow the measured gap considerably.

Sources

Back to the archive