An AI agent asks a tool to book an appointment. The reply contains a date, a time, and a confirmation field. Every expected box is filled, so the workflow moves on.
The appointment may still be missing.
This quieter failure survives because software usually checks whether a response has the right shape. A new arXiv preprint on silent tool failures examines what happens when the shape is correct but the requested outcome is wrong. The result may contain inconsistent data, an empty transaction, or a plausible value that fails the task.
Sugam Panthi and Rabab Abdelfattah call these responses silent failures because they arrive looking usable. Their proposed monitor checks results against deterministic “outcome contracts.” These contracts describe conditions the task actually needs, using patterns mined from separate task traces or public schemas.
When a result violates a contract, the monitor keeps the raw response and adds a nonbinding receipt. The receipt identifies the violated condition and lists public recovery tools available to the agent. It leaves the next move open. The monitor does not repair the result, expose the hidden fault label, consult the evaluator, or remove the agent’s other options.
Because the warning leaves the next move open, the separation between warning and recovery matters in the paper’s experiments. On a frozen ToolMaze benchmark with injected failures, aggregate completion across four models from two provider families rose from 10.9 percent, or 35 of 320 tasks, to 28.1 percent, or 90 of 320. A separate replication with a third provider family rose from 5 of 80 tasks to 20 of 80. The authors also report gains of 14 and 12 points on two Tau-bench retail tiers.
The ablations narrow the interpretation. Removing the list of recovery tools removed the measured gain. Restoring the list restored it. More diagnostic detail and changes in when the receipt appeared produced no detectable difference. In these tests, a warning became useful when it pointed toward actions the agent could take.
Verification bottleneck
A valid response format can hide a failed outcome.
But the evidence remains bounded. ToolMaze uses constructed faults inserted into a controlled benchmark. It shows recovery from those failures under the tested conditions. It does not establish a production reliability rate.
Transfer was uneven. The monitor produced no measured net effect on held-out AppWorld tasks. On an incident-derived suite containing faults outside the vocabulary used to build the contracts, detection fell to 46 percent. Delivery continued, but task completion did not improve. Clean ToolMaze controls also contained both rescues and harms, which means a false alarm can push an agent toward an unnecessary or damaging recovery action.
The paper’s StableToolBench check offers a limited real-API signal. It measured how often the detector fired on cached responses. No live agent or counterfactual recovery test was involved, so those results cannot show whether intervention would have helped. Adversarial tools also sit outside the paper’s scope.
This is a preprint, and its results were not independently reproduced for this review. The practical lesson should stay smaller than the headline numbers: define the outcome, inspect the returned evidence, and keep recovery visible without assuming the monitor is an oracle.
Opportunities
So teams building agent workflows could start with an outcome-contract worksheet for consequential tool calls. For each call, it would record the task’s success condition, the evidence expected in the result, known invalid states, and the person or system authorized to choose a recovery path. A booking workflow might require a verifiable reservation identifier. A file-writing workflow might require the file to exist at the requested location and pass a content check.
A companion recovery receipt could preserve the raw response, name the failed condition, and present bounded options such as retry, alternate tool, human review, or stop. Each option would retain its own permission boundary. The receipt would support a decision rather than silently making one.
There is also a useful evaluation question here: what happens on clean calls? A monitor should be tested for rescues, missed failures, false alarms, and harm caused by needless intervention. Those outcomes belong in the same acceptance record.
People still have to decide which outcomes matter enough to specify and which recovery choices carry consequences that software should not own. That judgment is the Human Premium in this design. The monitor can reveal a broken condition. Authority over the next move remains a separate layer.
Sources
- Panthi and Abdelfattah, Outcome Monitors: Recovery Affordances for Silent Tool Failures, arXiv:2608.19303v1, August 19, 2026.