HYPERNOVELTYINSTITUTE

Technical intelligence
for accelerating change.

Search the record

Home/Analysis

Daily Hypernovelty Lead · AI systems · October 17, 2026

The Customer Metric Was Missing From the Ticket

FDE-Gym, a preprint submitted on October 15, measures a part of AI work that most benchmarks simplify away: the distance between a sponsor’s request and a system that improves the customer’s own operating result. Its 95 tasks begin with production-like systems, held-out data, scripted stakeholders, and requests adapted from 20 delivery scenarios across 18 sectors. The agent can finish the requested artifact and still score poorly if it solves the wrong problem, causes an incident, or consumes too much stakeholder time.

Two observers overlooking a working parcel-handling facility

That distance is what the researchers tested by varying what the agent knew at the start while keeping the customer environment and grader fixed. At the most detailed level, the instruction supplies the problem, its constraints, and the scoring criteria. At the least detailed level, the agent receives the sponsor’s original request, which may omit important rules or point toward the wrong intervention. The design separates implementation from the work of discovering what the customer actually needs.

On the 20 original tasks, the three leading configurations scored between 0.73 and 0.82 when the rules and constraints were supplied. Their scores fell to between 0.39 and 0.62 when the problem remained clear but the constraints had to be found. The authors identify that search for unstated constraints as the largest observed bottleneck for the leading models.

But the full 95-task results add an important limit. In the least specified setting, the best configuration averaged 0.56 on the benchmark’s scale and resolved 14 tasks. No configuration resolved more than 15. The leading systems recovered 72% to 85% of the key facts available from stakeholders, yet their delivered systems produced only 0.60 to 0.70 of the reference business improvement before contact costs. For the best-scoring configuration, every key fact was recovered on 23 tasks, but the delivered system reached at least 0.8 of the reference improvement on only 10 of them. Knowing the facts, then, was necessary without being sufficient. The agent still had to interpret the rules, change the correct part of the system, preserve existing controls, and improve the customer’s measured result. A completed ticket could hide failure anywhere along that chain.

The lower end of the results is more sobering. Before the benchmark deducted stakeholder contact costs, systems delivered by two smaller configurations performed below the unchanged system on 51 and 55 of the 95 tasks. These are results from one preprint and one evaluation harness, not estimates of failure across real companies. They do show why “it shipped” is a weak acceptance test when a system can alter payments, route incidents, screen applicants, or make operational decisions.

Verification bottleneck

The scarce check is whether delivery improved the customer’s world rather than merely satisfying the visible request.

  • The buyer has to verify that the stated task matches the operating problem and that important constraints were found before the system changed.
  • The delivery team has to connect the artifact to a customer metric, an incident boundary, and a known unchanged baseline.
  • The next evidence to watch is the paper’s promised human baseline, independent use of the benchmark, and results under other agent frameworks. The current study uses scripted stakeholders, one minimal framework, and scenarios weighted toward regulated workflows; related training-data contamination also cannot be ruled out.

Opportunities

A practical AI workflow assessment could begin with an outcome map rather than a tool demo. For each proposed automation, the map would record the sponsor’s request, the customer result that matters, the rules the system must discover, the people who can confirm them, and the conditions that should stop or reverse deployment.

This same logic supports a “do nothing” control in pilot reviews. The new system would need to beat the current process on the agreed metric while staying inside incident and stakeholder-cost limits. That would make rollback a normal part of acceptance rather than an emergency decision after damage appears.

For vendors, the useful deliverable may be a constraint-and-outcome receipt attached to the software: what the team learned, which assumptions were confirmed, what changed, how improvement was measured, and what remains uncertain. FDE-Gym is early benchmark evidence, but it points toward a durable buying rule. The customer should be able to inspect the path from request to result before a finished artifact counts as finished work.

Sources

FDE-Gym is a version-one preprint. Its results are benchmark-specific and are not production deployment rates or a vendor recommendation.

Back to the archive