Signal
A reviewer can find the right problem and still fail to improve the work.
A new preprint tested that gap across 4,181 competition-math problems. One multi-agent system used a planner, an executor, and a specialized reviewer. Another put peer agents into a shared discussion around the same candidate answer.
But the specialized reviewer was better at identifying real errors. Its warnings were far less likely to change the answer that moved forward.
That split gives review systems a useful inspection sequence: was the critique right, did it alter the work, and did a later check show that the change fixed the problem?
What changed
The researchers evaluated four protocols on the ten difficulty tiers of Omni-MATH. Their primary matched runs used the open-weight gpt-oss-120b model for the acting agents and evaluator, with temperature set to zero.
On the easiest tiers, adding collaboration produced little gain over a single-agent iterative baseline. From tier four onward, the collaborative approaches began to separate. On tiers six through nine, they improved final accuracy by roughly 10 to 20 percentage points over the matched single-agent baseline, with broadcast peer discussion finishing ahead of the planner-executor-reviewer pipeline.
That is where the researchers followed the critique through the system.
The result was a sharp reversal between finding and follow-through. The hierarchical reviewer reached 0.861 precision, meaning its warnings were more likely to point to genuine errors than the broadcast peers' warnings at 0.644. Yet evaluator-confirmed useful critique changed the next candidate only 33.6 percent of the time in the hierarchy. Broadcast discussion changed it 93.5 percent of the time. Successful repair opened a second gap: the paper's reviewer-guided repair measure was 0.051 for the hierarchy and 0.286 for broadcast.
So the team tested two changes inside the hierarchical protocol. Requiring the solver to acknowledge the reviewer's feedback lowered final accuracy. Placing that guidance directly inside the solver's working context improved follow-through, though it recovered only part of the gap.
Why it matters
This is how a review receipt can look healthy while the artifact stays the same. The findings may be accurate. The reviewed party may acknowledge every one. Neither fact proves that a repair happened.
That distinction travels beyond math benchmarks. An editor can flag an unsupported claim that survives into the final copy. A code reviewer can identify a failure path that remains in the merged build. An audit can produce a careful recommendation that never changes the operating decision.
So the paper offers a measurement pattern rather than a ready-made rule for those settings. For each finding, a practical record could show whether it was accepted, rejected with evidence, superseded, or left unresolved. It could identify the affected passage, diff, or decision. Closure would require a relevant verification result instead of a polite acknowledgment.
What this does not prove
The paper is a new preprint and has not completed peer review. Its main setting uses competition math, one primary model family, and a fixed inference configuration.
But the two collaborative protocols also differ in several ways. Broadcast peers share candidate state and participate in collective approval. The hierarchy routes advice through specialized roles and leaves the next submission decision with the solver. Those differences prevent the comparison from isolating one causal variable.
A reduced replication with Gemma 3 actors changed the overall ranking between the protocols while preserving the separation between reviewer precision and critique uptake. So the study does not establish a universal winner for multi-agent design. It makes uptake visible as a separate thing to measure.
What to watch
Watch for evaluations that report the path from finding to changed artifact to verified repair. Reviewer accuracy and final pass rates can hide what happened between those points.
The next useful tests would move beyond competition math, compare more model families, and vary how much authority a reviewer has over the candidate that proceeds. Human review systems need their own evidence before adopting the paper's process measures as policy.
For the last review your team closed, can you point to the changed artifact and the check that proved the finding was resolved?