Lead analysis · October 5, 2026
When a Model Stays on Topic and Still Gets the Law Wrong
A preprint submitted to arXiv on October 2 describes a 1,042-item benchmark built to test how fifteen large language models handle Colombian law across ten legal areas. Its practical finding is a gap between recognizing the right answer on a closed test and producing a sound legal answer in free text. The questions covered constitutional, administrative, criminal, labor, civil, commercial, procedural, tax, family, and markets law.
That broad test set was assembled by students and a legal team, then passed through legal-assistant review, specialized evaluation, and committee approval. The final set contained 305 multiple-choice questions, 682 semi-open questions, and 55 open-ended problems.
Every model received the same items at temperature 0, without internet access, retrieval, or external tools. That boundary matters. The study measures what a model could produce from its internal training under a fixed prompt. It did not test a legal research system connected to current statutes, cases, or an official document collection.
That setup produced a wide range of closed-test results. The strongest model answered 90.5% of the multiple-choice questions correctly, while the weakest reached 57.7%. Free-text answers were much less reliable by the study’s primary metric. Across 737 semi-open and open-ended items, no model exceeded 0.451 on the paper’s reference-anchored answer-correctness score.
This drop in correctness was not matched by an obvious drop in responsiveness. Mean relevancy scores ranged from 0.73 to 0.82 across models. At the individual-answer level, relevancy and correctness had a Spearman correlation of 0.08, which is close to no relationship. A response could address the apparent question while getting the governing fact or legal source wrong.
That gap becomes easier to see in the citation analysis. On a 350-item subset, the researchers classified 5,140 numeric legal citations generated across the tested models. They found that 53.2% were correct, 45.2% named a real norm that did not support the answer, and 1.6% named a norm that did not exist. The large middle category deserves attention. An invented statute may look suspicious, while a real statute cited for the wrong proposition can pass a quick inspection because the authority itself exists. The review task is therefore two-part: confirm the source, then confirm that the source supports the specific sentence.
Because the paper used automated judges for much of its free-text analysis, it also tested those judgments against a blinded human sample. One legal expert, supported by two students who checked each cited norm, agreed with the rubric judge’s wrong-or-invented citation flag in 91% of cited-answer cases. That supports the citation finding, while leaving the study’s human-validation sample and automated scoring as real limits.
Asking more models did not solve the problem. A majority vote across all fifteen systems scored 84.3% on the closed questions, below the best single model’s 90.5%. When models missed a question, the failing systems selected the same distractor 87% of the time on average. Agreement among models can reflect a shared misconception rather than independent confirmation.
Verification bottleneck
- Generation moved faster: Models could produce relevant legal prose more consistently than they could produce correct free-text answers.
- The reviewer has two checks: Verify that every cited authority exists, then verify that it supports the exact claim attached to it.
- The benchmark also needs review: The authors found one flawed item and several questions that depended on source documents the models could not see. Most legal areas were reviewed by one specialized evaluator, without independent double coding.
- Watch next: Tests that give models access to official Colombian sources could show how much retrieval changes citation accuracy and whether errors remain correlated.
Opportunities
This points to a possible citation-verification layer for AI-assisted legal work. Such a service could extract each claim and citation, retrieve the official source text, display the relevant passage beside the generated sentence, and require a reviewer to mark the relationship as supported, unsupported, or ambiguous. The receipt should preserve the model version, prompt, sources consulted, reviewer, and final disposition.
The same pattern can become a jurisdiction-specific acceptance test. A law firm, court, publisher, university, or public agency could maintain a small set of current local questions and rerun it whenever a model, retrieval index, prompt, or source collection changes. The useful result would be a traceable error profile rather than a single global accuracy score.
This preprint does not establish a universal legal-AI error rate, and it offers no buying guide. It shows why fluency, multiple-choice performance, and model consensus are weak substitutes for checking the authority behind a legal claim.
Sources
This article is not legal advice. The preprint has not been peer reviewed, and its findings do not transfer directly to other jurisdictions or to production systems with retrieval and current legal sources.