ISSUE 013
Can You Send It Without Checking
Every conversation I have about AI in a small business arrives at the same place eventually, and it is not a conversation about jobs or strategy. It is one person asking whether they can send the thing it wrote without reading it first. That is an empirical question and there is a lot of evidence on it, almost none of which is reported in a way a six-person company can use. Here is what I found, including the study where two expensive tools marketed as hallucination-free turned out not to be.
An idea worth thinking about
It guesses because guessing scores better
The clearest explanation I have read of why these tools make things up was published by researchers at OpenAI, about OpenAI's own products. Their analogy is a student sitting an exam. On a question you do not know, you can leave it blank and score zero, or you can write a plausible answer and occasionally be right. Over enough questions, guessing wins.
Their argument is that this is what training and evaluation do to a language model. Almost every benchmark that models are graded on scores a confident wrong answer the same as no answer at all — both get nothing — while a confident right answer gets full marks. So the behaviour that maximises the score is to produce an answer whether or not there is any basis for it. The models are not malfunctioning when they invent a citation. They are doing the thing they were optimised to do.
The authors call the penalty on saying I don't know an epidemic, and say the fix is socio-technical rather than technical: rescore the existing benchmarks that dominate the leaderboards, rather than adding yet another hallucination test. That is a claim about an industry, published by a company inside it, and I think the willingness to publish it is the most informative thing about it.
What this means on a Tuesday in a six-person business is narrow and useful. The tool will not signal uncertainty by hesitating, hedging or sounding unsure, because sounding unsure is precisely what it was trained out of. Confidence carries no information about correctness. Any process you build that relies on noticing when the output sounds shaky is built on a signal that is not there.
A tool I disappeared into
A leaderboard of how often models invent things, run by a company that sells the fix
Vectara publishes a public leaderboard scoring how often each model hallucinates, and you can open it today for free. The disclosure first: Vectara sells retrieval-grounding software, the scoring is done by its own commercial detection model, and it is therefore benchmarking the exact problem it sells a remedy for. Take the ranking as indicative rather than authoritative. The methodology is published, which is more than most vendor benchmarks manage.
The task is deliberately easy. Each model is handed a document and asked to summarise it; the hallucination rate is the share of summaries containing something not in the source. There is no recall from memory, no research, no judgement. The answer is on the page in front of it.
As of the October 2026 update the best model listed sat at 1.8 percent and the worst at 24.2 percent, with widely used models scattered between. One well-known model is at 7.0 percent, another at 12.0 percent, another at 13.6 percent. Those are rates at summarising a document that was supplied.
That is the number I would carry around. On the simplest factual task there is, with the source text sitting right there, good models still introduce something that is not in the document somewhere between one and seven times in a hundred. Everything you actually ask it to do — draft a quote, answer a customer about your return policy, summarise a call — is harder than this. If one in a hundred customer emails containing an invented detail is acceptable to you, fine. Decide that deliberately rather than by default.
Learn something
Two thousand two hundred and eighty-nine times it went to court
Damien Charlotin maintains a free, public database of court and tribunal decisions in which a judge found, or implied, that someone relied on AI-generated material that turned out to be fabricated. As of 11 November 2026 it held 2,289 cases. The whole thing downloads as a CSV.
The distribution is the education. The United States accounts for 1,552 of them, Canada 245, Australia 112, the United Kingdom 90. By who did it: 1,314 involve people representing themselves and 910 involve lawyers — professionals, in their own field, with their licence exposed. There are 34 involving judges and 18 involving expert witnesses. Outcomes run from a warning through orders to show cause and bar referrals to monetary sanctions.
Charlotin states the limits himself, which is why I trust the thing. It only includes decisions where a court addressed the AI use as more than a passing mention. It excludes mere allegations. It does not attempt to cover the wider universe of fake citations or AI use in filings, and he calls it a work in progress rather than exhaustive. It is a record of the cases that got caught and written up by a judge, which is a small and unrepresentative slice of the cases that happened.
Read ten entries and the pattern is obvious and transferable to any business. Nobody in these cases set out to file a fabricated citation. They asked for something, got a confident and well-formatted answer, and did not check, because checking felt redundant against how plausible it looked. The failure is never at the point of generation. It is at the point where a person decided the output did not need verifying.
What we’re seeing
Sold as hallucination-free, measured at 17 to 33 percent
A Stanford team ran what they describe as the first preregistered empirical evaluation of AI legal research tools: 202 queries, agreed in advance, covering general research questions, jurisdiction and time-specific questions, questions built on a false premise, and plain factual recall. They tested the paid tools sold to law firms. The finding in the abstract is that the two major systems hallucinate between 17 and 33 percent of the time.
The per-tool detail matters more than the range. Lexis+ AI gave accurate answers 65 percent of the time, with about 17 percent hallucinated and 18 percent incomplete. Westlaw's AI-Assisted Research was accurate 41 percent of the time, with about a third of responses hallucinated and a quarter incomplete. Ask Practical Law AI was accurate 20 percent of the time, hallucinating at about 17 percent but incomplete in 62 percent of cases. The authors count a response as hallucinated if it is incorrect or misgrounded — a false statement, or a true statement falsely attributed to a source that does not support it.
Now the marketing, which the authors quote. LexisNexis advertised "100% hallucination-free linked legal citations". Casetext said its product "does not make up facts, or 'hallucinate'". Thomson Reuters said "We avoid [hallucinations] by relying on the trusted content within Westlaw", and an executive said the retrieval approach "dramatically reduces hallucinations to nearly zero". Those claims were made about the products measured above.
Three caveats belong immediately next to the number. This is an arXiv preprint, marked under review when posted in May 2024, not a peer-reviewed publication at that version — though it was preregistered, which is a stronger guarantee against fishing than most peer review provides. It is specific to legal research, a domain where being wrong is unusually costly and unusually detectable. And the tools have had two years of updates since. The authors are also clear that these tools did hallucinate less than a general-purpose chatbot on the same queries. Retrieval helps. It does not do what the advertising said it does.
The transferable finding is not the percentage, and I would resist anyone who quotes 17 to 33 percent as a general AI error rate. It is this: purpose-built, professionally sold, retrieval-grounded tools, marketed with explicit zero-hallucination language, were substantially wrong under independent measurement. If that is what the top of the market looks like when somebody checks, no claim about accuracy from any vendor should move you — including claims about whichever tool you are currently paying for.
So, the question this entry exists to answer. Can you send it to a customer without checking? No — not because AI is useless, but because every number available says the error rate is well above zero and nothing in the output signals which items are wrong. The workable version is to sort tasks by what a wrong answer costs. An internal first draft, a rough summary for yourself, a list of ideas: send it, the cost of an error is a minute. A price, a date, a specification, a legal or safety statement, a citation, anything a customer will act on: check it against the source, every time, no exceptions, because that is the category where the 17 to 33 percent lived.
One question
What is the last thing an AI wrote that you sent without reading it properly?
Everyone has one. Mine was a meta description. Reply with yours and whether it was fine.
More in this pillar as it's written.
— Shana
Sources link to the original publisher, never to a summary.