WASHINGTON – To a layman, the citations filed in an appeal of a denied Social Security claim sounded plausible, if not outright impressive: Brown v. Colvin. Wofford v. Berryhill. Hobbs v. Commissioner of Social Security Administration.
Each citation bore a case number and the initials of a judge in the federal court for the District of Arizona.
All of those judges exist. The three cases do not.
In all, 12 of the 19 cases cited were “fabricated, misleading, or unsupported,” U.S. District Judge Alison Bachus wrote in an Aug. 14 ruling sanctioning the lawyer behind a brief “replete with citation-related deficiencies, including those consistent with artificial intelligence generated hallucinations.”
As AI spreads in the workplace and classroom, it has crept increasingly into the courthouse.
DEEPER DIVE: Top 10 Arizona ZIP codes people are moving to in 2025
LOCAL NEWS: 100 best places to work and live in Arizona for 2025
The AI Hallucination Cases database – maintained by Damien Charlotin, a researcher at HEC Paris, a leading business school in France – identifies a half-dozen federal court filings in Arizona since September 2024 that include fabricated material from ChatGPT or another generative AI tool.
That’s more than any other U.S. jurisdiction besides the Southern District of Florida, which had seven such cases. Worldwide, the database shows 486 cases – 324 of them in U.S. federal, state and tribal courts.
Untrained individuals representing themselves accounted for four of the Arizona cases and 189 of the cases nationwide. But hallucinatory filings in the U.S. were also pinned on 128 lawyers and two judges.
“There are now hundreds of reported instances of lawyers (including those from top law firms), expert witnesses, and even judges filing documents in court that contain hallucinated citations and other content,” legal researcher Matthew Dahl said by email.
Dahl, who is pursuing both a law degree and a PhD at Yale University, co-wrote a 2024 paper titled, “Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models.”
When AI systems are used for legal research, he said, “we found that they hallucinate at a level far higher than would be acceptable for responsible legal practice.”
In one disciplinary case before the Colorado Supreme Court, a Denver attorney caught denying the use of AI accepted a 90-day suspension. The investigation found he had texted a paralegal about fabrications in a motion that ChatGPT helped draft and that, “like an idiot,” he hadn’t checked the work.
Errors generated by Microsoft Copilot, Google Gemini, Perplexity, Claude Sonnet and other AI models have also surfaced in courts nationwide.
Asked about its role in these fakery cases, ChatGPT-5 – which helped brainstorm headlines for this story – generated this answer:
“It’s definitely important and concerning from an ethical and practical standpoint. Cases like these highlight how misuse or overreliance on AI tools without verification can cause real harm — especially in domains like law, medicine or journalism where accuracy is critical.”
Such incidents, it added, “underscore why transparency, verification and human oversight are non-negotiable.”
Legal experts say the growing use of AI carries serious risks for a legal system that relies on case law and precedent.
“When lawyers cite hallucinated case opinions, those citations can mislead judges and clients,” said Christina Frohock, a University of Miami law professor. Her paper, “Ghosts at the Gate: A Call for Vigilance Against AI-Generated Case Hallucinations,” will appear in the winter 2025 issue of the Penn State Law Review.
“The hallucinations might then appear in a court order and sway an actual dispute between actual parties. To quote a federal court in California, that potential outcome is ‘scary,’” Frohock said. “If fake cases become prevalent and effective, they will undermine the integrity of the legal system and erode trust in judicial orders.”
Courts take such misconduct seriously. Violators often face public shaming.
In a recent South Florida case, a judge sanctioned a lawyer for including “false, fake, non-existent, AI-generated legal authorities” in eight related cases. The attorney will have to attach a copy of the sanctions order to every complaint he files for the next two years in that district.
In the Arizona case, Mavy v. Commissioner of Social Security Administration, Judge Bachus ordered attorney Maren Ann-Miller Bam to notify the three federal judges in Arizona whose names appeared on the fabricated opinions, in writing, that she had attributed fictitious cases to them.
Bam must also provide a copy of the sanctions order to the judge in any case in which she is the attorney of record.
“The Court does not take this action lightly,” the judge wrote.
Bam represented a 54-year-old man in a dispute over Social Security disability benefits. His application was denied, and he lost an administrative appeal before turning to the federal court.
Bam did not respond to questions about the sanctions. In a court filing, she acknowledged citing fake AI-generated cases and blamed an attorney assigned to prepare the brief. She assured the judge that “at no point did we intend to mislead the Court or submit citations not grounded in valid legal authority.”
The judge also ordered the court clerk to notify the bar in Bam’s home state, Washington, and revoked her temporary permission to appear in her courtroom as an out-of-state lawyer.
The Washington State Bar Association did not respond to queries about any potential discipline.
Since 2023, U.S. courts have taken a range of actions in response to false AI-generated legal material.
A New York federal judge fined one lawyer $5,000 in 2023 for submitting a ChatGPT-generated brief filled with fabricated cases, in one of many such cases that have drawn national attention.
Some lawyers have gotten away with warnings. Others have faced bar referrals, continuing-education orders or suspensions.
Some cases have been dismissed with prejudice, meaning the party is not allowed to try again.
Judges aren’t immune to the pitfalls of AI-assisted writing.
In Mississippi, federal Judge Henry Wingate admitted last week that an order he had issued – and quickly retracted – included false information generated by Perplexity. He blamed a clerk.
That order “was an early draft that had not gone through the standard review process. … This was a mistake,” the judge wrote to Senate Judiciary Chairman Chuck Grassley, R-Iowa, who’d sent him a chastising letter Oct. 6.
In July, U.S. District Judge Julien Xavier Neals of New Jersey withdrew an opinion after a lawyer notified the court that it included fabricated quotes and misstated case holdings.
The State Bar of Arizona has issued guidance that calls on lawyers to verify all AI-generated research before submitting them to a court or client.
The Arizona Supreme Court’s Steering Committee on AI and the Courts issued similar guidance last year, noting that “judges and attorneys, not AI tools, are responsible for their work product.”
In January 2023, SCOTUSblog, which covers the Supreme Court, tested ChatGPT’s legal acumen with 50 questions about the high court. Only 21 of the answers were correct. One egregiously wrong answer claimed that liberal Justice Ruth Bader Ginsburg had dissented in Obergefell v. Hodges, the 2015 ruling that recognized a constitutional right to same-sex marriage.
In his annual report on the federal judiciary at the end of 2023, Chief Justice John Roberts warned that “any use of AI requires caution and humility.”
Citing non-existent cases, he wrote, is “always a bad idea.”
Still, the use of AI continues to grow.
“Before this spring in 2025, we maybe had two cases per week,” said Charlotin, the French researcher who tracks such cases. “Now we’re at two cases per day or three cases per day.”
Charlotin noted that mistakes aren’t unique to AI; lawyers have long relied on interns, paralegals and junior associates whose work isn’t always perfect.
“What’s good about this hallucination thing is that it casts a spotlight on sloppy, bad lawyers,” he said.
This article first appeared on Cronkite News and is republished here under a Creative Commons Attribution-NoDerivatives 4.0 International License.
