A Lawyer Trusted ChatGPT in Court. The Cases Were Fake.

A lawyer asked ChatGPT to find case law for a real court filing. It handed him a set of decisions that sounded airtight. Not one of them was real.

According to the Associated Press, two New York lawyers and their firm were fined 5,000 dollars after submitting a legal brief built on citations that ChatGPT had invented. The cases did not exist. The quotes were never written. The rulings were never made. And here is the detail I cannot get past: one of the lawyers had actually asked the tool whether the cases were genuine, and it assured him they were. He filed anyway. The judge found the lawyers had abandoned their responsibility to check the work.

I spent thirty-six years in the fire service. You do not skip the size-up because the scene looks obvious. You verify before you act, because the cost of being confidently wrong is not measured in embarrassment, it is measured in people getting hurt. That instinct does not switch off when you leave the job. It is the whole reason a story about a legal brief stopped me cold.

Because the lawyers in this case were not fools. They were trained professionals who knew, better than almost anyone, that a citation has to be checked. They had the skill to catch the fabrication. They trusted the fluency instead. And that is the part worth sitting with, because it is a choice any of us can make on a busy day, in any field, when a tool hands us something polished and certain and we are tired and behind.

This is exactly the trap I wrote a book about. AI does not fail you by being dumb. It fails you by being fluent. It produces something confident and complete, with no gauge on the dashboard to warn you that it is wrong. A bad answer from a person often sounds unsure. A bad answer from AI sounds exactly like a good one. That is what makes it dangerous. It is glad to let you skip the one step that ever mattered, which is deciding for yourself whether the thing is actually true.

Notice what the tool did when the lawyer asked if the cases were real. It said yes. It affirmed. That is the default behavior of these systems, and it is worth understanding as a feature, not a glitch. They are built to be helpful and agreeable, which means they will confirm your hope right along with your error. A confident answer is not a true one. The confidence is manufactured. The truth still has to be earned, by you.

So the lesson is not that AI is useless, and it is not that these lawyers were uniquely careless. The lesson is that the tool removed the friction, and the humans let it. The verification step felt optional because the output looked finished. It was not optional. It never is.

Keep the step that matters. When a tool hands you an answer, especially a clean and confident one, ask where it came from, ask what it left out, and check it before you stake anything on it. That habit is unglamorous and slow and completely worth it. It is the difference between a tool that makes you sharper and one that quietly sets you up to fail in front of a judge.

The case is Roberto Mata v. Avianca, decided in Manhattan federal court in June 2023

The machine generates. The thinker decides.

This is part of Notes from The Amplified Mind, a series I am running in the months before the book comes out. The Amplified Mind arrives August 2026.