02/09/2026
A lawyer with 30 years' experience used ChatGPT to research a case. He submitted a 10-page brief citing real-sounding rulings. Every single one was invented.
The judge fined him and referred him for disciplinary action. It made global news, back in 2023. Two years later, in May 2025, a UK judge flagged another set of lawyers to their regulator after five more fake cases turned up in a filing.
Same mistake. Two years apart. From people whose entire job is checking facts.
Here's what's actually happening. AI doesn't know facts, it knows patterns. It's trained to always give an answer, so instead of saying "I don't know," it will sometimes generate something that sounds completely plausible and is entirely wrong. Add training data pulled from the whole internet, blogs, forums and opinion pieces sitting next to properly researched sources, and you've got a tool that's confident and wrong more often than most people expect. For new users, that happens roughly 𝟭𝟱 𝘁𝗼 𝟮𝟬% 𝗼𝗳 𝘁𝗵𝗲 𝘁𝗶𝗺𝗲.
This isn't a reason to avoid AI. It's a reason to use it with the same judgement you'd apply to any other source.
Three habits I teach people I work with:
1. Ask AI to rate its own confidence in what it just told you, 1 to 10.
2. Verify anything that matters against at least three independent sources before you act on it.
3. Keep a human in the loop for anything touching money, legal wording or a client's name.
None of this is about slowing you down. It's about making sure the judgement calls stay with people, and AI stays what it actually is: a very fast first draft, not a source of truth.
Where has AI already given your business a confidently wrong answer, and did anyone catch it before it went out the door?
📸 from my recent AI workshop.
