Confidently Wrong: What Happened When a Stanford Professor Trusted AI

In 2024, a Stanford professor named Jeff Hancock was asked to submit expert testimony in a Minnesota court case. The case was about a new law restricting the use of AI-generated deepfakes in elections. Hancock's job was to explain to the court why deepfakes are dangerous. Deepfakes are AI-generated content that looks real. It can mislead people, undermine trust, and cause serious harm.

He was, by any measure, the right person for the job. Hancock is a founding director of Stanford's Social Media Lab, one of the world's leading researchers on technology and misinformation, and has appeared in a Netflix documentary on the future of AI. He charged $600 an hour for his expertise.

To prepare his declaration, he used the large language model based chatbot ChatGPT to survey the research literature. ChatGPT provided citations. Hancock included them. He did not verify them.

When the opposing lawyers reviewed his declaration, they found two studies that did not exist. The journal was real. The study title was plausible. But no such research had ever been published.

Hancock's testimony was excluded from the case. The judge noted he had "shattered his credibility with this Court." The story was covered by outlets from the Stanford Daily to Gizmodo to Fox News.

Here was a world-recognized expert on AI misinformation, taken in by AI misinformation. In a case about AI misinformation.

And that's the point. LLMs produce fictional citations in the same confident, professional tones as real ones. If it can happen to Jeff Hancock, it can happen to anyone.