The Fluency Trap

The better an ungrounded AI system writes, the more dangerous it can become.

Anthropic's research on persuasion found that as models get larger and more capable, they become more persuasive, with frontier models arguing about as convincingly as humans [1]. Decades of psychology point the same way: statements that are easier to process feel more true, whatever their accuracy [2]. A Microsoft and Carnegie Mellon survey of knowledge workers found that the more confidence people place in generative AI, the less critical thinking they apply to its output [3]. An awkward or obviously incomplete answer invites scrutiny; a polished one can bypass it.

This is why early demos are so compelling, and we've built our share of them. Someone asks a complicated question and gets a clear, confident, well-organized response in seconds. It looks like hours of expert work compressed into a chat.

But then a scientist asks which experiment supports a biological claim, an analyst notices two similarly named customers have been merged, or a compliance reviewer discovers the cited policy was superseded last year. The legal profession has run this experiment in public: a running database of AI-fabricated citations in court filings has logged hundreds of cases, many ending in sanctions, most involving output that was too plausible to check [4].

The model reduced the effort of producing an answer without reducing the effort of verifying it.

While the agent produces high-quality prose, the user still has to work out what's true, where it came from, and whether the system interpreted it correctly. A few wrong details buried in an excellent analysis are harder to find than an answer that's obviously unreliable. This isn't an isolated problem with a few models: standard training and evaluation systems reward confident guessing over admitting uncertainty [5].

Grounded agentic systems

An ungrounded application asks the user to trust the model: here is the answer, assume the system knows what it's talking about. A grounded application asks the user to trust a chain. The system identified the right entity, queried an appropriate source, received a particular record, transformed it in a defined way, and produced a summary tied to the retrieved evidence. Each link can be inspected on its own.

The idea isn't entirely new. Retrieval-augmented generation was introduced partly because answers backed by retrieved documents can be checked and updated in ways a model's internal knowledge cannot [6]. That checkability is what we lean on when something goes wrong, because it lets us ask useful diagnostic questions. Did the system resolve the wrong company, gene, or customer? Query the wrong source? Was the record outdated? Did a transformation strip context, or did the model misread good evidence? A traceable chain of reasoning can be debugged and fixed.

Grounding reduces hallucinations and makes the remaining errors visible, localizable, and correctable.

A citation is not the same as grounding

A system can produce an answer first and then search for links that appear to support it, or cite a document without identifying the passage a claim depends on. That creates the look of traceability without the substance, and it's common. When the Tow Center tested eight AI search tools, more than 60% of responses cited sources incorrectly, usually with complete confidence and sometimes with fabricated URLs [7].

In a grounded system, the sources participate in producing the answer: retrieval happens before the conclusion is formed, identifiers and metadata survive the pipeline, and important claims point back to the records that informed them. The direction of travel matters.

The unglamorous layers do most of the work

In our experience, four capabilities make grounding real:

1. 01Source integration. Reliable access to the data that matters, with the boring questions answered: what happens when a source is down, can the system tell "no evidence exists" from "the query failed," when was the data last updated, and can the retrieval be reproduced later?
2. 02Entity normalization. Before retrieving anything, the system must know what the user means. Products, companies, genes, and customers all carry multiple names across systems. Retrieve evidence for the wrong entity and everything downstream is coherent and irrelevant. When two interpretations are plausible, the right behavior is often to ask.
3. 03Provenance tracking. Source, query, identifiers, retrieval time, version, transformations applied. Evidence shouldn't become anonymous just because it passed through a language model.
4. 04Evidence-aware synthesis. The model reasons from what was retrieved: it separates what a source reports from what the system infers, surfaces conflicts instead of blending them, and says when the evidence doesn't resolve the question. A grounded answer may be less elegant and carry more caveats. In production, that nuance is a feature.

Build the parts that survive a model swap

Like most teams, we spend real time choosing models, and the choice matters. It also keeps changing. A system built around one model's distinctive behavior gets reworked with every upgrade. A system built around its data, identifiers, provenance, and evaluations has a more durable center: source integrations, normalized entities, provenance records, and evaluation sets all keep their value across model generations. The model becomes a replaceable reasoning and communication component inside a system we own.

When we start a grounded project, the sequence looks like this: define the decision the system supports, identify the evidence a knowledgeable person would need, integrate a few authoritative sources properly, preserve provenance through every transformation, require the model to separate evidence from inference, and evaluate the whole chain on realistic cases. Prompt design still matters, but it comes after the system has something reliable to say.

We've found that initial results from grounded systems are, paradoxically, less impressive than just letting the model produce free-form prose. Forcing the model to use grounded sources limits its creativity. But as we improve the evidence that the model can access, we can achieve persuasive results that are also well-grounded and auditable.

The goal is justified fluency

The prose should be an interface to the evidence, not a substitute for it.

None of this argues against good writing. Clear synthesis is one of the most valuable things language models do, and most organizations are full of data and documents that people struggle to navigate. The ideal system retrieves deliberately, preserves identity and provenance, reasons within the limits of what it found, and explains itself clearly, including what's missing and where sources disagree. The language model shifts from being an impressive writer to being part of a system people can trust with real work.

References

1. [1]Anthropic, "Measuring the Persuasiveness of Language Models," Anthropic Research, April 2024. anthropic.com
2. [2]Rolf Reber and Christian Unkelbach, "The Epistemic Status of Processing Fluency as Source for Judgments of Truth," Review of Philosophy and Psychology 1 (2010): 563–581. springer.com
3. [3]Hao-Ping (Hank) Lee et al., "The Impact of Generative AI on Critical Thinking," Proceedings of CHI 2025. dl.acm.org
4. [4]Damien Charlotin, "AI Hallucination Cases Database." damiencharlotin.com
5. [5]Adam Tauman Kalai et al., "Why Language Models Hallucinate," OpenAI, September 2025. openai.com
6. [6]Patrick Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," NeurIPS 2020. arxiv.org/abs/2005.11401
7. [7]Klaudia Jaźwińska and Aisvarya Chandrasekar, "AI Search Has a Citation Problem," Columbia Journalism Review (Tow Center), March 2025. cjr.org