The Drag-and-Drop Illusion

You drag a hundred-page PDF into the chat. Then a whole folder. The upload bar fills, the assistant announces it has read everything, and for a moment the feeling is hard to beat: every document you never had time to read, suddenly on tap, answering in full sentences.

Then you use it for something that matters.

Here’s a test worth running tonight. Take a claim you half-believe and ask your AI to check it across the web. You’ll get a verdict in seconds, with sources. Click one. The page is real, the sentence is right there, word for word. But read around it: the AI has quoted a myth-busting article’s restatement of the myth it was about to demolish, or a comment beneath the study, or a 2019 figure that was true in 2019. The quote is real. It just answers a different question than the one you asked. By next week you’re repeating it as a checked fact. None of this is special to the open web. Whether the AI searched ten websites or ten of your own PDFs, a webpage is just a document it didn’t have to upload. It’s what your company’s tools are doing behind the polish, too: point Microsoft Copilot at your SharePoint, or an assistant at a data lake or a knowledge base, and the same machine is reaching into a bigger pile of text, pulling out what looks relevant. That reach is the whole pitch for putting AI to work. Retrieval is where the value shows up, and where the perceived value shows up too, long before anyone checks whether the two are the same.

Most of us picture these systems the way the products draw them, because the drag-and-drop gesture teaches the picture: files in, a capable model in the middle, answer out.

The picture is honest about the convenience and quiet about everything else. Inside that middle box the system is making calls the whole way down: which documents to open, whether the table parser kept each number with its label, which of six near-identical versions counts. And the call that decides everything: whether the passage it found actually settles your question. All of it out of sight. What surfaces is one confident reply.

Six good links make an unreliable chain

Break the chain into six stages, my own working split rather than an industry standard: capture, retrieval, authority, extraction, support, presentation. Say each stage gets it right 95 per cent of the time, a number any vendor would happily print on a slide. Multiply the six together and the chain works 74 per cent of the time. And that 95 assumes everything upstream went right. Add two more stages and you’re at 66. Nobody prints the chain number.

Real systems are messier than the arithmetic, because failures travel together: one badly scanned table can wreck capture, retrieval and extraction at once, and a superseded policy can be retrieved cleanly, quoted faithfully, and still be wrong. But the toy maths makes the point a single accuracy score hides: a chain of good components fails about one decision in four, right where you act on it.

I distrust model benchmarks for the same reason: they grade one link and sell the whole chain. Retrieval-augmented generation wired models to real sources instead of memory, ^[1] which fixed access. Getting from access to proof was left open, and mostly still is. The model is one link in the drawing.

A real citation can still launder a claim

Fabricated citations are the easy case. The source doesn’t exist, one click exposes it, everyone moves on. The failure that worries me keeps everything real: the system attaches genuine evidence to a stronger claim than the evidence can carry. Researchers studying cited AI systems recently gave this a name I like, citation laundering: a related source presented as warrant for an over-strong claim. ^[2] Put money on the question and you can feel how much it stings. Ask a document AI for a company’s public liability cover and it answers $110 million: correct policy, real page, the number printed exactly where the citation says it is. Except that figure is the building sum insured. The liability amount was two lines away, unread.

This has been measured. On one long-form question-answering benchmark, even the best models failed to fully support their claims half the time. ^[3] A 2026 study of deep-research agents found the strongest ones kept their links working and their sources on-topic most of the time, while factual accuracy against those same sources ran between 39 and 77 per cent. ^[4] Working links, plausible sources, and a coin-flip’s chance the facts survive contact with the evidence.

“A citation proves the system found a source. The chain proves the source supports the answer.”

Here’s the perverse part. A missing citation at least creates friction; you know the checking is yours. A working one invites you to relax at the worst possible moment: you clicked once, saw a real page, and counted it as verification. The green tick changed the subject. Once a team learns the citations can be decorative, they go back to checking every answer by hand, so the AI does the work once and a human does it again to see whether the first pass can be trusted. That’s the clean demo running on a dirty organisation, in production.

Relevance is cheap. Authority is the hard link.

Picture the folder the system is actually searching. A signed contract from last year. A newer amendment nobody signed. A policy marked draft, a final policy with an older file-modification date, an email summarising it wrongly, and three copies of everything scattered around. All of it is relevant to almost any question you could ask. Maybe two of those files carry any authority at all.

That judgement is the one a retriever was never built to make. It finds text related to your question; someone still has to decide which source gets to decide. You make these authority calls yourself without noticing: signatures, approval status, effective dates, which legal entity a schedule belongs to.

Chunk-and-search products flatten most of those signals. The knowledge becomes passages of language ranked by similarity, and an expired clause with the right vocabulary can outrank the clause that actually governs.

A bigger context window just rents a bigger warehouse, and nobody ever found a contract faster that way. Evidence buried in the middle of a long input is what models recover worst, ^[5] and ten near-identical policies in one window are ten chances to quote the wrong one fluently. The fixes that move the needle happen before the model writes a word: when Anthropic labelled its chunks with context, mixed keyword with semantic search, and reranked the results, its top-20 retrieval failure rate fell from 5.7 per cent to 1.9. ^[6]

A lot of what gets called hallucination begins life as a parsing failure, or a retrieval failure, or an authority failure. File everything under one word and none of it gets fixed. The underlying discipline is the one that separates serious AI users from frustrated ones: deciding what the machine reads, and in what order.

The serious version looks like an assembly line

Here’s the same job again, drawn with the hidden decisions showing. Read it as a proposal, one honest way to build the thing, and start at the three exits along the bottom.

A cited answer, a surfaced conflict, or abstain and fetch a human. That third exit is the most underrated feature in the field, because a question honestly has four outcomes (supported, conflicted, insufficient, unavailable), and most chat interfaces default to the first. “No active issue was found” is a conclusion. “The available documents don’t establish whether an issue exists” is a report on the evidence, and a system willing to say the second is one you can start believing on the first. A confident answer on insufficient evidence is a failure, even when it turns out to be right. You know the moment: the documents are silent, but the meeting still wants a number.

For now the humility runs backwards. The system saves it for after you’ve caught the mistake yourself, and if you’re lucky enough to catch it, spends it on the sycophant’s favourite line: “You’re absolutely right.”

The machinery upstream is what earns those exits: several kinds of search feeding a small, checked evidence pack, with the arithmetic handed to something that can’t improvise. A model can explain why two clauses conflict; it should not be the thing adding up the numbers. The diagram carries the rest.

Test it the same way, stage by stage, on messy documents rather than clean ones, since clean sets only test search. NIST’s generative-AI risk profile now tells organisations to do exactly that: check the sources behind an answer, and stop reading broad capability into narrow demos. ^[7]

The first diagram works fine for personal notes, a quick question, a summary where a miss costs nothing: a capable agent over your files is plenty. The design only has to match the cost of being wrong, and that cost jumps the moment people stop checking. A system that quotes contractual obligations or insurance limits inherits real authority from its first unchecked answer.

Johnny’s verdict

Stop asking whether an enterprise AI system is accurate. Ask where its accuracy comes from, and make the vendor walk the chain with you:

• Show me the parse. The document as the system read it, tables and footnotes intact.
• Show me the passage, and why that source outranked the drafts, duplicates and older versions of itself.
• Show me the connection between the number and the field: the exact words that support the claim, at the claim’s strength.
• Show me the conflicts it found, and the rule it used to resolve them.
• Show me the refusals. A system that never says “insufficient” is hiding the state you most need to see.

No vendor in the room? The list still works. Before you carry an AI’s number into a meeting, make it show you the passage and admit what it couldn’t establish.

My bet is that a modest model inside a well-governed chain beats a brilliant one inside a mystery, which makes “so which model do you use?” close to the least interesting question you can ask. Both diagrams here will hand you a clean paragraph and a source link; only one can tell you where the paragraph came from.

So run the test from the top of this essay: one claim, one question, then click the citation and read what the words actually prove. Any system can land on the right page, or a real sentence on a real site. Trust the one that shows you the distance between the evidence it found and the question you asked.

You are auditing an AI-generated answer as a chain of trust.

First, gather the evidence: ask me to paste the answer, its citations, and the source passages they point to. If I can only supply the answer, say plainly that the audit is limited to internal consistency, and proceed.

Judge each link in order:
1. Capture: did tables, footnotes, dates and page structure survive, or arrive flattened?
2. Retrieval: does the cited passage answer this exact question, or a nearby one?
3. Authority: is the source current, final, approved, and for the right entity and period? Name anything that could outrank it.
4. Extraction: does every number and clause belong to the field claimed? Quote the exact supporting words.
5. Support: mark each claim supported, conflicted, insufficient, or unavailable. Never upgrade weak evidence.

Deliver a table: claim, weakest link, verdict, and the one human check required before anyone acts on it.

And yes: this prompt launders my framework into your audit. Treat my claims as evidence to check, not rules to obey.

If you can browse the web, read the full essay first — it carries the complete argument and sources: https://thejop.com/essays/the-drag-and-drop-illusion/

Prompt from "The Drag-and-Drop Illusion" — Johnny Opinion Press, thejop.com