Retrieval is a data problem

01The wrong suspect

A team ships a retrieval-augmented assistant. The answers come back wrong, or vague, or confidently out of date. The instinctive fix is to swap the model for a bigger one, and when the upgrade lands, the failures stay. We have watched this loop play out more than once, and the reason it fails is simple: the model was never the component that broke.

The best public evidence here is Barnett et al., who catalogued seven failure points across three production RAG systems in research, education, and biomedicine.^[1] Three of the seven sit entirely upstream of generation. FP1 is missing content: the answer is not in the corpus at all. FP2 is ranking: the answer exists in a document that never reached the top of the list. FP3 is consolidation: the right chunks were retrieved but cut before the context was assembled. No prompt change and no model upgrade touches any of these. The same paper's operational conclusion deserves to hang over every RAG roadmap: "Validation of a RAG system is only feasible during operation," and robustness "evolves rather than designed in at the start."^[1]

End-to-end accuracy actively hides where the failures happen. RAG-X, a stage-level diagnosis of medical question answering, decomposed correct-looking behavior and found that only 49.2 percent of it was genuinely grounded, meaning the retriever found the evidence and the generator used it. Another 33.9 percent were lucky guesses, where the retriever failed and the generator answered from its own memory, and 8.5 percent were information blindness, where retrieval succeeded and the generator ignored it. The authors name the resulting illusion the Accuracy Fallacy: a 14 percent gap between perceived success and evidence-based grounding.^[2] A system can demo beautifully while its retriever fails on a third of queries. That is why we treat retrieval as what it is, a data pipeline, and engineer it with the rigor we would demand of any pipeline feeding production.

02Parsing fails silently

Everything downstream inherits the parse. If a parser scrambles a table in a PDF, the scramble is embedded, indexed, retrieved, and summarized with full confidence at every later stage. There is no checkpoint where the system notices.

OHRBench measured this cascade directly: 8,561 unstructured document images from seven domains, 8,498 question and answer pairs, and a blunt verdict on current OCR and parsing tools: "none is competent for constructing high-quality knowledge bases for RAG systems."^[3] Even the best pipeline in the study lost 14 percent end-to-end F1 against ground-truth text, and retrieval absorbed the largest share of the damage: the strongest model scored 59.2 retrieval F1 versus 70.0 on clean text. The study also separates semantic noise, errors that alter content, from formatting noise, and shows quality degrading steadily as noise grows.^[3]

What makes parsing dangerous is that it fails without failing. A parser rarely throws an exception; it emits plausible text that embeds without complaint and retrieves wrong. In our engagements we start ingestion work by running candidate parsers over a sample of the customer's own documents, with tables, multi-column layouts, and scanned pages oversampled, and diffing the output against hand-checked transcriptions. Vendor benchmarks disagree with each other and tend to crown the vendor running them. A sample of your own corpus is the only parser benchmark that matters.

03Defaults are not engineering

Chunking looks like a minor preprocessing choice. Measured, it is not. Chroma's evaluation held the corpus and the embedding model constant and varied only the chunker: recall moved by up to 9 points between strategies.^[4] The more striking result was the default. OpenAI's then-default splitter, 800-token chunks with 400 tokens of overlap, scored 87.9 percent recall but 1.4 percent precision, the worst efficiency in the study.^[4] Thousands of production systems inherited that configuration by leaving a parameter unset.

Sophistication is not automatically the fix either. A 2026 systematic study found that plain sentence chunking matched semantic chunking up to roughly 5,000 tokens of context at a fraction of the indexing cost, that overlap provided no measurable benefit, and that quality fell off a cliff beyond roughly 2,500 retrieved tokens.^[5] The overlap finding directly contradicts other published sweeps, including Microsoft's, which measured best recall at 512-token chunks with 25 percent overlap.^[6]

We read the disagreement itself as the finding. Chunk size, boundary strategy, and overlap are corpus-dependent parameters, and the literature cannot settle them for you because the literature was not run on your documents. Structure-aware chunking that respects headings, tables, and sections tends to pay off on structured corpora, and simple splitters often hold their own on flat prose. The rule is to treat every chunking decision as a tunable and measure it against a labeled query set, exactly as you would treat a partitioning scheme in a warehouse.

04Hybrid is a data decision

The retriever itself is an architecture decision, and the public data on it is unusually consistent. BEIR established in 2021 that "BM25 is a robust baseline" that dense retrievers often failed to beat outside their training domain.^[7] Modern embedding models have closed much of that gap, but the complementarity has not gone away, because lexical and semantic retrieval fail on different query classes.

Microsoft's benchmark on customer datasets makes the ladder explicit:^[6]

The per-query-type breakdown matters more than the averages. On keyword-style queries, exact identifiers, product codes, error strings, vector-only search scored 11.7 against 79.2 for plain keyword search: a catastrophic failure that no embedding upgrade fixes, because the query has no semantics to embed. On misspelled queries the collapse inverts, keyword search scoring 28.8 against 54.6 for hybrid with reranking.^[6]

Anthropic's contextual retrieval work is the cleanest demonstration that these are data gains, not model gains. Starting from a 5.7 percent top-20 retrieval failure rate, prepending a short generated context to each chunk before embedding cut failures by 35 percent. Adding BM25 alongside the embeddings cut them by 49 percent, and adding a reranker reached 67 percent, a final failure rate of 1.9 percent. The one-time contextualization cost was $1.02 per million document tokens, and the generator was never touched.^[8] Every rung of that ladder is pipeline work: what you index, how you index it, how you rank it.

05Freshness and permissions

Two properties never appear in a demo and decide whether the system survives contact with an enterprise: freshness and permissions.

Freshness first. The entire case for retrieval over model memory is that the world changes faster than weights. FreshQA measured language models on fast-changing facts and found strict-mode accuracy between 0.8 and 32.0 percent, with flat scaling curves: bigger models did not help. Augmenting GPT-4 with search improved strict accuracy by 49 points.^[9] Retrieval is the fix, which means the index inherits the obligation. A system fed by a nightly batch job answers questions about the present with yesterday's data and gives no signal that it is doing so. The reference pattern is change data capture from source systems flowing through parsing, chunking, and embedding as a continuous pipeline. Where that is overkill, a scheduled re-index with an explicit staleness SLO, stated as p95 lag from source change to queryable vector, is the minimum bar. If nobody on the team can state the SLO, nobody is engineering freshness.

Permissions are harsher, because retrieval does not create permission mistakes, it operationalizes them. Concentric AI's analysis of 500 terabytes of enterprise data found an average of 802,000 files per organization at risk from oversharing, 83 percent of them overshared internally.^[10] Those files were always technically readable; nobody could find them. A semantic index finds them on the first well-phrased query. Microsoft's readiness guidance for Copilot concedes the shape of the problem: Copilot and agents "retrieve data from Microsoft Graph and respect existing permissions, sharing settings, and policies," and every mitigation on that page is a data-layer control: oversharing assessments, restricted access policies, and excluding high-risk sites from Copilot discovery.^[11]

EchoLeak made the stakes concrete. CVE-2025-32711, disclosed in June 2025 with a CVSS score of 9.3, was a zero-click attack: a single crafted email caused Microsoft 365 Copilot to retrieve and exfiltrate content from across the victim's retrieval scope, with no user interaction.^[12] The lesson outlives the patch: whatever your index can reach, an injected instruction can request. Index-scope minimization and query-time permission checks are security controls, and they live in the data pipeline, not in the model.

06Measure at the retriever boundary

The thread running through all of this is that failures localize to stages, and end-to-end scores cannot localize them. So we evaluate retrieval the way we evaluate any data pipeline: per stage, with owned metrics at every boundary. The procedure we default to:

1. Build a golden query set per corpus. Fifty to two hundred real user queries, each labeled with the passages that answer it. This is the pipeline's test suite; nothing below works without it.
2. Audit the parse before tuning anything else. Diff parser output against hand-checked transcriptions on a stratified sample. OHRBench shows the loss cascades through every later stage, so parser fixes have the highest leverage.^[3]
3. Measure at the retriever boundary. Recall at k, NDCG or MRR, and token-level precision against the golden set, reported separately from answer quality. Separate reporting is what exposes lucky guesses; RAG-X put the gap between perceived and grounded success at 14 percent.^[2]
4. Sweep the tunables. Chunk size, boundaries, overlap, hybrid weighting, reranker on or off, all against golden-set recall. Copy no defaults; the worst measured configuration in Chroma's study was one.^[4]
5. Attribute every failure to a stage. For each failed golden query, ask in order: was the answer in the corpus, did it survive parsing, did it land in a retrievable chunk, did it rank, did it reach the context, did the generator use it. This is Barnett's taxonomy turned into a triage runbook.^[1]
6. Set a freshness SLO and measure lag continuously, from source-of-truth change to queryable vector, with an alert when the budget is blown.
7. Re-run the golden set on every pipeline change, including parser upgrades and embedding model changes. An embedding model change is a schema migration: vectors from different models cannot share an index, so the change forces a full re-embed and a full re-evaluation.

None of this is exotic. It is the discipline data engineering has applied to ETL for two decades: contracts at boundaries, tests on real data, lineage when something breaks. The only novelty is that the pipeline's output is ranked context instead of a table.

07Where the thesis bends

Two qualifications keep us honest. The first: sometimes it really is the model. Four of Barnett's seven failure points are generation-side,^[1] and RAG-X's 8.5 percent information blindness, evidence retrieved and then ignored, is a generator failure no pipeline can fix.^[2] Our claim is that most failures start upstream, not all of them, and stage-level measurement is exactly what tells you which case you are in.

The second: long context is a real alternative for part of the problem. Li et al. found that when resourced sufficiently, long-context prompting "consistently outperforms RAG in terms of average performance," with cost as RAG's remaining advantage, and proposed routing between the two.^[13] For a small, static, low-traffic corpus, putting everything in context is a legitimate answer, and it deletes chunking bugs outright. But claimed windows are not effective windows. NoLiMa, which strips lexical shortcuts out of needle tests, found that 11 of 13 models claiming 128K contexts or more dropped below half their short-context baselines at just 32K tokens, with GPT-4o falling from 99.3 to 69.7.^[14] And nothing about a large window repeals the rest of the pipeline: the documents you stuff into it still have to be parsed correctly, kept fresh, and filtered by permission. Long context changes the retrieval budget. It does not change the data problem.

Our position, then. When a RAG system disappoints, work upstream in the order the pipeline runs: parse, chunk, index, rank, and only then the model. Treat the retrieval layer as a data product, with schema decisions, freshness SLOs, access controls, and a golden query suite that runs on every change. The teams that do this stop arguing about models, because their dashboards tell them where the problem lives. The teams that skip it keep buying bigger models to compensate for data they never looked at.