Your Vector Database Is Not the Hard Part

A recognisable way to start a retrieval project: several weeks comparing vector stores. Benchmarks, index types, hosted against self-managed, recall at various cut-offs.

It is a comfortable way to begin because it is a bounded technical question with published numbers. It is also, in most projects, close to irrelevant. The mature options are all good enough, and the thing that decides whether retrieval works is almost never which one you picked.

What actually determines retrieval quality

Three things, roughly in this order.

How the documents were split. A chunk is the unit of retrieval, so it is also the unit of failure. Split a contract every 500 tokens and clause 14.2 ends up divided across two chunks, neither of which is a complete thought. Split on structure instead — sections, clauses, headings — and each chunk is something a person would recognise as a unit. This single decision moves retrieval quality more than any index tuning, and it is specific to your document type, which is why it cannot be lifted from a tutorial.

What metadata travels with the chunk. Most real queries are not purely semantic. "What did we agree with this supplier last year" is a filter on party and date with a semantic query attached. If effective dates, entity, document type and version are not stored as fields, that query becomes a similarity search over everything and the answer is whatever happens to be phrased most like the question.

Whether relationships are represented at all. This is the one that quietly breaks legal, clinical and financial retrieval. Documents refer to other documents. An amendment modifies an agreement. A policy supersedes an earlier policy. A test result belongs to an episode of care.

Similarity search cannot follow that. It returns the passage that looks most like the question, and the superseding document usually looks less like the question than the original does, because it is terse and procedural. So the system returns a term that was replaced, cites a real clause from a real signed document, and gives you no indication that anything is missing.

Where those relationships matter, they have to be stored as edges and traversed, not inferred from text. The retrieval step becomes: find the governing document, walk the amendment chain in date order, apply what is still in force. That is a graph operation, and no amount of embedding quality substitutes for it.

The failure mode is silence

The reason these problems persist is that retrieval failures do not announce themselves.

A wrong query plan produces a slow query you can see. A wrong retrieval produces a fluent, well-cited, plausible answer that is wrong, and nothing in the output distinguishes it from the correct case. The person reading it has no signal. Neither does your monitoring.

This has a practical consequence: you cannot evaluate retrieval by trying it. Somebody has to build a set of questions with known correct sources and measure whether those sources were returned — not whether the final answer sounded right. Recall on the retrieval step is the number that predicts whether the system is trustworthy, and it is separable from generation quality, which means it is testable on its own.

Where the vector store choice does matter

Having said all that, a few of the criteria are real, and they are mostly operational rather than about search quality.

Can you filter and search in one operation, or does filtering happen after retrieval and quietly gut your result set? How does re-indexing work when the corpus is large and the embedding model changes — which it will? Does it run where your data is allowed to be? And can the team you already have operate it at three in the morning?

That last one decides more projects than recall benchmarks do. A store your team already runs, monitors and backs up is usually the right answer even if something else scores a couple of points higher.

A better order of work

Start with a hundred real questions from the people who will use the system, with the documents that ought to answer them. Chunk on document structure. Attach every field you might want to filter on, including the ones you are not sure about. Model the relationships between documents explicitly if there are any. Then measure retrieval recall against your question set.

Only after that does the store comparison become a decision with evidence behind it — and by then you will usually find that any of the credible options clears the bar, because the problem was never the index.