Modelling a Knowledge Graph That Retrieval Can Actually Use

Graph RAG has an appealing pitch: vector search returns passages that look like the question, a graph returns facts that are connected to it, and connection is closer to what people mean when they ask something.

That pitch is correct. It also skips the part that decides whether the project works, which is the schema. In practice most of these systems disappoint not because the traversal is wrong but because the graph was modelled without a clear idea of which questions it existed to answer.

The failure mode is modelling everything

The instinct on a first Graph RAG build is to extract every entity from the corpus and connect everything to everything. Every person, product, date, place and organisation becomes a node; every co-occurrence becomes an edge.

The result is technically a knowledge graph. It is also close to useless. Traversals return enormous neighbourhoods with no signal about which paths matter, extraction cost scales with a corpus you are mostly not going to query, and maintenance becomes a permanent tax because every document change touches dozens of nodes.

A graph earns its complexity by answering questions a vector index cannot. Everything else in it is cost.

Start from the questions that need a traversal

The useful design step is to write down the questions the system must answer, then sort them by whether answering requires following a relationship.

"What does this document say about X" needs no graph. That is similarity search, and a vector index does it well and cheaply. Building a graph to serve it is effort spent to arrive back where you started.

"Which agreements governing this entity have been amended, and what is in force now" is a traversal. So is "which projects used this technology in this sector", and "what changed between this version and the one the customer is running". These questions share a shape: the answer is not in any single passage, and assembling it means following relationships between documents.

Model those relationships. Then stop.

Edges need direction and time

The two properties most often omitted, and the two that cause the most wrong answers.

Direction matters because amends and is amended by are not the same fact, and a traversal that treats them as one will happily return a superseded term as current. Time matters for the same reason: an edge that was true in 2023 and is not true now has to say so, or the graph will confidently answer with history.

Where documents supersede one another, the superseding relationship needs an effective date on the edge, not just an ordering. Ordering breaks the moment two amendments are signed on the same day, which happens more than it should.

The graph and the vectors are not competing

The common framing treats these as alternatives. They are complementary, and the hybrid is what actually performs.

Vectors are good at finding a starting point from a vaguely worded question. Graphs are good at expanding from a known point along meaningful relationships. Lexical search is good at exact proper nouns — client names, product names, terms of art — where embeddings are unreliable in a way that surprises people.

A retrieval layer that uses all three, with the graph expanding from what similarity found rather than replacing it, outperforms any one of them. It is also more debuggable, because when an answer is wrong you can see which stage produced the bad candidate.

Extraction is where accuracy is won or lost

Everything downstream inherits the quality of entity resolution. If the same company appears as three nodes because it was written three ways, no amount of clever traversal recovers.

This is unglamorous work — normalisation, aliasing, deduplication, deciding when two similar names are the same organisation — and it deserves more of the schedule than it usually gets. Where resolution is uncertain, recording the ambiguity beats guessing. A node that says it might be one of two organisations is honest. A node that silently picked one is a wrong answer waiting to be believed.

Keep it reproducible

Graphs built by hand drift. The extraction should be a repeatable pipeline, runnable from the corpus in one command, so the graph can be rebuilt rather than repaired. Once it can be rebuilt cheaply, schema changes stop being frightening, and a schema you are afraid to change is one that will be wrong for as long as the system lives.

---

iLeaf builds RAG and Graph RAG systems in production, including contract intelligence over document estates where the operative term is rarely in the document you opened.