Context Layer vs Knowledge Graph vs RAG vs Agent Memory: What’s the Difference?

Four terms get used interchangeably in AI architecture discussions and they describe four different things: RAG, a knowledge graph, agent memory, and a context layer — also called a context graph, though the two names have drifted apart and we will say where.

They are not competitors. Each answers a different question, and a system that needs one usually needs two or three. This page defines each precisely, shows what happens when you pick the wrong one, and gives the decision table.

If your question is “what should I use instead of RAG”, that is a different question and it has its own page: RAG alternatives.

The one-line version

	What it stores	What it answers	Fails at
RAG	passages of text, as vectors	“what does the material say about X”	counting, relationships, permissions
Knowledge graph	entities and the edges between them	“what connects A to B”	what a document actually says
Agent memory	what this agent learned, per user or thread	“what did we establish earlier”	anything it was never told
Context layer	nothing of its own — it assembles the others	“what may this caller see, right now”	being a substitute for any of them

RAG: retrieval, and nothing more

Retrieval-augmented generation splits documents into chunks, embeds each chunk as a vector, and at question time finds the chunks closest to the question and puts them in the prompt.

That is the whole idea, and its limits follow directly from it. Similarity is not understanding: two passages can be close in vector space and unrelated in fact. Retrieval cannot count, because counting requires seeing every row and top-k by definition does not. And a passage carries no notion of who is allowed to read it, so permission has to come from somewhere else.

Where it genuinely wins: “what does our leave policy say about carry-over” — prose, written once, that somebody needs read back to them accurately with a citation.

The upgrade most teams skip. Pure vector search misses exact strings — invoice numbers, error codes, surnames. Full-text search misses paraphrase. Trigram matching catches the typo that defeats both. Running all three and fusing the rankings can catch those misses, but it can also add noise on ordinary questions, so compare it with vector-only on your own queries. In PostgreSQL that is three stock extensions — vector, pg_trgm and unaccent — and no new infrastructure.

Knowledge graph: relationships, made explicit

A knowledge graph stores entities as nodes and relationships as typed edges. Instead of searching for text that resembles your question, you traverse from one thing to another.

This is what makes multi-hop questions possible. Which suppliers are affected if this factory closes requires walking factory → components → products → suppliers. No amount of better chunking turns that into a similarity search, because the answer exists in no single passage — it exists in the connections.

The cost is honest and often understated: the graph is only as good as the extraction that built it. That extraction is usually an LLM reading your documents, and its mistakes become nodes that look exactly as authoritative as the correct ones. Schema design and maintenance is ongoing work owned by a person.

Where it wins: connections, dependencies, impact analysis, provenance. Where it does not: telling you what a paragraph says.

Agent memory: continuity, not knowledge

Agent memory is what an agent retains across turns and across sessions — the fact that you already said which entity you work for, that a previous run established a preference, that a question was answered yesterday.

It is frequently confused with retrieval, and the confusion is expensive in both directions. Memory is written by the agent; retrieval reads what your organisation wrote. Memory is therefore a cache of conclusions, and caches go stale silently. If a policy changed last week and the agent “remembers” the old one, memory is now actively wrong in a way retrieval would not have been.

It also creates a privacy surface that did not exist before: whatever the agent chose to remember is now stored, about a named person, with a retention period someone has to own.

Where it wins: continuity, personalisation, not repeating questions. Where it does not: ground truth. Anything that can change should be retrieved, not remembered.

Context layer: the system that assembles the rest

A context layer is not a fourth store. It is the layer that decides, for one caller and one question, what actually reaches the model — and, crucially, what may not.

Concretely it owns:

• ingestion — how documents arrive, get parsed, chunked and kept fresh;
• retrieval — running the legs above and fusing their rankings into one order;
• authorization — compiling the caller’s permissions into the query;
• assembly — deciding what fits in the window and in what order;
• tools — letting the agent ask a system of record when retrieval is the wrong instrument;
• governance — approvals before consequential actions, and an audit trail after.

On “context graph” versus “context layer”: the terms are often used for the same thing. Context graph usually emphasises the structure — the map of entities, documents and permissions and how they relate. Context layer emphasises the runtime — the thing standing between a question and a model. If someone uses them interchangeably, they are not making a mistake; if they are contrasting them, they usually mean structure versus runtime.

The failure that decides the architecture: who may see the answer

Every layer above can be built without a permissions model, and most tutorials do exactly that. The usual approach is to retrieve the top k and then drop what the caller may not see.

That ordering is the defect. Filtering after ranking means the ranking never considered the caller’s documents. A person with narrow access gets a short list or an empty one — not because the answer is missing, but because it was never a candidate.

The correct ordering compiles the caller’s permissions into the query, in the SQL predicate of every retrieval leg, so ranking only ever sees rows that person could open.

How much this matters is measurable, and the published benchmarks we know of understate it. They assign permissions at random. Real permissions are clustered — a team’s documents are semantically adjacent because a team works on one subject. On 100,000 documents with synthetic permissions, holding corpus, index, queries and the number of visible documents constant and changing only the arrangement, a hybrid search filtered after a tenfold fetch recovered 0.730 of the results for a caller who sees 10% when the permissions were random, and 0.419, with 13.0% of searches returning nothing, when they were clustered. At 1% visibility it recovered 0.086 and 0.021. With the permissions inside the query the caller got the same top ten as a corpus holding only their documents, under either layout (that is agreement with such a corpus, not relevance, and the vector leg is exact at or below 50,000 visible chunks). Every table is on the benchmarks page, and the method and its caveats are in Your ACL benchmark is measuring the easy case.

If more than one group of people uses your system, this is not an optimisation. It is the architecture.

Which one do you need?

If you are trying to…	Use
Answer from documents, with citations	RAG, with hybrid retrieval
Trace dependencies or impact	Knowledge graph
Stop re-asking what the user already told you	Agent memory
Give different people different answers from one corpus	Context layer with permission-aware retrieval
Count, list or check status	Neither — call the system of record
Ship any of the above without maintaining a pipeline	Context layer

The practical order for most teams: get hybrid retrieval and authorization right first, add tools for anything a database already knows, add memory when users return, and add a graph when the questions are genuinely about connections. Building the graph first is the most common expensive detour.

Frequently asked questions

What is the difference between a context layer and a knowledge graph?

A knowledge graph is a store: entities and the typed relationships between them. A context layer is a runtime: it decides what reaches the model for a given caller and question, and it may use a knowledge graph as one of its sources. One holds structure, the other applies it.

Is a context graph the same as a knowledge graph?

No. A knowledge graph maps concepts and their relationships in a domain. A context graph maps what is available to an AI system — documents, entities, tools and the permissions over them — so it can be assembled per caller. A knowledge graph describes the world; a context graph describes what this system may use, for this person.

What is the difference between RAG and a knowledge graph?

RAG finds passages that resemble your question and shows them to the model. A knowledge graph walks explicit relationships between entities. Use RAG when the answer is written down somewhere; use a graph when the answer is the connection between things and appears in no single document.

Is agent memory a replacement for RAG?

No, and treating it as one causes stale answers. Memory holds what the agent concluded; retrieval reads what your organisation wrote. Anything that can change should be retrieved fresh. Memory is for continuity — who the user is, what was already established.

Do I need all four?

Rarely at once, and almost never in that order. Most production systems run hybrid retrieval plus authorization first, add tool calls for anything a database already knows, then memory, then a graph if the questions demand it.

Does a context layer replace RAG?

No — it contains it. Retrieval becomes one leg among several, running under a permission predicate, alongside tools and governance, instead of being the whole architecture.