Your ACL Benchmark Is Measuring the Easy Case

Every benchmark of access-filtered vector search that we are aware of assigns permissions at random. Ours did too. Random is the obvious null hypothesis, and it is the wrong one.

Permissions are never random. group:hr covers HR documents, and HR documents are semantically adjacent by construction. A group’s visible slice is a region of the embedding space, not a uniform sample of it — and that turns out to matter more than how large the slice is.

We measured it on 200,000 public documents at production embedding width. Holding the corpus, the index, the queries and the number of visible rows all constant, and changing only how those rows are arranged:

Recall falls from 0.980 to 0.473, and a quarter of queries return nothing at all.

Nothing errors. Nothing is logged. The results that do come back look entirely reasonable.

This post explains the mechanism, publishes the numbers, and shows the fix.

The two halves of the problem (only one gets written about)

Search “multi-tenant RAG” and you will find a lot of good writing about the first half:

Half one: not leaking. If you filter in application code, one forgotten if leaks another tenant’s documents. The standard advice — push the check into the database, use PostgreSQL Row-Level Security, make isolation structural rather than remembered — is right, and you should follow it.

Half two: still finding things. Once the filter is in place, does the query still return what it should? This half is barely discussed, and it is where the silent failure lives. A leak is loud once discovered. An under-returning query looks exactly like “no relevant documents found.”

The rest of this post is about half two.

Why a filter over a vector index loses results

Exact search over a filtered set is straightforward: check the predicate, score what survives, return the best. Vector search at any scale is not exact. It uses an approximate nearest neighbour index — usually HNSW, a navigable graph you walk toward the query vector.

The walk has a fixed budget. In pgvector that budget is hnsw.ef_search, and it defaults to 40 candidates.

Now add an access-control predicate. Postgres walks the graph in order of distance to the query and discards rows the predicate rejects as it meets them. This is a post-filter: the filter is applied after the index chooses candidates, not before.

So if a user can see 1% of your corpus, roughly 99 out of every 100 candidates the walk examines get thrown away. The budget is spent on rows that user is not allowed to have. The walk finishes, the query succeeds, and it returns whatever survived — which can be nothing.

Three properties make this genuinely dangerous:

1. It raises no error. An empty result is a valid result. Nothing is logged.
2. It is invisible in testing. At full visibility there is no problem at all — the filter rejects nothing, so the budget is never wasted. Single-tenant staging environments will never show it.
3. It gets worse as security gets tighter. The more carefully you scope access, the worse retrieval becomes for the most-restricted users.

The measurements

Everything below is reproducible. Dataset: BEIR nq, 200,000 documents, 120 queries with human relevance judgments, real text-embedding-3-small embeddings (1536-dim — production width, not a toy model). Postgres 16, pgvector 0.8.5, HNSW with m=16, ef_construction=64, ef_search=200, LIMIT 50, k=10. Ground truth is exact brute-force cosine similarity over the eligible rows, so every figure is measured against the true answer rather than against another approximation.

How clustered is built — deliberately simple, so it cannot be accused of being tuned to produce the result. Draw 40 seed documents uniformly at random; assign every document to its nearest seed by cosine similarity (a nearest-centroid partition, not k-means — the partition does not need to be optimal, only contiguous); then take whole clusters, in random order, until the target count is reached. The visible set is therefore a union of a few topics, which is what group membership actually looks like. random draws the same number of documents uniformly. One variable.

Permissions	Visible	post-filter recall	Empty	With iterative scan	Empty
random	10%	0.980	0.0%	0.980	0%
clustered	10%	0.473	25.0%	0.928	0%
random	5%	0.860	0.0%	0.974	0%
clustered	5%	0.327	40.8%	0.906	0%
random	1%	0.175	20.8%	0.966	0%
clustered	1%	0.087	75.0%	0.883	0%

Three things to read from it.

Clustering roughly doubles the loss. At 10% visibility, random permissions produce no problem at all — 0.980, and iterative scan buys nothing. Clustered permissions at the same visibility produce 0.473 and a quarter of queries empty.

The mechanism explains itself. HNSW descends greedily from an entry point toward the query. Scattered eligible rows get met all along the walk; concentrated ones do not. A query pointing away from the visible region spends its entire candidate budget on rows the caller may not see.

Tuning does not fully close it. With iterative scan enabled, clustered permissions reach 0.883–0.928 while random reaches 0.966–0.980 — same setting, persistent gap. That residual is the argument for deciding between an exact and an approximate scan per query, rather than tuning a parameter and hoping.

You may not be able to reproduce this, and that is also a result

Run the same corpus without forcing the index and the failure does not appear. With a GIN index on the ACL column, all twelve default-planner rows chose an exact bitmap+sort — recall 1.000, zero empty, at every level from 5% to 25%. At 1536 dimensions the vectors are large, an HNSW scan is expensive, and a well-indexed ACL predicate wins on cost. The table above therefore forces the vector index, and says so.

That is not reassurance. Which plan Postgres picks is a cost estimate, and it moves with embedding dimension, corpus size, random_page_cost, effective_cache_size and how fresh your statistics are. Selectivity estimation for array overlap (&&) on a GIN index is unreliable enough that the choice can flip without warning. So the honest statement is not “HNSW is dangerous” — it is that whether you are exposed is decided for you, per query, by a cost model you are not watching.

The exact path is not free either: 44–107 ms p50, up to 198 ms p95 over 10k–50k eligible rows at 1536 dimensions. That is the price of correctness at this scale, and publishing it is what makes the trade-off legible.

An earlier version of this post reported a 5,183-document run showing 69% of queries empty at 1% visibility. That result is retracted. Its baseline table carried no index on the ACL column while the shipped schema has one, so it compared against a configuration worse than the product. The numbers above replace it. We would rather correct our own benchmark in public than have a reader find it.

One thing the benchmark does not show

We also measured nDCG@10, expecting it to show restricted users receiving measurably worse answers. It did not separate the two modes — 0.094 versus 0.097 at 10% visibility.

That is a limitation of the experiment, not evidence the problem is harmless. ACLs in this benchmark are assigned uniformly at random, so they hide relevant documents at the same rate as everything else, and ranking quality falls for both modes because the answers genuinely are not in the visible set any more. Real access control is not random — the documents a user can see are strongly correlated with the ones relevant to them, because they are their team’s documents. Under a realistic assignment, the recall collapse would translate into a ranking-quality gap. Under this one, it cannot.

So: recall and empty-result rate are the honest headline metrics here, and nDCG is a control that behaved as expected. We are reporting it rather than quietly dropping it.

How to fix it

1. Turn on iterative scan (pgvector 0.8+)

pgvector 0.8 added iterative index scans, which keep resuming the graph walk until the LIMIT is satisfied or a scan ceiling is reached, instead of giving up when the first budget is spent.

SET LOCAL hnsw.iterative_scan = relaxed_order;

Use SET LOCAL rather than SET. It is scoped to the current transaction and reverts automatically, so it cannot leak onto a pooled connection and change behaviour for an unrelated later request.

There are two modes. strict_order preserves exact distance ordering; relaxed_order allows slight local reordering and recovers more. In our measurements strict recovered 19 of 20 expected rows where relaxed recovered all 20. If your results feed a fusion step that reads rank position rather than raw distance, relaxed is usually the better trade: a row missing entirely costs far more than a row two places out of position.

hnsw.max_scan_tuples (default 20,000) bounds the work so a pathological filter cannot walk the entire index.

2. Make sure the candidate budget is at least your limit

This one is easy to miss and affects every query, filtered or not:

SET LOCAL hnsw.ef_search = 50;  -- must be >= your LIMIT

ef_search defaults to 40. An HNSW scan cannot return more rows than its candidate budget, so if you ask for LIMIT 50 you will get at most 40 — a silent 20% shortfall on every search, with no filtering involved at all. We found exactly this in our own code while investigating the ACL problem.

3. Index the column you are filtering on

If your ACL is a PostgreSQL array and you match with the overlap operator, a btree index will not help — that operator cannot use one:

CREATE INDEX idx_chunks_acl ON chunks USING gin (acl);

Without it, the most frequently executed query shape in a multi-tenant deployment is a sequential scan, and it gets worse as the corpus grows.

4. Verify the index is actually being used

At small scale, the planner may choose an exact sequential scan, which filters perfectly and hides the whole problem. Your test suite can pass while proving nothing:

EXPLAIN ANALYZE
SELECT id FROM chunks
WHERE acl && ARRAY['group:finance']
ORDER BY embedding <=> $1
LIMIT 50;

If the plan says Seq Scan, you are not testing what you think you are testing. Force the index path in your benchmark, and keep a guard assertion so the test cannot silently degrade later.

How to check your own system in an afternoon

You do not need our benchmark to find out whether this affects you.

1. Pick a real query and a real restricted user.
2. Run your normal filtered retrieval and record how many rows come back.
3. Run the same query with the vector index disabled — SET enable_indexscan = off — so Postgres computes the exact answer.
4. Compare the counts.

If the exact answer has rows your indexed query missed, you have this problem. The gap will be widest for your most-restricted users, which usually means your most sensitive data.

If you are not on pgvector

The mechanism is not pgvector-specific. Any system that applies an access predicate as a filter over an approximate index is post-filtering, and has some version of this behaviour. What differs is the vocabulary and the escape hatch — most managed vector databases expose an equivalent of “search harder when filtering,” and some support pre-filtering, which sidesteps it at a different cost.

The question to ask your vector store is precise: when a filter rejects most candidates, does the engine keep searching, or return short? If the documentation does not answer that, measure it.

Why this matters more in 2026

Access-controlled retrieval used to be a nice-to-have. Two things changed it.

First, retrieval moved into workflows where being wrong is expensive — support agents quoting policy, analysts querying financials, clinicians reading notes. “No relevant documents found” is not a neutral outcome there; it is a wrong answer delivered confidently.

Second, regulation. The EU AI Act’s Article 12 logging obligations for high-risk systems became enforceable on 2 August 2026, and the deployer remains accountable regardless of who built the system. If you have to demonstrate that your AI behaved correctly for a given user, “our vector index silently skipped their documents” is a difficult sentence to write in an audit response.

FAQ

Does Row-Level Security fix this?

No. RLS solves the leaking half correctly — and you should use it — but an RLS policy is still a predicate applied over the index scan. It prevents the wrong rows coming back; it does not make the right ones come back.

Does this affect keyword or full-text search too?

Much less. Full-text and trigram search over a GIN index are exact for the predicate — they do not approximate. The problem is specific to approximate vector indexes, which is why a hybrid stack can behave strangely: the lexical legs return correctly while the vector leg quietly under-returns.

Will I notice this in production monitoring?

Almost certainly not, unless you are looking for it. The query succeeds, returns HTTP 200, and produces an empty or short result set. It looks identical to a genuine no-match. The only reliable signal is comparing against an exact oracle.

Is iterative scan safe to leave on?

In our measurements it cost nothing at full visibility (0.983 vs 0.977 recall, ~1 ms) and mattered enormously under tight filters. The cost scales with how much of the index has to be walked, bounded by max_scan_tuples. Enabling it only when a scope filter is present keeps the unfiltered path unchanged.

What if I cannot upgrade pgvector?

Below 0.8 there is no iterative scan. Options: raise ef_search substantially (helps, does not solve), partition by tenant so each tenant’s index is dense in its own rows, or pre-filter into a smaller candidate set before the vector step. Upgrading is the cleaner answer.

What we did about it

We build context-engine: a Python retrieval library that enforces access control inside the SQL predicate of every retrieval leg, so a document a caller may not see is never fetched, never ranked, and never reaches a reranker.

That design is exactly what made this bug ours to find. Our own vector leg was post-filtering and returned 0 of 20 matching chunks for a restricted principal. We fixed it, wrote a regression test that fails if it ever comes back, and built the benchmark above — including the context-engine rows, so the claim is measured rather than asserted.

In the library the four fixes are not something you configure. You pass the caller’s identity, and the scan tuning, the GIN index and the iterative scan are applied for you:

from context_engine import ContextEngine

result = await engine.search(
    "what is our refund window",
    principals=["group:support"],   # None = trusted internal; [] = anonymous
    top_k=10,
)

The same principals contract runs through every read path — search, document fetch, structured query — and the predicate is shared by all of them, so a new retrieval leg cannot forget it. Installation, configuration and the full ACL semantics are in the context-engine documentation.

We are writing this up because the mechanism is general. If you run access-controlled retrieval on any approximate vector index, it is worth thirty minutes of your afternoon — whatever you built it with. Run the four-step check above against your own database; the answer is either “no gap” or a number you needed to know.

---

Whether you are exposed is decided per query, by a cost model that moves with dimension, corpus size and statistics freshness. Check your own database:

context-engine check-acl-exposure --database-url postgresql://...

Runs entirely offline against your corpus and your real ACL distribution — no embedding provider is called, nothing is written, and the oracle is computed inside Postgres. Source and harness on GitHub.

---

The context-engine repository is Apache 2.0. More on the design in the documentation, or talk to us about governed retrieval.