Why can post-filtering by tenant return no results for small tenants?

Top-k is computed over all tenants; if other tenants' chunks fill the top k, filtering afterwards removes them all and the small tenant's relevant chunks were never retrieved.

How do you make a missing tenant filter fail closed?

Enforce isolation below the application: separate indexes or namespaces per tenant, or row-level security in the database, so a query without the right context cannot see other tenants' data.

What does document-level permission filtering require at indexing time?

Storing each chunk's access list (groups, roles, owners) as metadata, keeping it in sync when permissions change, and filtering by the current user's groups at query time.

Filters, Permissions and Tenant Isolation — RAG in Production

Scenario

Example company: a B2B platform adds an assistant that answers from each customer's own uploaded documents. All customers' chunks share one vector index; results are filtered by tenant_id after the top 5 are retrieved.

What you see in production

Two bugs appear in the same week. Small customers get "I couldn't find anything" even for questions their own documents answer. Worse, a code path that forgot the post-filter showed a customer a snippet from another company's contract.

The naive code

hits = index.query(embed(question), top_k=5)                    # searches every tenant's documents
hits = [h for h in hits if h.metadata["tenant_id"] == tenant_id]   # filter afterwards (when remembered)

Reproduce it

One large tenant dominates the index; a small tenant's question is filtered after and before retrieval:

Post-filtering loses results; pre-filtering searches only allowed documentsTry it yourself

index = [{"tenant": "big-co", "text": f"big-co policy document {i} about refunds", "score": 0.90 - i * 0.001}
         for i in range(200)]
index += [{"tenant": "small-co", "text": "small-co refund policy: 30 days", "score": 0.82}]

def post_filter(tenant, k=5):
    top = sorted(index, key=lambda c: -c["score"])[:k]
    return [c["text"] for c in top if c["tenant"] == tenant]

def pre_filter(tenant, k=5):
    allowed = [c for c in index if c["tenant"] == tenant]           # the filter is part of the query
    return [c["text"] for c in sorted(allowed, key=lambda c: -c["score"])[:k]]

print("post-filter, small-co:", post_filter("small-co"))
print("pre-filter,  small-co:", pre_filter("small-co"))
print("forgot the filter:", [c["text"] for c in sorted(index, key=lambda c: -c["score"])[:1]])

Output

post-filter, small-co: []
pre-filter,  small-co: ['small-co refund policy: 30 days']
forgot the filter: ['big-co policy document 0 about refunds']

Root cause

Top-k is computed over everything the index contains. If most of the index belongs to other tenants, a small tenant's documents rarely make the top k, and filtering afterwards leaves nothing. And because the filter is an extra step in application code rather than part of the query, any code path that forgets it leaks data across tenants.

The fix

Filter inside the search: use the vector store's metadata filters (or a WHERE tenant_id = :tenant clause with pgvector) so candidates come only from allowed documents.
Enforce isolation below the application: separate indexes or namespaces per tenant, or row-level security in PostgreSQL, so a missing filter fails closed.
Document-level permissions: store access lists (groups, roles) on chunks at indexing time, sync them when permissions change, and filter by the current user's groups.
Check with your filter in place that the vector index still returns enough candidates (some approximate indexes need settings for filtered queries).

Trade-offs

Per-tenant indexes isolate strongly but are harder to operate with many small tenants; shared indexes need strict, tested filters.
Permission sync adds work at indexing time and a delay before permission changes take effect; define and monitor that delay.
Heavy filtering with approximate nearest-neighbour indexes can reduce recall; test filtered queries specifically.

Detect it in production

Automated test: a query as tenant A must never return tenant B's chunks, run on every deploy.
Rate of "no results" answers per tenant, especially for small tenants.
Audit log of retrieved chunk IDs per answer, to investigate suspected leaks.

Key points

Filtering after top-k loses results and is easy to forget.
Put tenant and permission filters in the search itself, and back them with isolation below the app.
Test cross-tenant isolation automatically.

Further reading: OWASP LLM08: Vector and embedding weaknesses · pgvector: filtering · PostgreSQL row security policies