Stale Answers: Keeping the Index Fresh

Scenario

Example company: a help centre with 40,000 articles. The RAG index is rebuilt every night by re-embedding everything, which takes five hours.

What you see in production

The assistant still explains a feature that was removed last month: the article was deleted, but its chunks were never removed from the index because the nightly job only adds and overwrites. Edits made in the morning are not visible until the next day, and the embedding bill is dominated by re-embedding articles that did not change.

The naive code

for article in cms.all_articles():                   # 40,000 articles, every night
    for chunk in chunk_article(article):
        index.upsert(id=f"{article.id}-{chunk.n}", vector=embed(chunk.text), text=chunk.text)
# deleted articles and removed chunks stay in the index forever

Reproduce it

Sync a small index with content hashes: embed only what changed and delete what disappeared:

import hashlib

def h(text):
    return hashlib.sha256(text.encode()).hexdigest()[:10]

index = {  # chunk id -> content hash
    "a1-0": h("Export to CSV from Reports."), "a1-1": h("Exports run hourly."),
    "a2-0": h("Legacy dashboards: how to enable."),          # article a2 was deleted from the CMS
}
source = {"a1": ["Export to CSV or XLSX from Reports.", "Exports run hourly."],
          "a3": ["New: scheduled reports by email."]}

wanted = {f"{doc}-{i}": h(text) for doc, chunks in source.items() for i, text in enumerate(chunks)}
to_embed = [cid for cid, digest in wanted.items() if index.get(cid) != digest]
to_delete = [cid for cid in index if cid not in wanted]
print("embed:", to_embed)
print("delete:", to_delete)
print("unchanged:", [cid for cid in wanted if cid not in to_embed])

embed: ['a1-0', 'a3-0']
delete: ['a2-0']
unchanged: ['a1-1']

Root cause

A full rebuild that only upserts cannot notice deletions, and re-embedding everything makes the job too slow to run often. The index drifts from the source of truth in both directions: stale content stays, new content arrives late.

The fix

• Incremental sync: trigger re-indexing on change events (webhooks from the CMS) or poll with updated_since.
• Content hashes per chunk: re-embed only chunks whose text changed; skip the rest.
• Delete by document: when a document changes, replace all its chunks (delete by doc_id, then insert), and remove documents that no longer exist.
• Store source metadata: document ID, version, updated_at and URL on every chunk; show the date in answers where it matters.
• Run a periodic full reconciliation (weekly) comparing IDs in the source and the index, to catch missed events.
• Re-embed everything deliberately when you change the embedding model, into a new index, then switch over.

Trade-offs

• Event-driven indexing needs reliable delivery; keep the reconciliation job as a safety net.
• Replacing all of a document's chunks on any edit is simple but re-embeds more than needed; chunk-level hashes avoid that.
• Showing dates builds trust but reveals when content is old; that is usually a feature.

Detect it in production

• Index freshness: time between a source change and the index update (p95).
• Count of index documents that no longer exist in the source (should be zero after reconciliation).
• Embedding tokens per day, compared with the volume of content changes.

Key points

• Full rebuilds that only upsert never delete, and are too slow to run often.
• Sync incrementally with content hashes, and delete chunks of removed or changed documents.
• Reconcile periodically and re-embed into a new index when the embedding model changes.

Further reading: OWASP LLM08: Vector and embedding weaknesses · pgvector