Retrieval regression

How to Regression-Test AI Knowledge Base Retrieval After Changes

Use a versioned golden query set to find RAG retrieval regressions after corpus, embedding, chunking, hybrid search, or reranker changes.

Qi-Xuan LuUpdated Aug 20268 min read

Article packet

Workflows

Developers changing an AI knowledge base corpus, embedding model, chunking, hybrid retrieval, or reranker

8 min read

Freeze representative questions and their expected sources before changing retrieval.

Change one factor, then compare retrieval before evaluating answer generation.

Inspect every regression and keep rollback available until the new baseline is justified.

Quick answer

To regression-test an AI knowledge base, version a small golden query set with expected source IDs or documents, no-answer cases, and a baseline manifest. Run the same set before and after one corpus or retrieval change, compare retrieval-only results first, inspect every lost or newly introduced source, and accept the change only when the differences are understood.

This is different from citation verification. Citation verification starts with one produced answer and checks whether its evidence supports each claim. Retrieval regression starts before generation and asks whether the same questions still retrieve the intended evidence after the system changes.

Run the test whenever retrieval inputs change

Re-run the suite after adding, deleting, or revising source documents; changing embeddings or chunk size; adjusting lexical-vector fusion; replacing a reranker; or changing filters and metadata. A fluent demo query is not enough because a change can improve one topic while silently removing evidence for another.

Include frequent real questions, known failures, edge cases, and questions the corpus should not answer.
Keep the authoritative source revision beside each expected result so an intentional corpus change can invalidate the old label explicitly.
Do not refresh the golden set merely because the new system disagrees with it; investigate the disagreement first.

Define a versioned golden query set

Each case needs a stable ID, the natural user question, expected source IDs or documents, optional excluded sources, whether no answer is valid, and the reason the case matters. Store the corpus revision and retrieval configuration separately so two runs remain comparable.

Minimal product-neutral golden-set record

version: 1
corpus_revision: docs-2026-08-26
cases:
  - id: install-windows
    query: Which Windows package includes the desktop app?
    expected_sources: [release-v0.16.0]
    excluded_sources: [runtime-zip]
    no_answer: false
  - id: unsupported-pricing
    query: What is the enterprise price?
    expected_sources: []
    no_answer: true

Freeze the baseline, then change one factor

Record the corpus revision, embedding model, chunker and parameters, filters, hybrid weights, reranker version, top-k, and runtime environment. Change one factor at a time when possible. Otherwise the result may reveal drift without showing which change caused it.

Compare source-level Recall@k or Hit@k first, then MRR or NDCG when rank order matters. Keep no-answer cases and latency visible, but do not combine every measure into one score that hides a lost critical source.

Inspect regressions before judging generated answers

For every failed case, inspect the rewritten query, retrieved chunks, scores, source IDs, filters, fusion result, reranker output, and final ordering. Separate missing documents, wrong labels, extraction defects, metadata filters, embedding drift, chunk-boundary loss, and reranker changes. A bad expected-source label can make the test wrong, while a good aggregate score can still hide one dangerous regression.

Only after the expected evidence is available should you evaluate grounding or answer quality. Retrieval success does not prove that a model will use the evidence correctly; citation verification remains a separate downstream check.

Use Wenlan's maintainer drift test honestly

Wenlan's repository maintains labeled retrieval fixtures, retrieval-only Recall@5, MRR, and NDCG@10 snapshots, frozen ranking goldens, and an ignored ranking-drift test used by the main canary. The test detects drift, not correctness: it compares the current ranking with a trusted reference.

This is a Wenlan maintainer workflow, not a released `wenlan eval` end-user command or hosted CI feature. Users can still apply the product-neutral golden-set method above to any knowledge base and keep the exact source and rollback decision in their own repository.

Wenlan repository maintainer-only drift check

cargo test -p wenlan-core --lib \
  eval::retrieval_drift::tests::ranking_drift_vs_golden \
  -- --ignored --nocapture

Freeze one retrieval baseline

Choose representative project questions, record their expected sources, and keep the baseline beside the source revisions before changing retrieval.

Get started View on GitHub

FAQ

How large should a RAG golden dataset be?+

Start with enough representative questions to cover important sources, known failures, and no-answer behavior. A small reviewed set is better than a large unverified one; expand it from real failures and record why each case belongs.

Should I update the golden set after the corpus changes?+

Only when the source contract intentionally changed. Review the old expectation against the new authoritative source, record the reason, and version the update instead of silently blessing the new result.

Does a passing retrieval regression test prove answer quality?+

No. It proves that the tested retrieval behavior stayed within its declared contract. Generation, citation faithfulness, and source correctness require separate checks.