Evaluation

Wenlan publishes a compact retrieval snapshot for the shipped hybrid retrieval path. The snapshot uses BGE-Base-EN-v1.5-Q embeddings, FTS5, Reciprocal Rank Fusion, and the local BGE reranker when enabled.

The numbers below should be read as retrieval-only metrics over the stated fixtures. LME_S is the deep LongMemEval substrate; the current representative snapshot uses the stratified N=90 fixture, with 84 gradeable retrieval rows.

Full replay supplies all conversation text as background for a question. Retrieval searches saved material and returns matching passages. It is a step before answer generation, not the AI answer itself. Document retrieval applies the same idea to files and pages; the historical token comparison below used conversation memory.

A token is a small unit of text processed by a model: a word, part of a word, or punctuation. The retrieval harness counts text with cl100k_base. Context-token counts measure supplied background, excluding the generated answer, other instructions, and the costs of indexing or retrieval. They are not a total bill or an answer-quality score.

The README at commit 43d905dceb92b7f6f454443c1097f090c3b61170 reported 4,505 context tokens per query for full replay and 168 for retrieval on LoCoMo. This is a historical published result, separate from the 500-question LME_Oracle retrieval score above.

The original raw token-run artifact and exact run settings were not recovered for this website update, and the test was not rerun. The README preserves the attribution, but does not establish current performance, a CE-reranked 500-question protocol, or a general cost-saving guarantee.

Recall@5 is the fraction of labeled relevant documents found among the first five results, averaged across questions. The 93.6% LME_Oracle result is this retrieval score, not the percentage of correct AI answers. MRR and NDCG@10 describe how well the results are ranked.

This is useful because Wenlan's job is to bring the right memories, pages, decisions, and graph context into the next agent session. It does not prove that a downstream model will always answer correctly.

The published table is not a full product-quality score, a guarantee of answer quality, or a latency benchmark. It is also not a cross-product claim unless the comparison page explicitly states the protocol.

Single-run snapshots are useful for development direction. Public claims that compare improvements should be regenerated under the current schema and, for headline claims, backed by repeated runs or documented methodology.

The eval harness lives in crates/wenlan-core/src/eval. The workflow docs live in docs/eval in the Wenlan repository.

Slow GPU or API-backed evals are manual. Normal CI avoids running heavy model benchmarks because hosted runners do not provide the right hardware, secrets, or cost profile.

Clone the repository, follow the eval docs, and run the appropriate ignored eval harness command for the benchmark you want to reproduce. Expect slow runs when local models or judges are involved.

When comparing two retrieval modes, regenerate both sides under the same schema and fixture revision. Cross-schema comparisons are treated as invalid by the repository's eval discipline.