Measuring Retrieval Quality Without Vanity Metrics

Similarity is not usefulness

Cosine similarity between query and chunk tells you the index retrieved something nearby in vector space. It does not tell you whether the chunk contains the fact needed to complete the user task, whether citations are faithful, or whether the system should have retrieved nothing. Optimizing for similarity alone rewards fluent irrelevance.

Staff-level measurement starts from product promises. If the feature claims grounded answers from your corpus, the metric is correct grounding under realistic queries—not how pretty the nearest neighbors look in a notebook.

Build an evaluation set that can fail you

Label query sets with expected document IDs or answer properties, including adversarial cases: synonyms, outdated entities, partial identifiers, multilingual phrasing, and questions whose answer is absent. Track recall@k for must-find docs, precision of citations, and abstention accuracy when nothing should match.

Refresh the set as the product changes. A frozen golden set from launch week becomes vanity once content and user language drift. Version the dataset with the index and chunking strategy so regressions are attributable.

• Report metrics by slice: tenant, language, content type, and risk tier
• Include 'should retrieve nothing' cases in every release gate
• Pair offline scores with human review on production samples
• Separate retrieval failures from generation failures in incident review

Online metrics must connect to outcomes

Instrument whether users complete the job after an AI answer, how often they regenerate, escalate to search or support, or correct citations. A high thumbs-up rate on stylistic answers can coexist with rising ticket volume when facts are wrong. Prefer outcome-linked rates over isolated engagement vanity.

When you change chunking, embeddings, or rerankers, run paired experiments with clear success criteria. Guard against regressions in latency and cost per successful grounded answer—cheaper retrieval that forces five regenerations is not cheaper.

Operate retrieval like a production dependency

Index freshness, permission filters, and empty-result rates belong on the same dashboard as API availability. A silent drop in recall after a crawler or ACL change is a product incident, even if the model still produces confident prose.

Vanity metrics make teams feel busy. Useful metrics make teams decide: ship, roll back, or invest in content and chunking. That decision quality is the point of measuring retrieval at all.

• NIST — AI RMF
• OpenAI — Evaluation best practices
• Anthropic — Measuring faithfulness