RCP-nDCG@10: A more complete way to measure retrieval relevance
Cohere has introduced RCP‑nDCG@10, a new retrieval evaluation metric that uses a calibrated AI judge to score every retrieved document against a fixed relevance rubric. Unlike traditional nDCG, which relies on pre‑existing relevance judgments that can miss relevant documents, RCP‑nDCG@10 grades all results, filling coverage gaps and better reflecting real‑world search quality. In a 46‑person human study across multiple benchmarks, RCP‑nDCG@10 matched reviewer preferences 77% of the time versus 52% for conventional nDCG, with higher agreement when margin differences were larger.
The Core Development
How well does nDCG capture the quality of modern retrieval systems? nDCG is the established yardstick for measuring how well search systems rank results, and is widely used across benchmarks such as MTEB and BEIR . It is preferred because it reflects both relevance and position, giving more weight to useful results that appear near the top. However, it relies on pre-existing relevance labels, which, if incomplete, mean other relevant results can be missed or scored incorrectly.
To address this, we are introducing Rubric-Calibrated Preferences nDCG@10 (RCP-nDCG@10), an evaluation methodology that grades each retrieved document against the same explicit relevance criteria for every query, using a calibrated AI judge. Designed to better reflect real-world enterprise retrieval quality, RCP-nDCG@10 is the metric against which we have optimized our next generation search models. Here, we explain how it works, what's different from the status quo, and how we validated it against human judgment.
In enterprise search, model performance is typically compared using a portfolio of established offline retrieval metrics. The most common are Recall@k, Mean Reciprocal Rank (MRR), and Normalized Discounted Cumulative Gain (nDCG@k). Each captures a different aspect of retrieval quality.
Real-World Use Cases & Applications
To test whether RCP-nDCG reflects real-world relevance, we ran a blind human study. 46 contracted annotators, each with at least a BSc in the relevant field, graded the top-five results of competing systems on queries from NanoBEIR , BRIGHT and ViDoRe v3, without seeing system names, rankings, or the answer key. Three annotators reviewed every head-to-head contest, with the majority deciding the winner; 289 contests across 273 queries reached a verdict.
Strategic Perspective & Ecosystem Impact
Recall@10 measures how much relevant material appears within the top ten results. It is useful when missing information is costly, but it pays no attention to ordering: a relevant result at position one is the same as one at position ten. MRR, by contrast, measures how high the first relevant result appears, but ignores everything that follows. It is therefore most useful when a single correct result is enough, such as FAQ search or simple question answering.
nDCG became widely adopted because it improved on simpler metrics by accounting for both degree of relevance and ranking position. More relevant documents contribute more to the score, and those near the top are weighted more heavily. Over time, this made nDCG a standard across search benchmarks and leaderboards.
Because each metric rewards something different, the three can reach different verdicts on the same results (Figure 1).
Why it matters
By providing a more complete and realistic assessment of retrieval relevance, RCP‑nDCG@10 can guide the development of search models that better meet user needs, potentially improving user satisfaction and trust in AI‑powered search systems.