Evaluation

Reading a ranking metric: NDCG, MRR and the rest

Precision, recall, MRR, average precision and NDCG computed live from a list you can reorder and regrade — including the reorderings that move one metric and leave another completely still.

5 min read evaluationrankingmetrics

Someone asks whether your new ranker is better. You run the evaluation and it comes back: NDCG@10 up 1.2%, MRR flat, recall unchanged.

Knowing what that sentence means — which of those numbers could have moved, which could not, and which one you should have been watching — is most of what ranking evaluation is.

The fastest way to learn it is to break the metrics on purpose.

The list

Ten results for the query “how do I combine two rankers”, each with a graded relevance judgment from an assessor: not relevant, marginal, relevant, perfect.

Reorder the list with the buttons. Click any judgment to change it. Drag the cut-off. Everything recomputes from the list as it actually stands.

This section computes ranking metrics live — it needs JavaScript.

The curve at the top plots all three against every cut-off at once, and the shapes are the lesson on their own. Recall only ever climbs — a bigger window can only find more. Precision usually falls, because the good results are near the front. NDCG does neither reliably, because it is the only one of the three reading position and grade together.

What each one can and cannot see

Precision@k — blind to order

The share of the top that is relevant. It has no idea what order those are in.

Try it: set and hit Reverse. Precision@5 will very likely be unchanged, because reversing the whole list mostly swaps items between the top five and the bottom five — but hit Worst first and it collapses. Precision notices what is in the window, never where in the window.

Recall@k — depends on the pool

The share of everything relevant that you managed to surface. Note the denominator: it counts relevant documents in the judged pool.

That makes recall treacherous to average across queries. A query with two relevant documents and a query with forty are being scored on completely different scales, and the mean of the two means very little. It is also why recall is the natural metric for the retrieval stage — where the job is literally “do not lose the good ones” — and a poor one for the final ranking.

MRR — one result deep

The reciprocal rank of the first relevant result, averaged over queries.

Try it: grade the top result “not relevant” and watch MRR halve. Now change anything at all below position two — MRR does not move at all.

That is not a flaw; it is the metric’s opinion. For known-item search — someone looking for one specific thing they know exists — the rest of the list genuinely does not matter. For a browsing feed it is the wrong metric entirely.

MRR and precision both collapse graded judgments to a binary. If your assessors carefully distinguished “perfect” from “marginal”, these two metrics throw that work away.

NDCG — graded, and position-aware

The one that reads both dimensions. Gain from the grade, discounted by position:

Then normalize by the best ordering possible with those same judgments:

Two design decisions are doing the work.

The exponential gain, , means a “perfect” result is worth 7 while “marginal” is worth 1. Grades are not treated as a linear scale — one excellent result outweighs several mediocre ones.

The logarithmic discount is much gentler than people assume. Position 10 is divided by , so it still contributes about 29% of what position 1 would. If your product only ever shows three results, NDCG@10 is generously crediting you for seven you never displayed.

The normalization is what makes it comparable across queries: a query where the best achievable DCG is small gets scored against that ceiling, not against a query with ten perfect documents in the pool.

Average precision — the compromise

Precision recomputed at every position where a relevant document appears, then averaged. It is order-aware like NDCG and binary like precision — a reasonable default when your judgments are binary anyway.

The experiment worth running

Set so nothing is hidden, then click Ideal order and note every number. Now swap just the top two results by hitting Reverse twice on a shuffled list — or simply regrade to force a swap.

  • Precision@10: identical. Same items in the window.
  • Recall@10: identical. Same items found.
  • MRR: changes only if the swap moved the first relevant result.
  • NDCG@10: changes, slightly.

Four metrics, one change, one number moving. If you are reporting the wrong one, a genuine ranking improvement is invisible — and so is a genuine regression.

What I would actually report

One metric as the decision variable, chosen to match the decision the user is making. Known-item lookup: MRR. Browsing feed with graded judgments: NDCG at the number of results you actually display. Retrieval stage feeding a reranker: recall at the candidate-set size.

The others as guardrails, so you notice when the headline moved for a stupid reason.

A distribution, not a mean. “NDCG@10 improved 1.2%” hides the case where 80% of queries got slightly worse and 20% got dramatically better. Report the mean with the spread, and look at the queries that regressed. That list is usually more informative than the average.