Retrieval precision, recall, relevance, and ranking

Overview

• Precision measures how many retrieved items are relevant.
• Recall measures how much relevant evidence was retrieved.
• Relevance indicates usefulness for the query; ranking metrics also consider whether good evidence appears early enough to be used.

Why this matters

Generation cannot answer from evidence it never receives, and irrelevant evidence adds cost and confusion.

An everyday example

Finding two needed books and eight unrelated books is different from finding only one of four needed books.

A technical example

Relevant IDs are {A,B,C} and top-3 results are {A,X,B}: precision@3 is 2/3, recall@3 is 2/3.

How it works

1. Label eligible relevant chunks/documents
2. Retrieve top-k
3. Compare sets and positions
4. Aggregate by query type
5. Inspect gaps.

Architecture and flow

Query + relevant source labels → retrieval → precision/recall/rank metrics.

The insurance assistant example

Evaluate whether both a coverage clause and its exception appear in the selected evidence.

When to use it

Chunking, embedding, hybrid-search, index, and reranker comparisons.

When to avoid it

Do not calculate recall from incomplete labels without acknowledging the limit or count duplicated overlap chunks as independent relevant evidence.

Interview answer

“Recall tells me whether needed evidence was found; precision and rank tell me how useful the returned context is. I evaluate at a consistent evidence granularity.”

Benefits, trade-offs, and production considerations

• MRR, mean reciprocal rank, rewards the position of the first relevant result. nDCG rewards useful graded results near the top.
• Label source sections or normalized evidence units when chunker changes make exact chunk IDs unstable.

function retrievalMetrics(retrieved: string[], relevant: Set<string>) {
  const unique = [...new Set(retrieved)];
  const hits = unique.filter((id) => relevant.has(id)).length;
  return {
    precision: unique.length ? hits / unique.length : 0,
    recall: relevant.size ? hits / relevant.size : null,
  };
}
// null recall marks an unlabeled/no-relevant-evidence case for separate scoring.

For a query with no answer in the corpus, evaluate correct abstention and irrelevant retrieval separately rather than pretending conventional recall is meaningful.