Reranking

Retrieval is optimised for speed across millions of documents, which costs precision. A reranker takes the top few dozen candidates and scores them carefully.

It is usually the cheapest single accuracy improvement available to a RAG system.

The cost is latency and tokens: a reranker adds a model call between retrieval and generation. For most enterprise systems that trade is clearly worth it, because a wrong answer costs more than 200 milliseconds does. For high-volume consumer search it may not be, and that is a decision to make explicitly rather than by omission.

Where it pays back most visibly is a corpus with many near-duplicate documents, policy versions, revised contracts, repeated product descriptions. First-pass retrieval treats them as equally relevant; a reranker is what distinguishes the current version from the superseded one.

There is a useful middle setting most teams miss: rerank only when the first-pass scores are close together. When retrieval is already confident, the extra call adds latency and changes nothing, and skipping it there recovers most of the cost.