Reranking

The distinction is bi-encoder versus cross-encoder. First-stage retrieval embeds documents in advance and the query at request time, then compares vectors — fast and precomputable, but the document was encoded with no knowledge of the query. A cross-encoder reranker feeds query and passage through a model together, so it can weigh whether this specific passage answers this specific question. That is far more expensive per pair, which is exactly why it runs on a shortlist rather than the corpus.

The standard arrangement is therefore two-stage: retrieve generously — enough candidates that the right passage is very likely somewhere in the set — then rerank and keep only the top few for the prompt. This directly improves the thing that determines answer quality, which is not whether the right passage was retrieved at all but whether it survived into the limited context the model actually reads.

The misconception is that reranking rescues weak retrieval. It can only reorder what it was given; a passage missing from the candidate set cannot be promoted into it. If recall at the first stage is poor, the fix is upstream — better chunking, hybrid scoring, query expansion — and reranking is what you add once the right answer is reliably present but arriving in eleventh place.