BGE-Reranker Multi-Stage Retrieval Pipeline

Retrieve 100 with fast search, rerank to 5 with a cross-encoder, and degrade safely on timeout.

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

Fast retrieval casts a wide net but ranks roughly. A cross-encoder reads the question and each candidate together and ranks far better, but too slowly to run on everything. So retrieve many, then rerank a few.

Why it matters

Retrieve 100 with fast search, rerank to 5 with a cross-encoder, and degrade safely on timeout.

1.2Key concepts

One idea per card, in plain words.

01Two-stage retrieval

Recall comes from the first stage, precision from the second.

02Cross-encoder

Scores one question-passage pair per forward pass. Accurate, but costs scale with candidates.

03Latency budget

Bounds how long the reranker may run before the request moves on.

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

Recall@100 reaches 95%, but precision@5 sits at 40%. Add a reranker within 150 ms at p95, and never fail a request because of it.

Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Step by step

The mechanism, in the order you would explain it.

  1. Retrieve widely

    Take the top 100 from hybrid search.

  2. Score in batches

    Batch question-passage pairs on a GPU, and truncate passages to the model's limit.

  3. Enforce the budget

    Race the reranker against a timeout.

  4. Fall back safely

    On timeout or error, keep the retrieval order and record that the rerank failed.

  5. Measure the lift

    Compare precision@5 with and without reranking on your eval set.

3.2Code

Working code for the core of it.

Retrieve widely, rerank precisely

A cross-encoder is accurate but slow, so it only sees the top candidates, and it's allowed to fail.

rerank-pipeline.tsTypeScriptCopy
1export interface Candidate {2  id: string;3  text: string;4}5 6interface Options {7  candidates?: number;8  top?: number;9  timeoutMs?: number;10}11 12/** Retrieve widely, rerank precisely. If the reranker fails or runs late, keep the retrieval order. */13export async function retrieveAndRerank(14  query: string,15  retrieve: (q: string, n: number) => Promise<Candidate[]>,16  rerank: (q: string, docs: Candidate[]) => Promise<number[]>,17  { candidates = 100, top = 5, timeoutMs = 300 }: Options = {}18): Promise<{ results: Candidate[]; reranked: boolean }> {19  const docs = await retrieve(query, candidates);20  const timeout = new Promise<never>((_, reject) => setTimeout(() => reject(new Error("rerank timeout")), timeoutMs));21  try {22    const scores = await Promise.race([rerank(query, docs), timeout]);23    const order = docs.map((d, i) => ({ d, s: scores[i] })).sort((a, b) => b.s - a.s);24    return { results: order.slice(0, top).map((x) => x.d), reranked: true };25  } catch {26    return { results: docs.slice(0, top), reranked: false };27  }28}
28 linesUTF-8TypeScript
candidates = 100
Cast the net wide first, because reranking can't recover what retrieval missed.
Promise.race([rerank(...), timeout])
Puts a limit on latency.
reranked: false
On failure, fall back to retrieval order and say so, rather than failing the request.

Tested by npm run test:modules.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

How many candidates?
Enough to hold the answer, few enough to fit the budget. Plot recall@n, then pick the smallest n near its plateau.
Why not rerank everything?
Cost grows linearly with candidates, and each one needs a forward pass.
Should scores act as thresholds?
Be careful: raw scores vary by query. Sigmoid-normalised scores help but still need calibration.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
Silent degradation
Alert on the fallback rate. A permanently failing reranker looks like a small quality drop.
Counterpoint
If retrieval already places answers in the top 5, a reranker adds cost and latency for little gain.