Selective Context Window Token Compression

Spend the token budget on evidence: prune sentences and summarise history before they reach the model.

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

The context window holds system prompt, history, retrieved text and the reply. Everything competes, and every token costs money and latency. Compression keeps what the answer needs and drops the rest.

Why it matters

Spend the token budget on evidence: prune sentences and summarise history before they reach the model.

1.2Key concepts

One idea per card, in plain words.

01Token budget

Divides the window between system prompt, history, evidence and reply.

02Extractive pruning

Keeps the most relevant original sentences. Nothing gets rewritten, so citations survive.

03Abstractive summary

Rewrites text shorter. Saves more, but can introduce errors.

04History rollup

Replaces old chat turns with a running summary.

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

Your assistant sends 12,000 tokens per request and costs too much. Answers cite only a few sentences. Cut the prompt in half without hurting answer quality.

Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Step by step

The mechanism, in the order you would explain it.

  1. Measure the split

    Log tokens per prompt part. History often outweighs evidence.

  2. Cap each part

    Give system, history, evidence and reply explicit limits.

  3. Rerank first

    Fewer, better passages save more than clever compression.

  4. Prune sentences

    Score sentences against the question, keep the best within budget, and restore reading order.

  5. Roll up history

    Summarise old turns, and keep the last few verbatim.

  6. Verify on evals

    Compare faithfulness and answer quality before and after.

3.2Code

Working code for the core of it.

Fitting context into a budget

Retrieval usually returns more than fits. This keeps the best sentences, in the order they were written.

context-budget.tsTypeScriptCopy
1export interface Sentence {2  text: string;3  score: number;4}5 6/** Keeps the highest-scoring sentences that fit the budget, then restores reading order. */7export function fitContext(sentences: Sentence[], budget: number, count: (s: string) => number): string[] {8  const ranked = sentences.map((s, i) => ({ ...s, i })).sort((a, b) => b.score - a.score);9  const kept: { text: string; i: number }[] = [];10  let used = 0;11  for (const s of ranked) {12    const cost = count(s.text);13    if (used + cost > budget) continue;14    used += cost;15    kept.push(s);16  }17  return kept.sort((a, b) => a.i - b.i).map((s) => s.text);18}
18 linesUTF-8TypeScript
sort((a, b) => b.score - a.score)
Picks by relevance.
continue
Skips one that doesn't fit and tries smaller ones, instead of stopping.
sort((a, b) => a.i - b.i)
Restores reading order, because shuffled sentences confuse a model.

Tested by npm run test:modules.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

Why restore reading order?
Sentences often depend on what came before. Shuffled evidence confuses the model.
Does prompt caching change this?
Yes. Cached static prefixes cost little, so compress the changing parts first.
What about model-based compression?
LLMLingua drops low-information tokens using a small model. It compresses hard, but the text turns hard to read and audit.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
Lost conditions
Pruning can drop an 'unless' clause. Keep a sentence's neighbours when it contains conditions.
Counterpoint
Windows keep growing and prices keep falling. Don't compress before you've measured what it saves.