Redis Vector Semantic Response Caching

Answers repeat questions from a vector cache, cutting LLM cost, as long as false hits stay rare.

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

Users ask the same things in different words. A semantic cache embeds each question, searches earlier questions, and returns the stored answer if one sits close enough. A hit costs milliseconds instead of an LLM call. A false hit serves a wrong answer, so the threshold decides everything.

Why it matters

Answers repeat questions from a vector cache, cutting LLM cost, as long as false hits stay rare.

1.2Key concepts

One idea per card, in plain words.

01Similarity threshold

Sets how close a cached question must be. Too low serves wrong answers; too high never hits.

02False hit

Treats two different questions as the same: 'cancel my order' against 'cancel my subscription'.

03Scope

Limits hits to answers this user may see, usually by tenant and sometimes by role.

04TTL

Expires entries so stale answers leave the cache.

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

Your support assistant handles 2 million LLM calls a day at $0.004 each, and you suspect many repeat. Design a cache that cuts cost without serving wrong answers.

2.2Clarify first

Each answer removes a guess.

You askAnswer
Are answers personalised?Some use account data. Those must bypass the cache.
How fresh must they be?Policy answers can age for an hour; order status cannot be cached.
What repeat rate?Measure first by replaying a day of logs against the cache offline.
Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Architecture

The components and how they connect.

Assistant APIclassifies the requestEmbeddersmall, fast modelRedis vector indexHNSW · tenant tag · TTLLLMonly on a missShadow evalsamples hits for reviewContent changesinvalidates entriesembed questionsearch above thresholdmissstore answersampled hitspurge by source
ClientServiceData storeAI componentExternal
Boxes are components; arrows show calls and data flow.

3.2Step by step

The mechanism, in the order you would explain it.

  1. Decide what to cache

    Classify first. Cache general questions; bypass anything that uses personal data or live state.

  2. Embed and search

    Embed the question and search within the tenant's tag. This adds 5–15 ms to every request.

  3. Serve or call

    Above the threshold, return the stored answer. Below it, call the LLM and store the result with a TTL.

  4. Set the threshold from data

    Label a few hundred question pairs, plot false hits against hit rate, and pick the point you can defend.

  5. Invalidate by source

    Tag entries with the documents behind them, and purge when a document changes.

  6. Shadow-check hits

    Sample hits and also call the LLM. Compare the answers to catch threshold drift.

3.3Savings at a 30% hit rate

ItemPer day
LLM calls avoided600,000
LLM cost saved$2,400
Added embeddings2 million small calls, a few dollars

3.4Code

Working code for the core of it.

A cache keyed by meaning

Similar questions can share an answer, if you're careful about who shares with whom.

semantic-cache.tsTypeScriptCopy
1const cosine = (a: number[], b: number[]) => {2  const dot = a.reduce((s, x, i) => s + x * b[i], 0);3  return dot / (Math.hypot(...a) * Math.hypot(...b));4};5 6interface Entry<V> {7  vector: number[];8  value: V;9  expires: number;10}11 12/** Answers a question from a previous similar question. Tenants never share entries. */13export class SemanticCache<V> {14  private entries = new Map<string, Entry<V>[]>();15  private embed: (text: string) => Promise<number[]>;16  private threshold: number;17  private ttlMs: number;18  private now: () => number;19 20  constructor(embed: (text: string) => Promise<number[]>, opts: { threshold?: number; ttlMs?: number; now?: () => number } = {}) {21    this.embed = embed;22    this.threshold = opts.threshold ?? 0.92;23    this.ttlMs = opts.ttlMs ?? 3_600_000;24    this.now = opts.now ?? Date.now;25  }26 27  async get(tenant: string, query: string): Promise<V | null> {28    const v = await this.embed(query);29    const live = (this.entries.get(tenant) ?? []).filter((e) => e.expires > this.now());30    this.entries.set(tenant, live);31    let best: Entry<V> | null = null;32    let bestScore = this.threshold;33    for (const e of live) {34      const score = cosine(v, e.vector);35      if (score >= bestScore) {36        best = e;37        bestScore = score;38      }39    }40    return best?.value ?? null;41  }42 43  async put(tenant: string, query: string, value: V): Promise<void> {44    const entry = { vector: await this.embed(query), value, expires: this.now() + this.ttlMs };45    this.entries.set(tenant, [...(this.entries.get(tenant) ?? []), entry]);46  }47}
47 linesUTF-8TypeScript
threshold ?? 0.92
Set this too low, and one question gets the answer to a different one. Tune it on real data.
this.entries.get(tenant)
Tenants never share entries, because an answer may contain private data.
e.expires > this.now()
Answers go stale, so everything has a time limit.

Tested by npm run test:modules.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

How do you handle negation?
Embeddings place 'can I cancel' and 'can't I cancel' close together. A higher threshold, or a reranker on hits, reduces this, and sensitive intents should bypass the cache.
Where can it go wrong for privacy?
A shared cache leaks one user's answer to another. Scope entries by tenant and never store personalised replies.
How do you measure success?
Track hit rate, false-hit rate from shadow checks, latency, and net savings after embedding and Redis costs.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
Prompt changes
A new prompt version changes answers. Add the version to the cache scope.
Streaming
Replay cached answers at a steady pace, so hits and misses look the same in the UI.
Counterpoint
Below about a 10% repeat rate the cache adds cost, latency and risk for little return. An exact-match cache may be enough.