CLIP and ViT Multimodal Embeddings

CLIP places images and text in one vector space, which enables image search with no labels.

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

A Vision Transformer cuts an image into patches and treats them like words. CLIP trains an image encoder and a text encoder together until matching pairs land close in one vector space. Then a text query can find images directly.

Why it matters

CLIP places images and text in one vector space, which enables image search with no labels.

1.2Key concepts

One idea per card, in plain words.

01Patch embedding

Cuts an image into squares of, say, 16 × 16 pixels, and projects each into a token.

02Contrastive loss

Pulls each image towards its caption and pushes it away from every other caption in the batch.

03Temperature

Sharpens or softens the similarity scores.

04Joint embedding space

Compares image and text vectors with cosine similarity.

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

A marketplace wants 'red running shoes' to find matching photos, with no tagged images. Explain the model behind it, and design the search.

Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Step by step

The mechanism, in the order you would explain it.

  1. Patch the image

    A 224 × 224 image with 16-pixel patches gives 196 tokens.

  2. Encode both sides

    A ViT encodes images; a text transformer encodes captions.

  3. Normalise and compare

    Cosine similarity across a batch gives an N × N matrix, where the diagonal holds true pairs.

  4. Train symmetrically

    Apply cross-entropy across rows and columns.

  5. Build the index

    Embed every photo offline and store vectors in HNSW. Embed the query online.

3.2Code

Working code for the core of it.

CLIP's training loss

It teaches images and captions to share one vector space.

clip-loss.tsTypeScriptCopy
1const normalise = (v: number[]) => {2  const n = Math.hypot(...v);3  return v.map((x) => x / n);4};5 6/** CLIP's symmetric InfoNCE: each image should match its own caption, and each caption its image. */7export function clipLoss(images: number[][], texts: number[][], temperature = 0.07): number {8  const I = images.map(normalise);9  const T = texts.map(normalise);10  const n = I.length;11  const logits = I.map((a) => T.map((b) => a.reduce((s, x, i) => s + x * b[i], 0) / temperature));12  const crossEntropy = (m: number[][]) =>13    m.reduce((sum, row, i) => {14      const denom = row.reduce((s, x) => s + Math.exp(x), 0);15      return sum - Math.log(Math.exp(row[i]) / denom);16    }, 0) / n;17  const transpose = logits[0].map((_, j) => logits.map((row) => row[j]));18  return (crossEntropy(logits) + crossEntropy(transpose)) / 2;19}
19 linesUTF-8TypeScript
normalise
Compares direction only, not length.
temperature
Sharpens the distribution, and 0.07 is CLIP's starting value.
crossEntropy(transpose)
Symmetric: each image finds its caption, and each caption its image.

Tested by npm run test:modules.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

Why do batch sizes matter?
Each other item in a batch acts as a negative example. Bigger batches give harder training.
How do audio and video fit?
Audio becomes spectrogram patches or learned tokens. Video usually samples frames and pools their embeddings.
What fails first in practice?
Fine details and counting. Shoppers' product attributes often need a fine-tuned model or metadata filters.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
Domain gap
CLIP trained on web images may misread medical or technical images. Evaluate on your own data.
Counterpoint
If you have good tags, keyword search stays cheaper and easier to explain. Combine both.