The LLM fundamentals screen

What do you need to know about models to pass a first technical screen?

A screen checks whether you've really built with these models. You don't need to derive anything. You do need to know how tokens, context, temperature, embeddings, caching and hallucination affect a product, and to do cost sums without a calculator.

A model is a function that predicts the next token. Almost everything else follows from that. It has no memory between calls, so you send the conversation each time. It has a fixed window, so long input pushes something out. It predicts plausible text, so it can be fluently wrong.

  • Token: The unit a model reads and writes: about 4 characters of English on average.
  • Context window: The limit on input and output together, per request.
  • Temperature: How much randomness goes into choosing the next token, on models that expose it. It doesn't make answers true.
  • Embedding: A list of numbers representing meaning. Similar text gets similar numbers.
  • Prompt caching: Reusing an unchanged start of a prompt at a lower price. Any change early on breaks the match.
  • Hallucination: Confident output that's false. It's a consequence of predicting plausible text, so you design around it.

Steps:

  1. Learn the vocabulary. The critical basics page has cards for this.
  2. Practise the cost sum. Fresh input, cached input and output have different rates.
  3. Know the limits. Input plus reply must fit the window, so reserve room for the reply.
  4. Know the failure modes. Hallucination, prompt injection, cut-off output, and drift between model versions.
  5. Build one small thing. Nothing prepares you better. The last path builds one.

If a prompt won't fit: In order: send less irrelevant text, retrieve better chunks, summarise old conversation turns, then consider a larger window. Don't silently cut text off: the model won't know what's missing, and neither will you.

Prices and limits go out of date: Don't memorise a specific model's prices or limits. Learn the sum and look up the numbers. Saying 'I'd check the current price sheet' is the right answer.

Code, Fitting a prompt into the context. Your prompt, the retrieved text and the model's reply all share one limit.

/** About four characters per token in English. Good for planning; use a real counter before sending. */
export const estimateTokens = (text: string) => Math.ceil(text.length / 4);

export interface Chunk {
  id: string;
  text: string;
  score: number;
}

/** Take the best chunks that fit, after saving room for the fixed text and the reply. */
export function fitContext(fixedText: string, chunks: Chunk[], contextLimit: number, replyTokens: number) {
  let left = contextLimit - replyTokens - estimateTokens(fixedText);
  if (left <= 0) throw new Error("the fixed text and reply already fill the context");
  const kept: Chunk[] = [];
  const dropped: string[] = [];
  for (const c of [...chunks].sort((a, b) => b.score - a.score)) {
    const need = estimateTokens(c.text);
    if (need <= left) {
      kept.push(c);
      left -= need;
    } else {
      dropped.push(c.id);
    }
  }
  return { kept, dropped, tokensLeft: left };
}
  • text.length / 4: The usual English rule of thumb. Fine for planning, but use a real token counter before sending.
  • contextLimit - replyTokens - estimateTokens(fixedText): Reserve the reply first. Forgetting this is a classic cause of cut-off answers.
  • dropped.push(c.id): Records what got left out, so a bad answer can be traced back to it.

Code, The cost of a request. The sum every LLM screen expects you to do in your head.

export interface Prices {
  /** dollars per million tokens */
  input: number;
  output: number;
  cachedInput: number;
}

export function requestCost(inputTokens: number, cachedTokens: number, outputTokens: number, p: Prices): number {
  const fresh = inputTokens - cachedTokens;
  return (fresh * p.input + cachedTokens * p.cachedInput + outputTokens * p.output) / 1_000_000;
}

export const monthlyCost = (costPerRequest: number, requestsPerDay: number) => costPerRequest * requestsPerDay * 30;
  • inputTokens - cachedTokens: Cached tokens are billed at their own lower rate, so split them out first.

  • / 1_000_000: Prices are quoted per million tokens.

  • costPerRequest * requestsPerDay * 30: Monthly cost is what makes a design decision real.

  • Failure, Saying temperature 0 makes it accurate: It reduces randomness. A wrong answer stays wrong.

  • Failure, Forgetting the reply in the context limit: The reply gets cut off, and it looks like a model bug.

  • Failure, Quoting stale numbers: Interviewers can tell. Give the method and say you'd check the numbers.

Why does a chatbot seem to remember, if models have no memory?
The application sends the conversation again with each request. That's why long chats get more expensive, and why they eventually hit the limit.
What breaks prompt caching?
Any change early in the prompt, such as a timestamp in the system text. Keep fixed content first, and what varies last.
What's an embedding for?
It turns text into numbers so you can find similar meaning by comparing vectors. That's the basis of semantic search.
How do you reduce hallucination in a product?
Give the model the facts in the prompt, ask it to cite them, check the citations in code, let it say 'I don't know', and measure with evals.
What is prompt injection?
Text in the input that tries to act as instructions, such as a web page telling an agent to send data elsewhere. You limit what tools can do, and treat retrieved text as data.

Sources: