A screen checks whether you've really built with these models. You don't need to derive anything. You do need to know how tokens, context, temperature, embeddings, caching and hallucination affect a product, and to do cost sums without a calculator.
A model is a function that predicts the next token. Almost everything else follows from that. It has no memory between calls, so you send the conversation each time. It has a fixed window, so long input pushes something out. It predicts plausible text, so it can be fluently wrong.
- Token: The unit a model reads and writes: about 4 characters of English on average.
- Context window: The limit on input and output together, per request.
- Temperature: How much randomness goes into choosing the next token, on models that expose it. It doesn't make answers true.
- Embedding: A list of numbers representing meaning. Similar text gets similar numbers.
- Prompt caching: Reusing an unchanged start of a prompt at a lower price. Any change early on breaks the match.
- Hallucination: Confident output that's false. It's a consequence of predicting plausible text, so you design around it.
Steps:
- Learn the vocabulary. The critical basics page has cards for this.
- Practise the cost sum. Fresh input, cached input and output have different rates.
- Know the limits. Input plus reply must fit the window, so reserve room for the reply.
- Know the failure modes. Hallucination, prompt injection, cut-off output, and drift between model versions.
- Build one small thing. Nothing prepares you better. The last path builds one.
If a prompt won't fit: In order: send less irrelevant text, retrieve better chunks, summarise old conversation turns, then consider a larger window. Don't silently cut text off: the model won't know what's missing, and neither will you.
Prices and limits go out of date: Don't memorise a specific model's prices or limits. Learn the sum and look up the numbers. Saying 'I'd check the current price sheet' is the right answer.
Code, Fitting a prompt into the context. Your prompt, the retrieved text and the model's reply all share one limit.
/** About four characters per token in English. Good for planning; use a real counter before sending. */
export const estimateTokens = (text: string) => Math.ceil(text.length / 4);
export interface Chunk {
id: string;
text: string;
score: number;
}
/** Take the best chunks that fit, after saving room for the fixed text and the reply. */
export function fitContext(fixedText: string, chunks: Chunk[], contextLimit: number, replyTokens: number) {
let left = contextLimit - replyTokens - estimateTokens(fixedText);
if (left <= 0) throw new Error("the fixed text and reply already fill the context");
const kept: Chunk[] = [];
const dropped: string[] = [];
for (const c of [...chunks].sort((a, b) => b.score - a.score)) {
const need = estimateTokens(c.text);
if (need <= left) {
kept.push(c);
left -= need;
} else {
dropped.push(c.id);
}
}
return { kept, dropped, tokensLeft: left };
}
text.length / 4: The usual English rule of thumb. Fine for planning, but use a real token counter before sending.contextLimit - replyTokens - estimateTokens(fixedText): Reserve the reply first. Forgetting this is a classic cause of cut-off answers.dropped.push(c.id): Records what got left out, so a bad answer can be traced back to it.
Code, The cost of a request. The sum every LLM screen expects you to do in your head.
export interface Prices {
/** dollars per million tokens */
input: number;
output: number;
cachedInput: number;
}
export function requestCost(inputTokens: number, cachedTokens: number, outputTokens: number, p: Prices): number {
const fresh = inputTokens - cachedTokens;
return (fresh * p.input + cachedTokens * p.cachedInput + outputTokens * p.output) / 1_000_000;
}
export const monthlyCost = (costPerRequest: number, requestsPerDay: number) => costPerRequest * requestsPerDay * 30;
inputTokens - cachedTokens: Cached tokens are billed at their own lower rate, so split them out first./ 1_000_000: Prices are quoted per million tokens.costPerRequest * requestsPerDay * 30: Monthly cost is what makes a design decision real.Failure, Saying temperature 0 makes it accurate: It reduces randomness. A wrong answer stays wrong.
Failure, Forgetting the reply in the context limit: The reply gets cut off, and it looks like a model bug.
Failure, Quoting stale numbers: Interviewers can tell. Give the method and say you'd check the numbers.
- Why does a chatbot seem to remember, if models have no memory?
- The application sends the conversation again with each request. That's why long chats get more expensive, and why they eventually hit the limit.
- What breaks prompt caching?
- Any change early in the prompt, such as a timestamp in the system text. Keep fixed content first, and what varies last.
- What's an embedding for?
- It turns text into numbers so you can find similar meaning by comparing vectors. That's the basis of semantic search.
- How do you reduce hallucination in a product?
- Give the model the facts in the prompt, ask it to cite them, check the citations in code, let it say 'I don't know', and measure with evals.
- What is prompt injection?
- Text in the input that tries to act as instructions, such as a web page telling an agent to send data elsewhere. You limit what tools can do, and treat retrieved text as data.
Sources:
- What applied-AI engineer interviews test (Individual): https://www.techinterview.org/post/3233476824/ai-engineer-interview-rag-agents-evals/