Working out a feature's cost on the spot is a quick way to show you've shipped something real.
The formula: fresh input × input price, plus cached input × cached price, plus output × output price, all divided by one million.
Output usually costs several times more than input. So long answers matter more than you might expect.
For latency, watch time to first token and the slow requests (p95), rather than the average. Streaming improves how fast it feels, though the total time stays the same.
- What's the request cost formula?
- (fresh input × input price + cached × cached price + output × output price) ÷ 1,000,000.
- How do you get a monthly cost from a request cost?
- Cost per request × requests per day × 30.
- Why report p95 latency rather than the average?
- An average hides the slow requests that users notice. p95 is what the slowest 1 in 20 at least experience.
- What does streaming change?
- Time to first token, and so how fast it feels. Total generation time stays about the same.
- Name three ways to cut cost.
- Cache fixed prompt text, send less irrelevant context, and shorten outputs. Consider a cheaper model where evals show quality holds.