AI.Cruz

Explain prefix caching / prompt caching. When should you use it, and what invalidates a cached prefix?

Common Questions Asked Across CompaniesInference, Serving and GPU PerformanceTopic: Prompt caching

Reported at:Moonshot AICharacter.AI

Model answer

A concise answer you could give in the room. Try your own version first, then compare.

What the interviewer is probing

Whether you can design prompts and systems so that shared prefixes are reused, and know exactly what breaks a cache hit.

How to structure your answer

  1. 1Explain the mechanism: reuse the KV cache for an identical token prefix.
  2. 2Structure the prompt: stable content first (system prompt, tools, documents), variable content last.
  3. 3List the invalidators: any token change in the prefix, reordering, timestamps, tool-schema edits, a different model.
  4. 4Quantify the benefit: lower TTFT and cost on repeated prefixes; measure the hit rate.

Key points to hit

0/3 revealed

Try to answer each one out loud before revealing it.

  1. Why must the match be an exact prefix?

  2. Biggest real-world win?

  3. How do you keep the hit rate high?

Common pitfalls

  • Putting the current date or a user ID at the start of the prompt.
  • Summarising history in a way that rewrites the prefix every turn.
  • Not tracking the cache hit rate as a metric.

Likely follow-ups

  • Build a prompt under a fixed token budget that maximises cache hits.
  • How does a server evict cached prefixes?
  • How do caching and per-user personalisation conflict?

Source: Outcome School · ai-engineering-interview-questions-company-wise. Question compiled by Outcome School from publicly reported interview experiences (Apache-2.0). The guidance on this page is original.