Explain disaggregated prefill/decode serving and when it pays for itself.
Reported at:Moonshot AIGroq
Model answer
A concise answer you could give in the room. Try your own version first, then compare.
What the interviewer is probing
Whether you can reason about LLM serving as a systems problem, separating compute-bound prefill from memory-bound decode, and trade latency against throughput with numbers.
How to structure your answer
- 1Separate the phases: prefill processes the prompt in parallel; decode generates one token at a time.
- 2Explain the bottlenecks: prefill is compute-bound; decode reads all the weights per step, so it is bandwidth-bound.
- 3Explain batching: static, then continuous (iteration-level) batching, chunked prefill, and disaggregation.
- 4Define the metrics (TTFT, TPOT/ITL, throughput, p99) and how each lever moves them.
Interactive · Batching on the decode roofline
Idealised decode bound on one H100 (3.35 TB/s, 989 TFLOPS BF16), ignoring KV cache and overheads.
bound by
memory
tok/s per user
24
tok/s total
24
Key points to hit
0/3 revealedTry to answer each one out loud before revealing it.
Why does batching help decode so much?
What is continuous batching?
What does prefill/decode disaggregation buy?
Common pitfalls
- Optimising throughput while ignoring tail latency SLOs.
- Forgetting that KV memory caps the batch size.
- Treating tokens per second as one number without saying whether it is per user or per GPU.
Likely follow-ups
- Estimate decode tokens/sec for a 70B model on one H100.
- How would you schedule mixed short and long prompts?
- Why did p99 latency double with no model change?
Source: Outcome School · ai-engineering-interview-questions-company-wise. Question compiled by Outcome School from publicly reported interview experiences (Apache-2.0). The guidance on this page is original.