AI.Cruz

What are TTFT, TPOT, ITL and throughput, and how do they trade against each other?

Common Questions Asked Across CompaniesInference, Serving and GPU PerformanceTopic: Inference & batching

Reported at:MicrosoftApplePerplexity

Model answer

A concise answer you could give in the room. Try your own version first, then compare.

What the interviewer is probing

Whether you can reason about LLM serving as a systems problem, separating compute-bound prefill from memory-bound decode, and trade latency against throughput with numbers.

How to structure your answer

  1. 1Separate the phases: prefill processes the prompt in parallel; decode generates one token at a time.
  2. 2Explain the bottlenecks: prefill is compute-bound; decode reads all the weights per step, so it is bandwidth-bound.
  3. 3Explain batching: static, then continuous (iteration-level) batching, chunked prefill, and disaggregation.
  4. 4Define the metrics (TTFT, TPOT/ITL, throughput, p99) and how each lever moves them.

Interactive · Batching on the decode roofline

Idealised decode bound on one H100 (3.35 TB/s, 989 TFLOPS BF16), ignoring KV cache and overheads.

bound by
memory
tok/s per user
24
tok/s total
24

Key points to hit

0/3 revealed

Try to answer each one out loud before revealing it.

  1. Why does batching help decode so much?

  2. What is continuous batching?

  3. What does prefill/decode disaggregation buy?

Common pitfalls

  • Optimising throughput while ignoring tail latency SLOs.
  • Forgetting that KV memory caps the batch size.
  • Treating tokens per second as one number without saying whether it is per user or per GPU.

Likely follow-ups

  • Estimate decode tokens/sec for a 70B model on one H100.
  • How would you schedule mixed short and long prompts?
  • Why did p99 latency double with no model change?

Source: Outcome School · ai-engineering-interview-questions-company-wise. Question compiled by Outcome School from publicly reported interview experiences (Apache-2.0). The guidance on this page is original.