AI.Cruz

Do the roofline maths: how many tokens/sec can one H100 produce for a 70B model at batch size 1?

Common Questions Asked Across CompaniesInference, Serving and GPU PerformanceTopic: GPU performance

Reported at:NVIDIATogether AI

Model answer

A concise answer you could give in the room. Try your own version first, then compare.

What the interviewer is probing

Whether you can reason with the roofline model: compute versus memory bandwidth, arithmetic intensity, and what that implies for kernels and batch sizes.

How to structure your answer

  1. 1Know the hardware numbers: peak FLOPs, HBM bandwidth and SRAM size for the target GPU.
  2. 2Compute the arithmetic intensity (FLOPs per byte) of the operation.
  3. 3Compare it with the ridge point to classify the operation as compute- or memory-bound.
  4. 4Pick optimisations to match: fusion, tiling, larger batches, lower precision, overlapping communication.

Interactive · Decode roofline

Idealised decode bound on one H100 (3.35 TB/s, 989 TFLOPS BF16), ignoring KV cache and overheads.

bound by
memory
tok/s per user
24
tok/s total
24

Key points to hit

0/3 revealed

Try to answer each one out loud before revealing it.

  1. Decode tokens/sec bound for a 70B model at batch 1?

  2. What does kernel fusion buy?

  3. How do you profile?

Common pitfalls

  • Quoting peak FLOPs for a memory-bound workload.
  • Ignoring launch overhead and CPU gaps.
  • Not mentioning CUDA graphs for small decode kernels.

Likely follow-ups

  • Write a tiled matmul kernel in Triton.
  • Why is softmax memory-bound?
  • How would you overlap communication and compute?

Source: Outcome School · ai-engineering-interview-questions-company-wise. Question compiled by Outcome School from publicly reported interview experiences (Apache-2.0). The guidance on this page is original.