AI.Cruz

Compare FP16, BF16, FP8, INT8, INT4 and FP4 for serving. What breaks at each step down?

Common Questions Asked Across CompaniesInference, Serving and GPU PerformanceTopic: Quantization

Reported at:Mistral AIAppleNVIDIATogether AICharacter.AI

Model answer

A concise answer you could give in the room. Try your own version first, then compare.

What the interviewer is probing

Whether you can match numeric formats to hardware and workloads, and know where quality breaks and how to recover it.

How to structure your answer

  1. 1Separate what is quantised: weights, activations, KV cache.
  2. 2Compare the formats: range versus precision (FP16, BF16, FP8, INT8, INT4, FP4).
  3. 3Explain the methods: PTQ (GPTQ, AWQ, SmoothQuant) versus QAT; per-channel and group scales.
  4. 4Validate: perplexity plus task evals, especially long-context, maths and code.

Interactive · Precision vs. KV cache memory

2 × layers × kv_heads × head_dim × seq × batch × bytes

share of one 80 GB GPU (cache only, excluding weights)
per token
320 KB
per sequence
2.5 GB
total
20.0 GB

Key points to hit

0/3 revealed

Try to answer each one out loud before revealing it.

  1. BF16 versus FP16?

  2. Why are activations harder than weights?

  3. Why does weight-only INT4 speed up decode?

Common pitfalls

  • Reporting only perplexity.
  • Ignoring kernel support (a format with no fast kernel is slower).
  • Quantising the KV cache without testing long contexts.

Likely follow-ups

  • How does QLoRA fine-tune a 4-bit model?
  • What breaks first when moving from FP8 to FP4?
  • How would you quantise an MoE model?

Source: Outcome School · ai-engineering-interview-questions-company-wise. Question compiled by Outcome School from publicly reported interview experiences (Apache-2.0). The guidance on this page is original.