What is speculative decoding? Why is output quality preserved, and when does it not help?
Reported at:NVIDIATogether AI
Model answer
A concise answer you could give in the room. Try your own version first, then compare.
What the interviewer is probing
Whether you understand why speculative decoding is lossless and can predict when it helps from acceptance rate and batch size.
How to structure your answer
- 1Describe the idea: a cheap draft proposes k tokens, and the target verifies them in one forward pass.
- 2Explain why it is lossless: the rejection-sampling acceptance rule preserves the target distribution.
- 3Derive the speed-up from acceptance rate α and draft cost.
- 4State when it hurts: high batch size (already compute-bound), low α, an expensive draft.
Interactive · Speculative decoding speed-up
Expected speed-up = E[accepted tokens] ÷ (1 + k · draft cost), with E = (1 − αᵏ⁺¹)/(1 − α).
tokens per verify
2.77
speed-up
1.98×
Key points to hit
0/3 revealedTry to answer each one out loud before revealing it.
Why is quality preserved?
Why does it help at low batch sizes?
Draft options?
Common pitfalls
- Claiming it changes the outputs.
- Ignoring the extra memory for the draft model and its KV cache.
- Forgetting the gains shrink under heavy batching.
Likely follow-ups
- How would you choose k adaptively?
- How does speculative decoding interact with sampling temperature?
- Implement the acceptance step.
Source: Outcome School · ai-engineering-interview-questions-company-wise. Question compiled by Outcome School from publicly reported interview experiences (Apache-2.0). The guidance on this page is original.