AI.Cruz

Compare greedy, beam search, top-k, top-p and temperature sampling. When does each fail?

Common Questions Asked Across CompaniesLLM Internals and ArchitectureTopic: Decoding & sampling

Reported at:Google DeepMindApplePerplexity

Model answer

A concise answer you could give in the room. Try your own version first, then compare.

What the interviewer is probing

Whether you can explain how logits become text and pick decoding settings deliberately for a task.

How to structure your answer

  1. 1Start from logits → softmax → distribution over the vocabulary.
  2. 2Explain each strategy: greedy, beam, temperature, top-k, top-p (nucleus), min-p.
  3. 3Give the failure mode of each (repetition, blandness, incoherence).
  4. 4Map settings to tasks: extraction and code low-temperature, creative writing higher.

Interactive · Decoding playground

Next-token distribution after temperature → top-k → top-p. Faded bars were filtered out.

theacatdogsatranonmatbluequantum
candidates
10
entropy
2.49 b

Key points to hit

0/3 revealed

Try to answer each one out loud before revealing it.

  1. What does temperature do mathematically?

  2. Top-k versus top-p?

  3. Why is beam search rare for chat?

Common pitfalls

  • Applying top-p before the temperature without saying so (order matters).
  • Thinking temperature 0 is deterministic across hardware or batches.
  • Forgetting repetition penalties and stop sequences.

Likely follow-ups

  • Implement top-p sampling over a logits vector.
  • How does sampling interact with speculative decoding?
  • How would you make outputs reproducible?

Source: Outcome School · ai-engineering-interview-questions-company-wise. Question compiled by Outcome School from publicly reported interview experiences (Apache-2.0). The guidance on this page is original.