AI.Cruz

Explain FlashAttention. It does not reduce FLOPs, so why is it faster?

Common Questions Asked Across CompaniesLLM Internals and ArchitectureTopic: Attention

Reported at:Together AI

Model answer

A concise answer you could give in the room. Try your own version first, then compare.

What the interviewer is probing

Whether you can derive attention from Q, K and V instead of reciting it, and connect the maths to numerical stability, memory traffic and cost at long sequence lengths.

How to structure your answer

  1. 1Write the formula: softmax(QKᵀ / √dₖ) · V, and name each tensor's shape.
  2. 2Explain what each step computes: similarity scores, normalised weights, weighted mix of values.
  3. 3Address the specific twist (scaling, masking, multi-head, FlashAttention) with a quantitative argument.
  4. 4Close with the cost: O(n²) compute and memory in sequence length, and what production systems do about it.

Interactive · Softmax saturation vs. √dₖ scaling

Attention weights of one query over 10 random keys. Increase dₖ and turn scaling off: the softmax saturates onto a single token.

k1k2k3k4k5k6k7k8k9k10
logit std
1.23
max weight
47%
entropy
2.32 b

Key points to hit

0/3 revealed

Try to answer each one out loud before revealing it.

  1. Why divide by √dₖ?

  2. Why multiple heads?

  3. Why is FlashAttention faster with the same FLOPs?

Common pitfalls

  • Saying the scale exists "to make numbers smaller" without the variance argument.
  • Forgetting the causal mask in decoder-only models.
  • Claiming FlashAttention is an approximation (it is exact).

Likely follow-ups

  • What happens to attention entropy as context length grows?
  • How does the KV cache change attention cost at decode time?
  • Implement masked multi-head attention in NumPy.

Source: Outcome School · ai-engineering-interview-questions-company-wise. Question compiled by Outcome School from publicly reported interview experiences (Apache-2.0). The guidance on this page is original.