AI.Cruz

What is a mixture-of-experts architecture and how does it scale capacity without scaling FLOPs?

Common Questions Asked Across CompaniesLLM Internals and ArchitectureTopic: Mixture of experts

Reported at:Mistral AICohereDeepSeekMoonshot AIZhipu AIAlibaba

Model answer

A concise answer you could give in the room. Try your own version first, then compare.

What the interviewer is probing

Whether you understand that MoE decouples total parameters from per-token FLOPs, and the systems costs this creates (memory, routing, load balance, communication).

How to structure your answer

  1. 1Describe the block: a router picks the top-k of N expert MLPs per token.
  2. 2Separate total parameters (memory) from active parameters (FLOPs per token).
  3. 3Explain training issues: load-balancing loss, router collapse, capacity factor, token dropping.
  4. 4Explain serving issues: all experts must be resident, expert parallelism and all-to-all traffic.

Key points to hit

0/3 revealed

Try to answer each one out loud before revealing it.

  1. Why is Mixtral ~47B total but ~13B active?

  2. Why add an auxiliary loss?

  3. Is MoE cheaper to serve?

Common pitfalls

  • Saying MoE uses less memory.
  • Ignoring routing imbalance at inference time.
  • Confusing experts with domain specialists (routing is mostly token-level).

Likely follow-ups

  • How does expert parallelism map onto GPUs?
  • What are shared experts (as in DeepSeek)?
  • How do you fine-tune an MoE model?

Source: Outcome School · ai-engineering-interview-questions-company-wise. Question compiled by Outcome School from publicly reported interview experiences (Apache-2.0). The guidance on this page is original.