AI.Cruz

Explain SwiGLU and why gated activations replaced ReLU/GELU in modern LLM MLP blocks.

Common Questions Asked Across CompaniesLLM Internals and ArchitectureTopic: Normalisation & activations

Reported at:Meta

Model answer

A concise answer you could give in the room. Try your own version first, then compare.

What the interviewer is probing

Whether you know why modern transformer blocks look the way they do, and can tie each choice to training stability or efficiency.

How to structure your answer

  1. 1Describe the residual stream and where the normalisation sits (pre- versus post-norm).
  2. 2Explain why pre-norm trains stably at depth.
  3. 3Define RMSNorm and why it replaced LayerNorm.
  4. 4Explain gated MLPs (SwiGLU) and their parameter accounting.

Key points to hit

0/3 revealed

Try to answer each one out loud before revealing it.

  1. Why pre-norm?

  2. RMSNorm versus LayerNorm?

  3. Why SwiGLU?

Common pitfalls

  • Mixing up batch norm and layer norm.
  • Forgetting the final norm before the LM head.
  • Ignoring the parameter matching for gated MLPs.

Likely follow-ups

  • What is QK-norm and why do some models add it?
  • What causes loss spikes and how does normalisation help?
  • Draw one transformer block with the shapes.

Source: Outcome School · ai-engineering-interview-questions-company-wise. Question compiled by Outcome School from publicly reported interview experiences (Apache-2.0). The guidance on this page is original.