AI.Cruz

Compare tensor, pipeline, data, sequence and expert parallelism. When do you combine them?

Common Questions Asked Across CompaniesInference, Serving and GPU PerformanceTopic: Distributed training

Reported at:Google DeepMindMetaAmazonNVIDIA

Model answer

A concise answer you could give in the room. Try your own version first, then compare.

What the interviewer is probing

Whether you can choose and combine parallelism strategies from first principles: memory per GPU, communication volume and interconnect topology.

How to structure your answer

  1. 1Account for memory: weights, gradients, optimiser states (~16 bytes per parameter with Adam in mixed precision) and activations.
  2. 2Describe each axis: data (+ZeRO/FSDP), tensor, pipeline, sequence/context and expert parallelism.
  3. 3Map axes to topology: tensor within a node (NVLink), pipeline and data across nodes.
  4. 4Address stability and efficiency: checkpointing, loss spikes, MFU, failure recovery.

Key points to hit

0/3 revealed

Try to answer each one out loud before revealing it.

  1. What does ZeRO-3 / FSDP shard?

  2. Why keep tensor parallelism within a node?

  3. What is the pipeline bubble?

Common pitfalls

  • Forgetting optimiser-state memory.
  • Ignoring activation memory at long sequence lengths.
  • Choosing a strategy without considering the interconnect.

Likely follow-ups

  • How would you debug a loss spike at step 300k?
  • How many GPUs do you need to train a 70B model?
  • What is MFU and what is a good value?

Source: Outcome School · ai-engineering-interview-questions-company-wise. Question compiled by Outcome School from publicly reported interview experiences (Apache-2.0). The guidance on this page is original.