AI.Cruz

Explain the difference between pre-training, supervised fine-tuning and preference optimisation.

Common Questions Asked Across CompaniesLLM Internals and ArchitectureTopic: Post-training & alignment

Reported at:MetaScale AI

Model answer

A concise answer you could give in the room. Try your own version first, then compare.

What the interviewer is probing

Whether you can explain the full post-training pipeline, why each stage exists, and the failure modes (reward hacking, over-optimisation, mode collapse) that teams actually fight.

How to structure your answer

  1. 1Lay out the stages: pre-training → SFT → preference optimisation (RLHF / DPO) → optionally RL with verifiable rewards.
  2. 2Explain the objective: maximise reward minus β·KL to a reference policy.
  3. 3Compare methods: PPO (reward model + RL), DPO (closed-form from preference pairs), GRPO (group-relative baselines).
  4. 4Discuss data and evaluation: preference collection, reward model accuracy, win-rates, reward-hacking checks.

Key points to hit

0/3 revealed

Try to answer each one out loud before revealing it.

  1. Why the KL penalty?

  2. DPO versus PPO?

  3. What do verifiable rewards add?

Common pitfalls

  • Describing RLHF without the reference model or KL term.
  • Ignoring how preference data is collected and its biases.
  • Assuming a higher reward always means a better model.

Likely follow-ups

  • Derive the DPO loss from the RLHF objective.
  • How would you detect reward hacking?
  • What is Constitutional AI / RLAIF?

Source: Outcome School · ai-engineering-interview-questions-company-wise. Question compiled by Outcome School from publicly reported interview experiences (Apache-2.0). The guidance on this page is original.