What do the Chinchilla scaling laws say, and how do they differ from earlier scaling intuitions?
Reported at:Anthropic
Model answer
A concise answer you could give in the room. Try your own version first, then compare.
What the interviewer is probing
Whether you can use scaling laws as a planning tool: allocating a compute budget between parameters and data, and knowing where the laws stop predicting.
How to structure your answer
- 1State the power-law form: loss falls predictably with parameters, data and compute.
- 2Contrast Kaplan (favour parameters) with Chinchilla (~20 tokens per parameter, balanced).
- 3Bring in inference cost: over-training smaller models is rational when serving dominates.
- 4Name the limits: emergent behaviour, data exhaustion, downstream metrics not tracking loss.
Key points to hit
0/3 revealedTry to answer each one out loud before revealing it.
Chinchilla's key finding?
Why train 8B models on 15T tokens?
Do scaling laws predict capabilities?
Common pitfalls
- Treating the 20:1 ratio as a universal constant.
- Ignoring data quality and repetition.
- Conflating loss with task performance.
Likely follow-ups
- How would you fit a scaling law from small runs?
- What happens when you run out of unique data?
- How do scaling laws apply to RL or inference-time compute?
Source: Outcome School · ai-engineering-interview-questions-company-wise. Question compiled by Outcome School from publicly reported interview experiences (Apache-2.0). The guidance on this page is original.