AI.Cruz

What is positional encoding in transformers, and how has it evolved (sinusoidal → learned → RoPE → ALiBi)?

Common Questions Asked Across CompaniesLLM Internals and ArchitectureTopic: Positional encoding

Asked across many companies

Model answer

A concise answer you could give in the room. Try your own version first, then compare.

What the interviewer is probing

Whether you know why attention needs position information at all, and how relative schemes like RoPE make length extrapolation possible.

How to structure your answer

  1. 1Attention is permutation-invariant, so position must be injected.
  2. 2Walk the evolution: sinusoidal → learned absolute → relative (T5, ALiBi) → RoPE.
  3. 3Explain RoPE: rotate q and k pairwise by angle ∝ position, so q·k depends on relative offset.
  4. 4Explain context extension: position interpolation, NTK scaling, YaRN, plus a short fine-tune.

Key points to hit

0/3 revealed

Try to answer each one out loud before revealing it.

  1. Why does RoPE encode relative position?

  2. Why does naive extrapolation fail?

  3. What does ALiBi do?

Common pitfalls

  • Saying RoPE is added to the embeddings (it is applied to q and k).
  • Ignoring that extended context still needs long-context training data.
  • Confusing a longer context window with better use of that context.

Likely follow-ups

  • Derive the RoPE rotation for a 2-D pair.
  • How would you evaluate a context-extended model?
  • Why does NTK-aware scaling treat frequencies differently?

Source: Outcome School · ai-engineering-interview-questions-company-wise. Question compiled by Outcome School from publicly reported interview experiences (Apache-2.0). The guidance on this page is original.