AI.Cruz

How does Byte Pair Encoding work, and what are its failure modes (numbers, code, non-Latin scripts)?

Common Questions Asked Across CompaniesLLM Internals and ArchitectureTopic: Tokenization

Reported at:AlibabaSarvam AIHugging Face

Model answer

A concise answer you could give in the room. Try your own version first, then compare.

What the interviewer is probing

Whether you understand that the tokenizer is part of the model, and that many reported model failures (arithmetic, code, non-English cost) are tokenizer artefacts.

How to structure your answer

  1. 1Describe the algorithm: start from bytes or characters and repeatedly merge the most frequent adjacent pair.
  2. 2Explain encoding: apply the learned merges in order to new text.
  3. 3Name the failure modes with concrete examples.
  4. 4Discuss the trade-offs of vocabulary size: sequence length versus embedding size and rare-token training.

Key points to hit

0/3 revealed

Try to answer each one out loud before revealing it.

  1. Why do LLMs struggle with digit arithmetic?

  2. Why is non-Latin text more expensive?

  3. Byte-level BPE versus SentencePiece?

Common pitfalls

  • Confusing training the merges with encoding text.
  • Ignoring pre-tokenization (regex splits) and special tokens.
  • Assuming one token equals one word.

Likely follow-ups

  • Implement BPE training in Python.
  • How would you extend a tokenizer for a new language?
  • What is a glitch token?

Source: Outcome School · ai-engineering-interview-questions-company-wise. Question compiled by Outcome School from publicly reported interview experiences (Apache-2.0). The guidance on this page is original.