Skip-gram and CBOW
Word2Vec offers two ways to set up the prediction:
- Skip-gram — given the centre word, predict the surrounding words. From “the cat sat on the mat”, given sat, predict the, cat, on, the. Slower, and better on rare words because each occurrence generates several training examples.
- CBOW — given the surrounding words, predict the centre. Faster, and better on frequent words.
The architecture is deliberately trivial: an input embedding layer, no hidden layer, and an output projection. The embedding matrix is the only thing anybody wants; the output layer is discarded after training. The vectors are a side effect of a task nobody cares about the answers to.
Negative sampling
The naive setup predicts a probability distribution over the whole vocabulary, which means a softmax over 50,000 words for every training example. That is prohibitively expensive.
Negative sampling replaces it with a much cheaper question. Instead of “which of 50,000 words is the context?”, ask “is this pair a real (word, context) pair or a fabricated one?” — a binary classification. For each true pair, draw perhaps 5 to 20 random words as negatives and train the model to score the real pair high and the fakes low.
Cost drops from 50,000 output computations to about 20, and quality is comparable. This is what made training on billions of words practical.
Learned from context, not assigned
Nobody writes embedding vectors by hand. They are learned, and every method rests on the same observation: words that appear in similar contexts tend to mean similar things. Fill in "I poured myself a cup of ___" and the plausible answers — tea, coffee, water — are related precisely because they share contexts.
Turn that into a training objective and the vectors fall out.
| Method | Objective | Output |
|---|
| Word2Vec skip-gram | Predict surrounding words from a centre word | One vector per word |
| Word2Vec CBOW | Predict the centre word from its context | One vector per word |
| GloVe | Factorise a co-occurrence count matrix | One vector per word |
| FastText | Skip-gram over character n-grams | Handles unseen words |
| BERT-style | Predict masked tokens in context | A different vector per occurrence |
| Sentence encoders | Bring related sentences together | One vector per sentence |
Word2Vec, concretely
Skip-gram takes a centre word and tries to predict the words around it within a window. Training on "the quick brown fox jumps" with a window of 2 produces pairs: (brown, the), (brown, quick), (brown, fox), (brown, jumps).
The model is a shallow network: an embedding lookup, then a projection to vocabulary size. After training, the projection layer is discarded and the embedding table is the product. The prediction task was only ever a means to shape those vectors.
The practical problem is the output layer. A softmax over 100,000 words for every training pair is prohibitively expensive, so Word2Vec uses negative sampling: instead of scoring all words, score the true context word and a handful (5–20) of random ones, and train the model to tell them apart. That turns a 100,000-way softmax into a few binary classifications and is the trick that made the method practical.
CBOW is the mirror image — predict the centre word from the average of its context. It trains faster and does slightly worse on rare words, since averaging washes them out.
GloVe takes a different route entirely: build a matrix of how often each word co-occurs with each other word, then factorise it so that the dot product of two vectors approximates the logarithm of their co-occurrence count. Global statistics rather than local windows, with comparable results.
What the window size decides
This is the parameter that changes what "similar" means, and it is worth knowing because it is easy to get wrong.
A small window (2–5) captures syntactic similarity — words that are interchangeable in a sentence. "Run" ends up near "walk" and "jog", and near "runs" and "ran".
A large window (10–15) captures topical similarity — words that appear in the same subject matter. "Run" ends up near "race", "marathon" and "athlete".
Neither is correct in general. For a part-of-speech task, small. For document retrieval or topic work, large. Setting it without thinking about which kind of similarity the task needs is how embeddings end up subtly unfit for purpose.
Other parameters that matter: dimension (100–300 for word vectors), minimum count (drop words appearing fewer than five times), and subsampling of very frequent words, which stops "the" dominating every context.
Trained by prediction, not by design
Nobody writes an embedding table. It falls out of training a model on a prediction task, and this trains one with skip-gram negative sampling -- the word2vec objective -- so you can watch the geometry appear.
Guided experiments
- Start from noise. Press Re-randomize and look at the layout. Vectors are random, so the arrangement is meaningless — no structure exists before training.
- Train one epoch. Press Train 1 Epoch and watch points shift slightly. Each step nudges words that co-occur closer together and pushes unrelated ones apart.
- Let clusters form. Press Train 20 Epochs. Related words gather into groups. Nothing told the model these words were related — it only ever tried to predict context.
- Re-randomize and train again. The clusters re-form, but in different positions and orientations. The absolute coordinates are arbitrary; only the relative geometry carries meaning, which is why you cannot compare vectors from two separately trained models.
Common mistakes
- Comparing vectors across models. Two training runs produce different, incompatible spaces. Vectors are only meaningful relative to others from the same run.
- Too little data. These methods need tens of millions of tokens. On a small corpus the vectors are noise, and a pretrained set is almost always better.
- Leaving frequent words unsubsampled. The and of co-occur with everything and carry almost no information. Word2Vec subsamples them aggressively, which improves both speed and quality.
- Assuming a bigger window is better. Small windows (2–5) capture syntactic similarity; large windows (10+) capture topical relatedness. They are different notions of “similar” and the right one depends on the task.
What to remember
Embeddings are learned by training a deliberately simple model on a proxy task — predict a word from its context, or the context from the word — and keeping the embedding matrix while discarding everything else. Semantic structure emerges because words in similar contexts need similar vectors to make the prediction work. Negative sampling replaces the vocabulary-wide softmax with a cheap binary decision, which is what made the whole approach scale.
Contextual embeddings, and why they superseded fixed ones
Word2Vec gives "bank" one vector, averaging the river and the financial senses into something that is neither.
A transformer produces a different vector for each occurrence, computed from the surrounding words. In "the river bank" the vector has absorbed "river"; in "the savings bank" it has absorbed "savings". Polysemy is resolved by construction.
The training objective is what generates them:
- Masked language modelling (BERT) hides 15% of tokens and predicts them from both directions.
- Next-token prediction (GPT) predicts each token from those before it.
Either way, the hidden states of the trained model are the contextual embeddings, and no separate embedding-training step is needed.
For sentence-level vectors, taking BERT's [CLS] output directly works poorly — it was not trained to make similar sentences nearby. Sentence-BERT fixes this by fine-tuning with a contrastive objective on pairs of related and unrelated sentences, which is why purpose-built embedding models substantially outperform pooled hidden states from a general model.
Training your own
Two situations justify it, and neither is the common case.
An embedding layer inside your model. This is the everyday one: a high-cardinality categorical feature (product id, user id) replaced by a learned dense vector, trained jointly with the rest of the network. No separate pretraining, no corpus needed.
emb = nn.Embedding(num_embeddings=50_000, embedding_dim=64)
vecs = emb(item_ids) # learned along with everything else
Domain-specific text embeddings. When your vocabulary genuinely differs from web text — clinical notes, legal filings, internal product codes — fine-tuning an existing sentence encoder on your own pairs beats a general model. Contrastive fine-tuning needs pairs of related and unrelated text, which can often be mined from existing structure (a question and its accepted answer, a ticket and its resolution).
Training word vectors from scratch on a small corpus is almost always worse than using a pretrained model. Embeddings need a great deal of text.
Questions people ask
Do I need to train embeddings myself? Rarely for text — use a pretrained sentence encoder. Yes for categorical features in your own model.
Skip-gram or CBOW? Skip-gram for rare words and smaller corpora; CBOW when speed matters and the corpus is large.
What is negative sampling? Replacing a full softmax over the vocabulary with a few binary comparisons against random words. It is what made Word2Vec tractable.
Why do I get different vectors on each run? Random initialisation and sampling. The geometry is stable even though the coordinates are not, so similarities are reproducible and individual dimensions are not.
Can embeddings handle words not in the vocabulary? Word2Vec cannot. FastText can, by composing character n-grams. Subword tokenisers sidestep the problem entirely.
How much text do I need? Millions of words for usable word vectors from scratch. Almost none if you fine-tune a pretrained model.
Recap in one screen
- Embeddings are learned from the principle that words in similar contexts have similar meanings.
- Skip-gram predicts context from a word; CBOW does the reverse; GloVe factorises co-occurrence counts.
- Negative sampling replaces an unaffordable softmax with a few binary comparisons.
- Window size chooses between syntactic and topical similarity — pick it for the task.
- Contextual embeddings from transformers give a different vector per occurrence, which resolves polysemy.