Why it hurts training
Sparsity causes three separate problems, and they are worth keeping apart:
Memory. Storing a 100,000×50,000 matrix densely at 4 bytes per float is 20 GB, nearly all of it zeros. Sparse formats such as CSR store only the non-zeros plus their indices, typically cutting this by two or three orders of magnitude.
Wasted computation. A dense matrix multiply performs a multiply-add for every element, and multiplying by zero contributes nothing. Most of the arithmetic produces no information.
Uneven gradients. This is the subtle one. A weight connected to a feature that is non-zero in 0.1% of samples receives a gradient in only 0.1% of steps. Rare features therefore learn roughly a thousand times more slowly than common ones under a single global learning rate — which is precisely the problem
Adagrad and
Adam solve, by giving each parameter its own effective rate based on how often it has received a gradient. That is why adaptive optimisers are the standard choice on sparse data.
Two different problems with the same name
"Sparse" describes two situations that need entirely different responses, and conflating them causes a lot of confusion.
Sparse features — most values in the input are zero. One-hot encoded categories, bag-of-words text, user-item interaction matrices. A vocabulary of 50,000 words means a document vector where 49,900 entries are zero.
Sparse labels — you have plenty of data and very few annotations. This is a labelling problem, addressed by semi-supervised learning, active learning or transfer learning.
Both are common; only the first is about representation, and the rest of this is mostly about that one.
Why sparse features are awkward
Memory. A million users by 50,000 items stored densely is 200GB of floats. Stored sparsely — only the non-zero positions and values — it may be a few hundred megabytes. Sparse formats are not an optimisation here; they are the difference between possible and impossible.
Thin statistics. A feature that is non-zero in 20 of a million rows gives the model almost nothing to learn from, so its weight is mostly noise.
Distances stop working. Two documents that share no words have identical Euclidean distance regardless of what else they contain. Cosine similarity, which compares direction, is the standard alternative for exactly this reason.
Gradients are sparse too. A weight connected to a feature that is zero in this batch receives no gradient, so it updates only in the batches where its feature appears. Rare features therefore learn very slowly — which is precisely the problem adaptive optimisers were designed for. Adam and AdaGrad give parameters with infrequent gradients larger effective steps.
Embeddings: the standard fix
Instead of a 50,000-dimensional sparse vector, learn a dense vector of 64 to 256 numbers per category, trained jointly with the rest of the model.
| | One-hot | Embedding |
|---|
| Width | One column per category | A chosen width, e.g. 64 |
| Similar categories | Equidistant — no notion of similarity | Nearby in the space |
| Rare categories | Almost no data per column | Share structure with others |
| Memory | Sparse, wide | Dense, compact |
The second row is the important one. One-hot encoding makes every category exactly as different from every other; an embedding can place "London" near "Manchester" and far from "banana" because the training signal pushes them there. That is why embeddings are the default for high-cardinality categorical features in deep models, and why they underpin every recommendation system.
A common width heuristic is min(50, (cardinality + 1) // 2), or the fourth root of the cardinality — both are starting points to be tuned rather than rules.
Mostly zeros, and what that costs
Sparse data breaks dense assumptions in three specific ways -- memory, gradient flow, and distance. Each one is measured here, along with the standard fix.
Guided experiments
- Start dense. Set Input Sparsity Level to 0 and press Stream Data. Every input carries signal and every weight receives a gradient on every sample.
- Introduce realistic sparsity. Set Input Sparsity Level to 70 and stream again. Most inputs are now zero, and the connections behind them go quiet — a weight whose input is zero gets no gradient at all that step.
- Push it to the recommender regime. Set Input Sparsity Level to 95 and press Next Sample repeatedly. Watch how rarely any particular input activates. Those weights update on a handful of samples in a hundred, which is exactly why they lag.
- Compare which weights move. At high sparsity, note that the weights attached to frequently non-zero inputs converge quickly while the rest barely move. A global learning rate cannot serve both.
Embeddings, the standard fix
For high-cardinality categorical data the usual answer is not to keep the one-hot vector at all. An embedding layer maps each category to a short dense vector — say 50 dimensions instead of 195 columns — learned during training.
Mathematically this is identical to multiplying the one-hot vector by a weight matrix, but the implementation is a row lookup instead of a matrix multiply, so it skips the zeros entirely. It also solves a modelling problem: one-hot categories are all equidistant, whereas learned embeddings place similar categories near each other, so information is shared between related values rather than each being learned in isolation.
Where this goes wrong
- Densifying a sparse matrix. Calling .toarray() on a large sparse matrix is the classic out-of-memory error. Keep it sparse through the whole pipeline.
- Plain SGD on sparse features. Rare features learn far too slowly. Use Adam or Adagrad, which adapt per-parameter learning rates to update frequency.
- One-hot encoding a high-cardinality column. Thousands of near-empty columns; use an embedding, target encoding or hashing instead.
- Mean-imputing a structural zero. In sparse data zero usually means “absent”, not “missing”. Replacing it with a column mean invents signal that was never there.
- Centring sparse data. Subtracting the mean makes every zero non-zero and destroys the sparsity outright. Scale without centring.
What to remember
Sparse data is the norm for one-hot categories, text and recommenders, and it costs memory, wasted arithmetic, and — least obviously — badly uneven learning rates, because a weight only updates when its input is non-zero. Keep sparse matrices in sparse formats, replace one-hot encodings of high-cardinality fields with learned embeddings, and use an adaptive optimiser so rare features are not left thousands of steps behind common ones.
Handling it in code
For classical models, keep the matrix sparse end to end:
from scipy import sparse
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
X = TfidfVectorizer(min_df=5).fit_transform(docs) # sparse matrix
print(X.shape, X.nnz / (X.shape[0] * X.shape[1])) # density, often < 1%
LogisticRegression().fit(X, y) # accepts sparse input
Never call .toarray() on a large sparse matrix — that is the line that exhausts memory. min_df=5 is also doing real work: dropping terms that appear in fewer than five documents removes most of the vocabulary and almost none of the signal.
For neural networks, use an embedding layer and pass integer indices rather than one-hot vectors:
emb = nn.Embedding(num_embeddings=50_000, embedding_dim=64)
out = emb(token_ids) # (batch, seq) -> (batch, seq, 64)
nn.Embedding is a lookup, not a matrix multiplication, so it never materialises the one-hot vector at all. nn.EmbeddingBag sums or averages several embeddings in one call, which is the efficient way to represent a bag of features.
Which models cope well
| Model | With sparse features |
|---|
| Linear / logistic regression | Excellent — sparse-aware and fast |
| Naive Bayes | Excellent — designed for high-dimensional counts |
| Gradient boosting (LightGBM, XGBoost) | Good — handles sparsity natively |
| Neural networks with embeddings | Excellent, and the standard for high cardinality |
| KNN, k-means | Poor without cosine distance and dimension reduction |
| Dense neural networks on one-hot input | Poor — wasteful and slow to learn |
Two further options worth knowing. Feature hashing maps categories into a fixed number of buckets, accepting collisions in exchange for bounded memory and no vocabulary to maintain — useful when new categories appear constantly. And truncated SVD reduces a sparse matrix to a few hundred dense components, which is what makes clustering and nearest-neighbour search on text feasible.
Sparse labels, briefly
The other meaning deserves its own short answer, since the techniques are different:
- Transfer learning — start from a pretrained model, so a few hundred labels suffice.
- Self-supervised pretraining on your unlabelled data, then fine-tune on the labelled subset.
- Active learning — let the model choose which examples to send for labelling, prioritising the ones it is least sure about.
- Pseudo-labelling — label the confident predictions on unlabelled data and train on them, carefully, since errors compound.
- Weak supervision — combine noisy rules and heuristics into probabilistic labels.
The first two are where the leverage is: pretraining plus a small labelled set now beats a large labelled set trained from scratch on most tasks.
Questions people ask
When is data "sparse"? Informally, when a large majority of entries are zero. Below about 10% density, sparse storage and sparse-aware models start to matter.
Should I use one-hot or an embedding? One-hot below about 10 categories; embeddings above roughly 50, especially in a network.
Does PCA work on sparse data? Not directly — centring destroys sparsity. Use TruncatedSVD, which does not centre.
Which distance for sparse vectors? Cosine, almost always. Euclidean distance is dominated by the shared zeros.
Do trees handle sparsity? LightGBM and XGBoost handle it natively and efficiently, including treating missing and zero appropriately.
Why do rare features learn slowly? They appear in few batches, so their weights receive few gradients. Adaptive optimisers partly compensate by giving them larger effective steps.
Recap in one screen
- "Sparse" means either mostly-zero features or mostly-missing labels — different problems, different fixes.
- Sparse features cost memory, give thin per-feature statistics, break Euclidean distance and slow learning for rare values.
- Keep sparse matrices sparse; never densify a large one.
- Embeddings replace wide one-hot encodings with compact dense vectors that can express similarity.
- For sparse labels, transfer learning and self-supervised pretraining give the most leverage.