AI.Cruz
MACHINE LEARNING

Elastic Net Regression

Elastic Net mixes L2 (Ridge) and L1 (Lasso) penalties via a mixing parameter α. It handles correlated predictors better than Lasso while still doing feature selection — the best of both worlds.

1Why Lasso Fails with Correlated Features

Lasso (L1 regularisation) is powerful because it drives some coefficients to exactly zero — automatic feature selection. But it has a critical weakness: when features are correlated, Lasso arbitrarily selects one and zeroes the rest, even when they all contribute genuine signal.

The pathological case is when you have more features than samples (d > n). Lasso can select at most n features — it simply can't fit a sparse solution with more coefficients than data points. For genomics data with 20,000 genes and 500 patients, Lasso breaks down.

Ridge (L2)
Shrinks all coefficients toward zero, none exactly to zero. Handles multicollinearity well. No feature selection.
Lasso (L1)
Drives some coefficients to exactly zero. Automatic feature selection. Breaks with correlated features or d > n.
Elastic Net
Combines both. Handles correlated features by sharing weight across them. Still performs feature selection.

2The Elastic Net Loss Function

Elastic Net adds both penalties to the RSS, controlled by two hyperparameters: overall strength λ and mixing ratio α.

Loss = RSS + λ((1-α)·‖β‖2² + α·‖β‖1). When α = 0: pure Ridge. When α = 1: pure Lasso. For 0 < α < 1: Elastic Net. The geometric constraint region is an “elastic ball” — rounder than the L1 diamond but more corner-like than the L2 sphere.

λ (lambda) — overall regularisation strength
Higher λ → more shrinkage → sparser solution. Choose via cross-validation. Too high: underfitting. Too low: overfitting.
α (alpha) — L1/L2 mixing ratio
α = 0 → Ridge. α = 1 → Lasso. Typical choice: α = 0.5 as starting point. Higher α gives more sparsity but less stability with correlated features.

3Grouped Sparsity and the Grouping Effect

A key advantage of Elastic Net over Lasso: when features are correlated, Elastic Net tends to select them as a group and distribute the coefficient weight among them, rather than arbitrarily zeroing all but one.

Intuition: if features x&sub1; and x&sub2; are perfectly correlated (x&sub1; = x&sub2;), Lasso picks one at random and zeroes the other. Elastic Net puts half the weight on each — a more stable and interpretable solution. This is called the grouping effect.

When to use Elastic Net

  • Features are correlated (multicollinearity)
  • d > n (more features than samples)
  • You want feature selection but Lasso is unstable
  • Genomics, NLP bag-of-words, sensor data

Limitations

  • Two hyperparameters to tune (λ and α) instead of one
  • No closed-form solution — requires coordinate descent or gradient descent
  • Harder to interpret coefficient paths than pure Ridge or Lasso

4Gradient Descent Implementation

Because the L1 norm has a non-smooth kink at zero, we use the subgradient of the absolute value: sign(β) for β ≠ 0, any value in [-1,1] for β = 0. In practice, we use the sign function throughout.

Update rule: β ← β - (η/n)(Xᵀ(Xβ - y) + λ(1-α)β + λα sign(β)). The L2 term is smooth (it's just scalar multiplication), while the L1 term uses sign(). Coordinate descent (updating one coefficient at a time with soft-thresholding) is typically faster in practice.

Related

Elastic Net — Ridge vs Lasso vs Elastic Net coefficient paths with numpy

Recall check

0 of …

Say the answer out loud before revealing — recalling is what makes it stick. ➜ Next: Multiclass Strategies

  1. What is the Elastic Net loss function and how do α and λ control it?

  2. Why does Lasso fail when features are correlated?

  3. What is the Elastic Net grouping effect?

  4. Why does Elastic Net have no closed-form solution?