With more than one predictor, OLS extends naturally via matrix algebra: β̂ = (XᵀX)⁻¹Xᵀy. The mechanics are the same, but interpretation and multicollinearity require more care.
Simple linear regression fits one line: y = β&sub0; + β&sub1;x. Multiple linear regression generalises this to d predictors: y = β&sub0; + β&sub1;x&sub1; + β&sub2;x&sub2; + … + βdxd + ε. The goal is identical — find the parameter vector β that minimises residuals — but the geometry shifts from fitting a line to fitting a hyperplane in a (d+1)-dimensional space.
Classic example: predicting sales from TV, radio, and newspaper advertising spend. TV alone explains 61% of sales variance. Adding radio increases it to 90%. Adding newspaper provides almost no additional explanatory power (R² barely changes) — because newspaper spend is effectively captured by the other two predictors.
With n observations and d features, we stack the system into matrix form: y = Xβ + ε. The design matrix X is n × (d+1), with a first column of 1s (for the intercept β&sub0;). Each row is one observation.
The Normal Equation is the same as in simple regression: β̂ = (XᵀX)⁻¹Xᵀy. Crucially, (XᵀX) must be invertible — which requires n > d+1 and no perfect multicollinearity among the columns of X. If X has 200 rows and 3 feature columns (plus the intercept column), XᵀX is 4×4 — trivial to invert. With thousands of features, this becomes expensive: matrix inversion is O(d³).
Each coefficient βj represents the expected change in y for a one-unit increase in xj while holding all other predictors constant. This “ceteris paribus” interpretation is why multiple regression is more useful than running separate simple regressions — it isolates each predictor's effect.
When two predictors are highly correlated, their individual coefficients become unreliable — the model can't tell which is responsible for the variance. The estimate is technically correct but numerically unstable: small data changes cause large coefficient swings.
Say the answer out loud before revealing — recalling is what makes it stick. ➜ Next: R² and Adjusted R²
What does the design matrix X contain and why add a column of 1s?
How do you interpret a coefficient β₁ = 0.05 in a model with TV and radio as predictors?
Why does adding more predictors not always improve a model?
What is the computational cost of OLS and when does it become prohibitive?