1What OLS Assumes
Ordinary Least Squares (OLS) finds parameters β̂ = (XᵀX)⁻¹Xᵀy that minimise the sum of squared residuals. The estimator is only unbiased and efficient when these five assumptions hold:
Linearity in parameters
The model must be linear in β, not necessarily in x. y = β&sub0; + β&sub1;x² is linear in parameters (OK). y = β&sub0; + β&sub1;²x is not (breaks OLS).
Homoscedasticity
Residuals must have constant variance across all values of x. If variance grows with x (heteroscedasticity), standard errors are wrong and inference fails.
No autocorrelation
Error terms must be independent of each other — IID (Independently and Identically Distributed). Time series data frequently violates this: each observation depends on the previous one.
No perfect multicollinearity
Input variables must not be perfectly linearly correlated. If area = length × breadth and all three are in the model, XᵀX is not invertible and β̂ cannot be computed.
Random sampling
Observations must be randomly sampled. Biased sampling (e.g., only expensive houses in a price model) produces estimates that don't generalise. Also requires n > d+1 (more observations than parameters).
2Checking Each Assumption
Each assumption has a diagnostic. Not checking them before trusting coefficients is one of the most common modelling mistakes.
Residual vs Fitted plot
A random scatter around zero indicates linearity and homoscedasticity. A funnel shape signals heteroscedasticity. A curve signals non-linearity.
Durbin-Watson test
Tests for autocorrelation in residuals. Value near 2 = no autocorrelation. Near 0 = positive autocorrelation. Near 4 = negative autocorrelation.
Variance Inflation Factor
VIF > 5 warns of multicollinearity. VIF > 10 is severe. Drop or combine correlated features. Use PCA or Ridge regression if multicollinearity is unavoidable.
3Parameter Estimation: OLS vs MLE
OLS minimises the Sum of Squared Errors (SSE). Maximum Likelihood Estimation (MLE) maximises the conditional probability of observing the data given parameters. When errors are Gaussian, both methods yield the same solution — OLS is a special case of MLE under normality.
OLS derivation: minimise SSE = Σ(yᵢ − ŷᵢ)². Take derivative with respect to β and set to zero. This gives the Normal Equation: β̂ = (XᵀX)⁻¹Xᵀy. No iteration required — it is a closed-form solution. The Gauss-Markov theorem proves it is the Best Linear Unbiased Estimator (BLUE) when assumptions hold.
OLS strengths
- + Closed-form: no iterations, exact solution
- + BLUE under Gauss-Markov conditions
- + Interpretable coefficients
OLS limitations
- ✗ Fails when XᵀX is not invertible (multicollinearity or n < d+1)
- ✗ O(d³) matrix inversion — slow for large feature spaces
- ✗ No regularisation — prone to overfitting
4When Assumptions Break
Violations don't always mean you should abandon linear regression — they determine which remedies to apply.
Heteroscedasticity fix
Apply Weighted Least Squares (WLS) or transform the target variable (log, sqrt). WLS assigns lower weight to high-variance observations.
Autocorrelation fix
Use GLS or ARIMA models for time series. Include lagged features or switch to a time-aware estimator that models the correlation structure.
Multicollinearity fix
Drop redundant features, apply PCA to decorrelate, or use Ridge regression which adds L2 penalty to stabilise (XᵀX + λI)⁻¹.