AI.Cruz - Learn AI Through Interactive Visualizations

MACHINE LEARNING

R² and Adjusted R²

R² measures explained variance — but always increases when you add predictors. Adjusted R² penalises complexity: it only improves if a new predictor earns its place.

1What R² Measures

The coefficient of determination R² answers: how much better is our model than just predicting the mean? It decomposes total variance into explained and unexplained parts.

R² = 1 - SS_res / SS_tot, where SS_tot = Σ(y_i - ŷ)² is total variance and SS_res = Σ(y_i - ŷ_i)² is unexplained variance. R² = 0 means the model predicts only the mean. R² = 1 means perfect prediction. R² can be negative for models that perform worse than the mean baseline.

SS_tot (total)
Variance if you always predict ŷ. Benchmark — how bad the dumbest model is.

SS_res (residual)
Unexplained variance after fitting. SS_res decreases (never increases) as features are added.

SS_reg (explained)
SS_tot - SS_res. How much the model reduces variance vs. the mean. R² = SS_reg/SS_tot.

2The R² Inflation Problem

R² never decreases when predictors are added — even if you add random noise as a feature. This makes it a poor metric for comparing models with different numbers of predictors.

Advertising dataset example: TV alone gives R² = 0.61. Adding radio jumps to R² = 0.897 — radio genuinely helps. Adding newspaper gives R² = 0.897 — almost identical. But R² still increases by a tiny amount. Without a penalty, you might keep newspaper just because it technically raised R².

Why this happens

When a predictor is added, OLS optimises the coefficients to reduce SS_res. Even a random predictor has some accidental correlation with the target in the training sample. SS_res can only decrease, so R² can only increase. The fix is to penalise the degrees of freedom consumed.

3Adjusted R²

Adjusted R² accounts for the number of predictors d and sample size n. It increases only if the new predictor improves the model more than expected by chance.

R²_adj = 1 - (SS_res/(n-d-1)) / (SS_tot/(n-1)) = 1 - (1-R²)(n-1)/(n-d-1). As d increases with n fixed, the penalty (n-d-1) shrinks the denominator, making the fraction larger and R²_adj smaller — unless SS_res drops enough to compensate.

Advertising results

TV only: R² = 0.610, R²_adj = 0.608
TV + radio: R² = 0.897, R²_adj = 0.896
TV + radio + newspaper: R² = 0.897, R²_adj = 0.895

Adding newspaper raised R² but lowered adjusted R² — correctly identifying it as useless.

Limitations of both

Both measure fit on training data — not out-of-sample performance
Neither detects overfitting caused by nonlinear effects
For model selection, prefer cross-validated RMSE or AIC/BIC
R² < 0 is valid — it means the model is worse than the mean

4Other Regression Metrics

R² is useful for communicating model quality to non-technical audiences, but practitioners also look at error-scale metrics.

MSE / RMSE
Mean squared / root mean squared error. RMSE is in the same units as y — directly interpretable. Sensitive to large outliers because errors are squared.

MAE
Mean absolute error. More robust to outliers than MSE because it doesn't square residuals. Trade-off: MAE is not differentiable at zero, which can complicate gradient-based fitting.

Relationship
R² = 1 - MSE/Var(y). So R² is a normalised MSE. A high-variance target can have low MSE and still yield a mediocre R².

Regression Metrics →Multiple Linear Regression →Ridge and Lasso →

R² and Adjusted R² — computed from scratch with numpy

import numpy as np

np.random.seed(0)
n = 200
TV      = np.random.uniform(0, 300, n)
radio   = np.random.uniform(0, 50, n)
newspaper = np.random.uniform(0, 100, n)
sales = 3 + 0.05 * TV + 0.19 * radio + np.random.normal(0, 2, n)

def ols_fit(X, y):
    return np.linalg.inv(X.T @ X) @ X.T @ y

def metrics(X, y, label):
    beta = ols_fit(X, y)
    y_hat = X @ beta
    n_obs = len(y)
    d = X.shape[1] - 1   # number of features (exclude intercept)
    ss_res = np.sum((y - y_hat) ** 2)
    ss_tot = np.sum((y - np.mean(y)) ** 2)
    mse  = ss_res / n_obs
    rmse = mse ** 0.5
    mae  = np.mean(np.abs(y - y_hat))
    r2   = 1 - ss_res / ss_tot
    r2_adj = 1 - (ss_res / (n_obs - d - 1)) / (ss_tot / (n_obs - 1))
    print(label)
    print("  R^2:          " + str(round(r2, 4)))
    print("  Adjusted R^2: " + str(round(r2_adj, 4)))
    print("  RMSE:         " + str(round(rmse, 4)))
    print("  MAE:          " + str(round(mae, 4)))
    print()

ones = np.ones(n)
metrics(np.column_stack([ones, TV]), sales, "TV only")
metrics(np.column_stack([ones, TV, radio]), sales, "TV + radio")
metrics(np.column_stack([ones, TV, radio, newspaper]), sales, "TV + radio + newspaper")

# Adding pure noise feature to show R^2 inflation
noise = np.random.randn(n)
metrics(np.column_stack([ones, TV, radio, newspaper, noise]), sales,
        "TV + radio + newspaper + noise")
print("Adj R^2 drops when noise is added — R^2 never does.")