Regression Analysis

Historical Context

The method of least squares was independently invented by Adrien-Marie Legendre in 1805 and Carl Friedrich Gauss around 1795 (published in 1809). Gauss used it to predict the orbit of the asteroid Ceres from sparse observations—a triumph that cemented the method’s reputation. The term “regression” itself was coined by Francis Galton in 1886, who observed that children’s heights tended to “regress” toward the population mean relative to their parents’ heights.

The modern matrix formulation of multiple regression was developed by R.A. Fisher and others in the early twentieth century, while the Gauss-Markov theorem (proved by Gauss and formalized by Andrey Markov) established the optimality of ordinary least squares. The generalization to non-normal responses through generalized linear models (GLMs) came in 1972 with the seminal paper by John Nelder and Robert Wedderburn, unifying logistic regression, Poisson regression, and classical linear models within a single framework based on exponential family distributions and link functions.

2.1 Simple Linear Regression and OLS

The simple linear regression model posits that the conditional mean of $Y$ given $x$ is a linear function:

Under the model assumptions, the OLS estimators are unbiased with $\text{Var}(\hat{\beta}_1) = \sigma^2 / S_{xx}$. If we further assume $\varepsilon_i \sim N(0, \sigma^2)$, the OLS estimators coincide with the maximum likelihood estimators, and the $t$-statistic $t = \hat{\beta}_1 / \text{SE}(\hat{\beta}_1)$ follows a $t_{n-2}$ distribution under $H_0: \beta_1 = 0$.

2.2 Multiple Regression in Matrix Form

With $p$ predictors, the model becomes $\mathbf{Y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}$, where $\mathbf{X}$ is the $n \times (p+1)$ design matrix (including an intercept column of ones) and $\boldsymbol{\varepsilon} \sim (0, \sigma^2 \mathbf{I}_n)$.

The residual vector is $\mathbf{e} = (\mathbf{I} - \mathbf{H})\mathbf{Y}$, and the unbiased variance estimate is $s^2 = \mathbf{e}^\top\mathbf{e}/(n - p - 1)$. The covariance of $\hat{\boldsymbol{\beta}}$ is $\sigma^2(\mathbf{X}^\top\mathbf{X})^{-1}$, estimated by $s^2(\mathbf{X}^\top\mathbf{X})^{-1}$.

2.3 The Gauss-Markov Theorem

The Gauss-Markov theorem is powerful but has important limitations. It applies only to linear, unbiased estimators. A biased estimator such as ridge regression can achieve lower MSE by trading bias for reduced variance—this is the bias-variance tradeoff that motivates regularization methods.

When the error covariance is $\text{Cov}(\boldsymbol{\varepsilon}) = \sigma^2 \boldsymbol{\Omega}$ for known $\boldsymbol{\Omega} \neq \mathbf{I}$ (heteroscedasticity or correlation), the BLUE becomes the generalized least squares (GLS) estimator:

2.4 Regression Diagnostics

Fitting a model is only half the work; we must also verify that the model assumptions hold. Key diagnostic tools include residual analysis, leverage, and influence measures.

2.5 Generalized Linear Models

GLMs extend regression to non-normal responses by specifying three components: a random component (an exponential family distribution for $Y$), a systematic component (the linear predictor $\eta = \mathbf{X}\boldsymbol{\beta}$), and a link function $g$ such that $g(\mu) = \eta$, where $\mu = E[Y]$.

For logistic regression, the coefficient $\beta_j$ has a natural interpretation: a unit increase in $x_j$ multiplies the odds $p/(1-p)$ by $e^{\beta_j}$. The deviance $D = -2(\ell(\hat{\boldsymbol{\beta}}) - \ell_{\text{saturated}})$ plays the role that RSS plays in linear regression, and deviance residuals are the GLM analogue of ordinary residuals.

Computational Lab: Regression Diagnostics

We fit an OLS model, examine residual plots and leverage, detect influential points via Cook’s distance, and fit a logistic regression with a decision boundary.