Linear model

From Wikipedia, the free encyclopedia

Not to be confused with linear model of innovation.

In statistics, given a (random) sample $(Y_i, X_{i1}, \ldots, X_{ip}), \, i = 1, \ldots, n$ the most general form of linear model is formulated as

$Y_i = \beta_0 + \beta_1 \phi_1(X_{i1}) + \ldots + \beta_p \phi_p(X_{ip}) + \varepsilon_i \qquad i = 1, \ldots, n$

where $\phi_1, \ldots, \phi_p$ may be nonlinear functions.

In matrix notation this model can be written as

$Y = X \beta + \varepsilon$

where Y is an n × 1 column vector, X is an n × (p + 1) matrix, β is a (p + 1) × 1 vector of (unobservable) parameters, and ε is an n × 1 vector of errors, which are uncorrelated random variables each with expected value 0 and variance σ². Note that depending on the context the sample can be seen as fixed (observable), or random.

Much of the theory of linear models is associated linear regression, especially with estimating the values of the parameters β and σ². Typically, this is done using the least-squares method, which has minimum variance among mean-unbiased estimators, according to the Gauss-Markov theorem.

If the error is known to follow a normal distribution, then the method of maximum likelihood can be used, which agrees with the linear least squares method for estimating the mean parameter. The methods disagree for estimating the variance, for which the maximum-likelihood method is biased.

[edit] Methods of inference

[edit] Maximum likelihood estimation of coefficients

[edit] Multivariate normal errors

The method of maximum-likelihood requires that the statistician know a parametric family of probability distributions. The most common assumption takes the components of the vector of errors to be independent and normally distributed, giving Y a multivariate normal distribution with mean Xβ and co-variance matrix σ² I, where I is the identity matrix.

Having observed the values of X and Y, the statistician must estimate β and σ².

[edit] β

The log-likelihood function (for $ε i$ independent and normally distributed) is

$l(\beta, \sigma^2) = -\frac{n}{2} \log (2 \pi \sigma^2) - \frac{1}{2\sigma^2} \sum_{i = 1}^n \left(Y_i - X_i^{\top} \beta \right)^2$

where $X_i^{\top}$ is the ith row of X. Differentiating with respect to β_j, we get

$\frac{\partial l}{\partial \beta_j} = \frac{1}{\sigma^2} \sum_{i = 1}^n X_{ij} \left( Y_i - X_i^{\top} \beta \right)$

so setting this set of p equations to zero and solving for β gives

$X^{\top} X \hat{\beta} = X^{\top} Y.$

Now, using the assumption that X has rank p, we can invert the matrix on the left hand side to give the maximum likelihood estimate for β:

$\hat{\beta} = (X^{\top} X)^{-1} X^{\top} Y$ .

We can check that this is a maximum by looking at the Hessian matrix of the log-likelihood function.

[edit] σ²

By setting the right hand side of

$\frac{\partial l}{\partial \sigma^2} = -\frac{n}{2\sigma^2} + \frac{1}{2 \sigma^4} \sum_{i=1}^n \left(Y_i - X_i^{\top} \beta \right)^2$

to zero and solving for σ² we find that

$\hat{\sigma}^2 = \frac{1}{n} \sum_{i=1}^n \left(Y_i - X_i^{\top} \hat{\beta} \right)^2 = \frac{1}{n} \| Y - X \hat{\beta} \|^2.$

[edit] Maximum likelihood estimation is biased

Since we have that Y follows a multivariate normal distribution with mean Xβ and co-variance matrix σ² I, we can deduce the distribution of the MLE of β:

$\hat{\beta} = (X^{\top} X)^{-1} X^{\top} Y \sim N_p (\beta, (X^{\top}X)^{-1} \sigma^2 ).$

So this estimate is unbiased for β, and we can show that this variance achieves the Cramér-Rao bound.

A more complicated argument^[1] shows that

$\hat{\sigma}^2 \sim \frac{\sigma^2}{n} \chi^2_{n-p};$

since a chi-squared distribution with n − p degrees of freedom has mean n − p, which is biased. However, for large samples, this bias is small and as the sample-size grows the bias approaches zero.

[edit] Best linear unbiased estimation (BLUE)

The least-squares estimator is often used to estimate the coefficients of a linear regression. The least-squares method minimizes the sum of the square of the residuals.

$\widehat{\theta}_{LS}=(\mathbf{X}^t\mathbf{X})^{-1}\mathbf{X}^t\vec{Y}$

We conclude by giving some qualities of this estimator and a geometrical interpretation.

[edit] Assumptions

For $p\in\mathbb{N}^+$ , let Y be a random variable taking values in $\mathbb{R}$ , we call observation.

We next define the function η, linear in $θ$ :

$\eta(X;\theta)=\sum_{j=1}^p \theta_j X_j,$

where

For $j\in \{1,...,p\}$ , $X j$ is a random variable taking values in $\mathbb{R}$ and is called a factor and
$θ j$ is a scalar, for $j\in \{1,...,p\}$ , and $\theta^t=(\theta_1,\cdots,\theta_p)$ , where $θ t$ denotes the transpose of vector $θ$ .

Let $X^t=(X_1,\cdots,X_p)$ . We can write $η(X;θ) = X t θ$ . Define the error to be:

$\varepsilon(\theta)=Y-X^t\theta$

We suppose that there exists a true parameter $\overline{\theta}\in\mathbb{R}^{p}$ such that $\mathbb{E}[\varepsilon(\overline{\theta})|X]=0$ . This means that, given the random variables $(X_1,\cdots,X_p)$ , the best prediction we can make of Y is $Y=\eta(X;\overline{\theta})=X^t\overline{\theta}$ . Henceforth, $\varepsilon$ will denote $\varepsilon(\overline{\theta})$ and η will represent $\eta(X;\overline{\theta})$ .

[edit] Least-squares estimator

The idea behind the least-squares estimator is to see linear regression as an orthogonal projection. Let F be the L2-space of all random variables whose square has a finite Lebesgue integral. Let G be the linear subspace of $F$ generated by $X_1,\cdots,X_p$ (supposing that $Y\in F$ and $(X_1,\cdots,X_p)\in F^p$ ). We show in this paragraph that the function $η$ is an orthogonal projection of Y on G and we will construct the least-squares estimator.

[edit] Seeing linear regression as an orthogonal projection

We have $\mathbb{E}(Y|X)=\eta$ , but $Y\mapsto\mathbb{E}(Y|X)$ is a projection, which means that $η$ is a projection of Y on G. What is more, this projection is an orthogonal one.

To see this, we can build a scalar product in F: for all couples of random variables $X,Y\in F$ , we define $\langle X,Y\rangle_2:=\mathbb{E}[X Y]$ . It is indeed a scalar product because if $\|X\|_2^2=0$ , then $X = 0$ almost everywhere (where $\|X\|_2^2:=\langle X,X\rangle_2$ is the norm corresponding to this scalar product).

For all $1\leq j\leq p$ ,

$\begin{align} \langle X_j,\varepsilon \rangle_2 & = \langle X_j,Y-X^t \overline{\theta}\rangle_2 \\ & =\langle X_j,Y\rangle_2-\langle X_j,\mathbb{E}[Y|X]\rangle_2 \\ & =\mathbb{E}[X_j Y] - \mathbb{E}[X_j \mathbb{E}[Y|X]] \\ & =X_j(\mathbb{E}Y-\mathbb{E}[\mathbb{E}[Y|X]]) \\ & =X_j(\mathbb{E}Y - \mathbb{E}Y) \\ \langle X_j,\varepsilon \rangle_2 & =0 \end{align}$

Therefore, $\varepsilon$ is orthogonal to any $X j$ and hence to the whole of the subspace G, which means that $η$ is a projection of Y on G, orthogonal with respect to the scalar product we have just defined. We have therefore shown:

$\eta(X;\overline{\theta})=\min_{f\in G}\|Y-f\|^2_2.$

[edit] Estimating the coefficients

If, for each $j\in\{1,\cdots,p\}$ we have a sample of size $n>p, (X^1_j,\cdots,X^n_j)$ of $X j$ , along with a vector $\vec{Y}$ of n observations of Y, we can build an estimation of the coefficients of this orthogonal projection. To do this, we can use an estimation of the scalar product defined earlier.

For all couples of samples of size n $\vec{U},\vec{V}\in F^n$ of random variables U and V, we define $\langle \vec{U},\vec{V}\rangle:=\vec{U}^t \vec{V}$ , where $\vec{U}^t$ is the transpose of vector $\vec{U}$ , and $\|\cdot\|:=\sqrt{\langle \cdot,\cdot\rangle}$ . Note that the scalar product $\langle \cdot,\cdot\rangle$ is defined in $F n$ and no longer in F.

Let us define the design matrix (or random design), a $n\times p$ random matrix: $\mathbf{X}=\left[\begin{matrix}X_1^1&\cdots&X^1_p\\\vdots&&\vdots\\X^n_1&\cdots&X^n_p\end{matrix}\right]$

We can now adapt the minimization of the sum of the residuals: the least-squares estimator $\widehat{\theta}_{LS}$ will be the value, if it exists, of $θ$ which minimizes $\|\mathbf{X}\theta-\vec{Y}\|^2$ . Therefore, $\langle \mathbf{X},\vec{\varepsilon}(\widehat{\theta}_{LS})\rangle= \mathbf{X}^t(\mathbf{X}\widehat{\theta}_{LS}-\vec{Y})=0$ .

This yields $\mathbf{X}^t \mathbf{X} \widehat{\theta}_{LS} = \mathbf{X}^t \vec{Y}$ . If $\mathbf{X}$ is of full rank, then so is $\mathbf{X}^t \mathbf{X}$ . In that case we can compute the least-squares estimator explicitly by inverting the $p\times p$ matrix $\mathbf{X}^t\mathbf{X}$ :

$\widehat{\theta}_{LS}=(\mathbf{X}^t\mathbf{X})^{-1} \mathbf{X}^t \vec{Y}$

[edit] Qualities and geometrical interpretation

[edit] Qualities of this estimator

The Gauss-Markov theorem states that the least-square estimators is the best linear unbiased estimator (BLUE) of $\overline{\theta}$ .

The vector of errors $\vec{\varepsilon}=\vec{Y}-\mathbf{X}\overline{\theta}$ is said to fulfil the Gauss-Markov assumptions if:

$\mathbb{E}\vec{\varepsilon}=\vec{0}$
$\mathbb{V}\vec{\varepsilon}=\sigma^2 \mathbf{I}_n$ (uncorrelated but not necessarily independent; homoscedastic but not necessarily identically distributed)

where $\sigma^2<+\infty$ and $\mathbf{I}_n$ is the $n\times n$ identity matrix.

This decisive advantage has led to a sometimes abusive use of least-squares. Least-squares depends on the fulfilment of the Gauss-Markov hypothesis and applying this method in a situation where these conditions are not met can lead to inaccurate results. For example, in the study of time-series, it is often difficult to assume independence of the residuals.

[edit] Geometrical interpretation

The situation described by the linear regression problem can be geometrically seen as follows:

The least-squares is also an M-estimator of $ρ$ -type for $\rho(r):=\frac{r^2}{2}$ .

[edit] Generalizations

[edit] Generalized least squares

If, rather than taking the variance of ε to be σ²I, where I is the n×n identity matrix, one assumes the variance is Ω, where Ω is a known matrix other than the identity matrix, then one estimates β by the method of "generalized least squares", in which, instead of minimizing the sum of squares of the residuals (the squared euclidean length of the residual), one minimizes the squared Mahalanobis Length of the residual vector:

${\min_{\beta}}\left(y-X\beta\right)^{\top}\Omega^{-1}\left(y-X\beta\right).$

This has the effect of "de-correlating" normal errors, and leads to the estimator

$\widehat{\beta}=\left(X^{\top}\Omega^{-1}X\right)^{-1}X^{\top}\Omega^{-1}y$

which is the best linear unbiased estimator for $β$ . If all of the off-diagonal entries in the matrix Ω are 0, then one normally estimates β by the method of weighted least squares, with weights proportional to the reciprocals of the diagonal entries. The GLS estimator is also known as the Aitken estimator, after Alexander Aitken, the Professor in the University of Otago Statistics Department who pioneered it.^[2]

[edit] Generalized linear models

Generalized linear models, for which rather than

E(Y) = Xβ,

one has

g(E(Y)) = Xβ,

where g is the "link function". The variance is also not restricted to being normal.

An example is the Poisson regression model, which states that Y_i has a Poisson distribution with expected value $e^{\beta_0+\beta_1x_i}$ . The link function is the natural logarithm function. Having observed x_i and Y_i for i = 1, ..., n, one can estimate $β 0$ and $β 1$ by the method of maximum likelihood.

[edit] References

^ A.C. Davidson Statistical Models. Cambridge University Press (2003).
^ Alexander Craig Aitken

[edit] See also

ANOVA, or analysis of variance, is historically a precursor to the development of linear models. Here the model parameters themselves are not computed, but X column contributions and their significance are identified using the ratios of within-group variances to the error variance and applying the F test.
Linear regression
Robust regression

[Davidson-0] A.C. Davidson Statistical Models. Cambridge University Press (2003).

[1] Alexander Craig Aitken

[1]

[2]

Linear model

From Wikipedia, the free encyclopedia

Contents

[edit] Methods of inference

[edit] Maximum likelihood estimation of coefficients

[edit] Multivariate normal errors

[edit] β

[edit] σ²

[edit] Maximum likelihood estimation is biased

[edit] Best linear unbiased estimation (BLUE)

[edit] Assumptions

[edit] Least-squares estimator

[edit] Seeing linear regression as an orthogonal projection

[edit] Estimating the coefficients

[edit] Qualities and geometrical interpretation

[edit] Qualities of this estimator

[edit] Geometrical interpretation

[edit] Generalizations

[edit] Generalized least squares

[edit] Generalized linear models

[edit] References

[edit] See also

Views

Personal tools

Navigation

Search

Interaction

Toolbox

Languages

Linear model

From Wikipedia, the free encyclopedia

Contents

[edit] Methods of inference

[edit] Maximum likelihood estimation of coefficients

[edit] Multivariate normal errors

[edit] β

[edit] σ2

[edit] Maximum likelihood estimation is biased

[edit] Best linear unbiased estimation (BLUE)

[edit] Assumptions

[edit] Least-squares estimator

[edit] Seeing linear regression as an orthogonal projection

[edit] Estimating the coefficients

[edit] Qualities and geometrical interpretation

[edit] Qualities of this estimator

[edit] Geometrical interpretation

[edit] Generalizations

[edit] Generalized least squares

[edit] Generalized linear models

[edit] References

[edit] See also

Views

Personal tools

Navigation

Search

Interaction

Toolbox

Languages

[edit] σ²