Ordinary least squares
Projection

For mathematicians, OLS is an approximate solution to an overdetermined system of linear equations Xβ ≈ y, where β is the unknown. Assuming the system cannot be solved exactly (the number of equations n is much larger than the number of unknowns p), we are looking for a solution that could provide the smallest discrepancy between the right- and left- hand sides. In other words, we are looking for the solution that satisfies

where ‖ · ‖ is the standard L norm in the n -dimensional Euclidean space R. The predicted quantity Xβ is just a certain linear combination of the vectors of regressors. Thus, the residual vector y − Xβ will have the smallest length when y is projected orthogonally onto the linear subspace spanned by the columns of X. The OLS estimator β ^ {\displaystyle {\hat {\beta }}} in this case can be interpreted as the coefficients of vector decomposition of y = Py along the basis of X.

In other words, the gradient equations at the minimum can be written as:

A geometrical interpretation of these equations is that the vector of residuals, y − X β ^ {\displaystyle \mathbf {y} -X{\hat {\boldsymbol {\beta }}}} is orthogonal to the column space of X, since the dot product (y − X β ^) ⋅ X v {\displaystyle (\mathbf {y} -\mathbf {X} {\hat {\boldsymbol {\beta }}})\cdot \mathbf {X} \mathbf {v} } is equal to zero for any conformal vector, v. This means that y − X β ^ {\displaystyle \mathbf {y} -\mathbf {X} {\boldsymbol {\hat {\beta }}}} is the shortest of all possible vectors y − X β {\displaystyle \mathbf {y} -\mathbf {X} {\boldsymbol {\beta }}}, that is, the variance of the residuals is the minimum possible. This is illustrated at the right.

Introducing γ ^ {\displaystyle {\hat {\boldsymbol {\gamma }}}} and a matrix K with the assumption that a matrix {\displaystyle } is non-singular and K X = 0 (cf. Orthogonal projections), the residual vector should satisfy the following equation:

The equation and solution of linear least squares are thus described as follows:

Another way of looking at it is to consider the regression line to be a weighted average of the lines passing through the combination of any two points in the dataset. Although this way of calculation is more computationally expensive, it provides a better intuition on OLS.
length_constraints:number_words[217]