quality:corruption, stylistic:style_transfer, linguistic:voice_change, structural:scholar_edition
Least-squares estimation and related techniques

Assuming that the independent variable is x i → = {\displaystyle {\vec {x_{i}}}=\left} and the model's parameters are β → = {\displaystyle {\vec {\beta }}=\left}, then the model's prediction would be

If x i → {\displaystyle {\vec {x_{i}}}} is extended to x i → = {\displaystyle {\vec {x_{i}}}=\left} then y i {\displaystyle y_{i}} would become a dot product of the parameter and the independent variable, i.e.

In the least-squares setting, the optimum parameter is defined as such that minimizes the sum of mean squared loss:

Now putting the independent and dependent variables in matrices X {\displaystyle X} and Y {\displaystyle Y} respectively, the loss function can be rewritten as:

As the loss is convex the optimum solution lies at gradient zero. The gradient of the loss function is (using Denominator layout convention):

Setting the gradient to zero produces the optimum parameter:

Note: To prove that the β ^ {\displaystyle {\hat {\beta }}} obtained is indeed the local minimum, one needs to differentiate once more to obtain the Hessian matrix and show that it is positive definite. This is provided by the Gauss–Markov theorem.

Linear least squares methods include mainly: