  This paper enhances and develops bridges between statistics, mechanics, and
geometry. For a given system of points in $\mathbb R^k$ representing a sample
of a full rank, we construct an explicit pencil of confocal quadrics with the
following properties: (i) All the hyperplanes for which the hyperplanar moments
of inertia for the given system of points are equal, are tangent to one of the
quadrics from the pencil of quadrics. As an application, we develop
regularization procedures for the orthogonal least square method, analogues of
lasso and ridge methods from linear regression. (ii) For any given point $P$
among all the hyperplanes that contain it, the best fit is the tangent
hyperplane to the quadric from the confocal pencil corresponding to the maximal
Jacobi coordinate of the point $P$; the worst fit among the hyperplanes
containing $P$ is the tangent hyperplane to the ellipsoid from the confocal
pencil that contains $P$. The confocal pencil of quadrics provides a universal
tool to solve the restricted principal component analysis restricted at any
given point. Both results (i) and (ii) can be seen as generalizations of the
classical result of Pearson on orthogonal regression. They have natural and
important applications in the statistics of the measurement error models (EIV).
For the classical linear regressions we provide a geometric characterization of
hyperplanes of least squares in a given direction among all hyperplanes which
contain a given point. The obtained results have applications in restricted
regressions, both ordinary and orthogonal ones. For the latter, a new formula
for test statistic is derived. The developed methods and results are
illustrated in natural statistics examples.
