  Regression analysis based on many covariates is becoming increasingly common.
However, when the number of covariates $p$ is of the same order as the number
of observations $n$, maximum likelihood regression becomes unreliable due to
overfitting. This typically leads to systematic estimation biases and increased
estimator variances. It is crucial for inference and prediction to quantify
these effects correctly. Several methods have been proposed in literature to
overcome overfitting bias or adjust estimates. The vast majority of these focus
on the regression parameters. But failure to estimate correctly also the
nuisance parameters may lead to significant errors in confidence statements and
outcome prediction.
  In this paper we present a jacknife method for deriving a compact set of
non-linear equations which describe the statistical properties of the ML
estimator in the regime where $p=O(n)$ and under the hypothesis of normally
distributed covariates. These equations enable one to compute the overfitting
bias of maximum likelihood (ML) estimators in parametric regression models as
functions of $\zeta = p/n$. We then use these equations to compute shrinkage
factors in order to remove the overfitting bias of maximum likelihood (ML)
estimators. This new derivation offers various benefits over the replica
approach in terms of increased transparency and reduced assumptions. To
illustrate the theory we performed simulation studies for multiple regression
models. In all cases we find excellent agreement between theory and
simulations.
