  Estimating the number of signals embedded in noise is a fundamental problem
in signal processing. As a classic estimator based on random matrix theory
(RMT), the RMT estimator estimates the number of signals via sequentially
testing the likelihood of an eigenvalue as arising from a signal or noise for a
given over-detection probability. However, it tends to under-estimate the
number of signals as weak signal eigenvalues may be immersed in the
non-negligible bias term among eigenvalues for finite sample size. In order to
solve this problem, we propose an RMT estimator with adaptive decision
criterion (termed as RMT-ADC estimator) by adaptively incorporating the bias
term into the decision criterion of the RMT estimator. Firstly, we analyze the
effect of this bias term among eigenvalues on the estimation performance of the
RMT estimator. Then, we derive both the decreased over-estimation probability
and the increased under-estimation probability of the RMT estimator incurred by
the bias term when assuming the eigenvalue being tested is arising from a
signal, and also derive the increased under-estimation probability of the RMT
estimator incurred by the bias term when assuming the eigenvalue being tested
is arising from noise. Based on these results, the RMT-ADC estimator can
adaptively determine whether the noise variance should be estimated under the
assumption that the eigenvalue being tested is arising from a signal or from
noise, and thus can adaptively select its decision criterion. Moreover, the
RMT-ADC estimator can adaptively determine whether the bias term among
eigenvalues should be incorporated into the selected decision criterion or not.
Therefore, the RMT-ADC estimator can avoid the higher under-estimation
probability of the RMT estimator. Finally, simulation results are presented to
show that the proposed RMT-ADC estimator significantly outperforms the existing
estimators.
