  Covariance matrices are fundamental to the analysis and forecast of economic,
physical and biological systems. Although the eigenvalues $\{\lambda_i\}$ and
eigenvectors $\{{\bf u}_i\}$ of a covariance matrix are central to such
endeavors, in practice one must inevitably approximate the covariance matrix
based on data with finite sample size $n$ to obtain empirical eigenvalues
$\{\tilde{\lambda}_i\}$ and eigenvectors $\{\tilde{{\bf u}}_i\}$, and therefore
understanding the error so introduced is of central importance. We analyze
eigenvector error $\|{\bf u}_i - \tilde{{\bf u}}_i \|^2$ while leveraging the
assumption that the true covariance matrix having size $p$ is drawn from a
matrix ensemble with known spectral properties---particularly, we assume the
distribution of population eigenvalues weakly converges as $p\to\infty$ to a
spectral density $\rho(\lambda)$ and that the spacing between population
eigenvalues is similar to that for the Gaussian orthogonal ensemble. Our
approach complements previous analyses of eigenvector error that require the
full set of eigenvalues to be known, which can be computationally infeasible
when $p$ is large. To provide a scalable approach for uncertainty
quantification of eigenvector error, we consider a fixed eigenvalue $\lambda$
and approximate the distribution of the expected square error $r=
\mathbb{E}\left[\| {\bf u}_i - \tilde{{\bf u}}_i \|^2\right]$ across the matrix
ensemble for all ${\bf u}_i$ associated with $\lambda_i=\lambda$. We find, for
example, that for sufficiently large matrix size $p$ and sample size $n>p$, the
probability density of $r$ scales as $1/nr^2$. This power-law scaling implies
that eigenvector error is extremely heterogeneous---even if $r$ is very small
for most eigenvectors, it can be large for others with non-negligible
probability. We support this and further results with numerical experiments.
