\section{Introduction}
\label{sec:intro}
Statistical inference on graphs is a burgeoning field of research in machine learning and statistics, with numerous applications to social network, neuroscience, etc. Many of the graphs in application domains are large and complex but nevertheless are believed to be composed of multiple smaller-scale communities. Thus, an essential task in graph inference is detecting/identifying local (sub)communities.
The resulting problem of community detection on graphs is well-studied (see the survey \cite{Fortunato2010Community}),
with many available techniques including those based on maximizing modularity and likelihood \cite{Bickel2009,Newman2004,Snijders1997Estimation,Celisse}, random walks \cite{pons05:_comput,rosvall08:_maps}, spectral clustering \cite{mcsherry,rohe2011spectral,chaudhuri12:_spect,sussman12,rinaldo_2013,von2007tutorial}, and semidefinite programming \cite{hajek,abbe}. It is widely known that under suitable models --- such as the popular stochastic blockmodel and its variants \cite{holland,karrer2011stochastic,lyzinski15_HSBM} --- one can consistently recover the underlying communities as the number of observed nodes increases, and furthermore, there exist deep and beautiful phase transitions phenomena with respect to the statistical and computational limits for recovery.

Another important question in graph inference is, subsequent to community detection, that of characterizing the nature and/or structure of these communities. One of the simplest and possibly most essential example is determining the (community specific) tendencies/probabilities for nodes to form link within and between communities. Consistent recovery of the underlying communities yields a straightforward and universally employed procedure for consistent estimation of these probabilities, namely averaging the number of edges within a community and/or between communities. This procedure, when intepreted in the context of estimating the parameters $\bm{\theta} = (\theta_1, \theta_2, \dots, \theta_r)$ of a collection of independent binomially distributed random variables $\{X_1,X_2, \dots, X_r\}$ with $X_i \sim \mathrm{Bin}(n_i, \theta_i)$, corresponds to maximum likelihood estimation with no restrictive assumption on
the $\{\theta_i\}$. However, in the context of graphs and communities, there are often natural relationships among the communities; hence a graph with $K$ communities need not require $K(K+1)/2$ parameters to describe the within and between community connection probabilities. The above procedure is therefore potentially sub-optimal.

Motivated by the above observation, our paper studies the asymptotic properties of three different estimators for $\mathbf{B}$, the matrix of edge probabilities among communities, of a stochastic blockmodel graph. Two estimators are based on maximum likelihood methods and the remaining estimator is based on spectral embedding. We show that, given an observed graph with adjacency matrix $\mathbf{A}$, the most commonly used estimator --- the MLE under no rank assumption on $\mathbf{B}$ --- is sub-optimal when $\mathbf{B}$ is not invertible. Moreover, when $\mathbf{B}$ is singular, the estimator based on spectral embedding $\mathbf{A}$ is often times better (smaller mean squared error) than the MLE under no rank assumption, and is almost as efficient as the asymptotically (first-order) efficient MLE whose parametrization depend on $\mathrm{rk}(\mathbf{B})$. Finally, when $\mathbf{B}$ is invertible, the three estimators are asymptotically first-order efficient.

\subsection{Background}
We now formalize the setting considered in this paper. We begin by recalling the notion of stochastic blockmodel graphs due to \cite{holland}. Stochastic blockmodel graphs and its variants, such as degree-corrected blockmodels and mixed membership models \cite{karrer2011stochastic,Airoldi2008} are the most popular models for graphs with intrinsic community structure. In addition, they are widely used as building blocks for constructing approximations (see e.g., \cite{gao,wolfe13:_nonpar,airoldi13:_stoch,klopp}) of the more general latent position graphs or graphons models \cite{Hoff2002,lovasz12:_large}.
\label{subsec:background}
\begin{definition}
  \label{def:SBM}
  Let $K \geq 1$ be a positive integer and let $\bm{\pi} \in \mathcal{S}_{K-1}$ be a non-negative vector in $\mathbb{R}^{K}$
  with $\sum_{k} \pi_k = 1$; here $\mathcal{S}_{K-1}$ denote the $K-1$ dimensional simplex in $\mathbb{R}^{K}$. Let $\mathbf{B} \in [0,1]^{K \times K}$
  be symmetric. We say that $(\mathbf{A}, \bm{\tau}) \sim
  \mathrm{SBM}(\mathbf{B}, \bm{\pi})$ with sparsity factor $\rho$ if the following hold. First
  $\bm{\tau} = (\tau_1, \dots, \tau_n)$ where $\tau_i$ are
  i.i.d. with $\Pr[\tau_i = k] = \pi_k$. Then
  $\mathbf{A} \in \{0,1\}^{n \times n}$ is a symmetric matrix such
  that, conditioned on $\bm{\tau}$, for all $i \leq j$ the $\mathbf{A}_{ij}$
  are independent Bernoulli random variables with $\mathbb{E}[\mathbf{A}_{ij}] = \rho \mathbf{B}_{\tau_i, \tau_j}$. We write $\mathbf{A} \sim \mathrm{SBM}(\mathbf{B}, \bm{\pi})$ when only $\mathbf{A}$ is observed, i.e., $\bm{\tau}$ is integrated out from $(\mathbf{A}, \bm{\tau})$.
\end{definition}
For $(\mathbf{A}, \bm{\tau}) \sim \mathrm{SBM}(\mathbf{B}, \bm{\pi})$ or $\mathbf{A} \sim \mathrm{SBM}(\mathbf{B}, \bm{\pi})$
with $\mathbf{A}$ having $n$ vertices and (known) sparsity factor $\rho$,
the likelihood of $(\mathbf{A}, \tau)$ and $\mathbf{A}$ are, respectively
\begin{gather} L(\mathbf{A}, \tau; \mathbf{B}, \bm{\pi}) = \Bigl(\prod_{i=1}^{n} \pi_{\tau_i} \Bigr) \Bigl(\prod_{i \leq j} (\rho \mathbf{B}_{\tau_i, \tau_j})^{\mathbf{A}_{ij}} (1 - \rho \mathbf{B}_{\tau_i,\tau_j})^{1 - \mathbf{A}_{ij}} \Bigr), \\
 L(\mathbf{A}; \mathbf{B}, \bm{\pi}) = \sum_{\bm{\tau} \in [K]^{n}} L(\mathbf{A}, \bm{\tau}; \mathbf{B}, \bm{\pi}). \end{gather}
When we observed $\mathbf{A} \sim \mathrm{SBM}(\mathbf{B}, \bm{\pi})$ with $\mathbf{B} \in [0,1]^{K}$ where $\rho$ and $K$ are assumed known,
a maximum likelihood estimate of $\bm{\pi}$ and $\mathbf{B}$ is given by
\begin{equation}
\label{eq:MLE_naive}
 (\hat{\mathbf{B}}^{(N)}, \hat{\bm{\pi}}) = \operatornamewithlimits{argmax}_{\mathbf{B} \in [0,1]^{K \times K}, \,\, \bm{\pi} \in \mathcal{S}_{K-1}}
L(\mathbf{A}; \mathbf{B}, \bm{\pi}).
\end{equation}
The maximum likelihood estimate (MLE) $\hat{\mathbf{B}}^{(N)}$ in Eq.~\eqref{eq:MLE_naive}
requires estimation of the $K(K+1)/2$ entries of $\mathbf{B}$ and is generally intractable as it requires marginalization over the latent vertex-to-block assignment vector $\bm{\tau}$. Another parametrization of $\mathbf{B}$ is via the eigendecomposition $\mathbf{B} = \mathbf{V} \mathbf{D} \mathbf{V}^{\top}$ where $d = \mathrm{rk}(\mathbf{B})$, $\mathbf{V}$ is a $K \times d$ matrix with $\mathbf{V}^{\top} \mathbf{V} = \mathbf{I}$ and $\mathbf{D}$ is diagonal.
This parametrization results in the estimation of $d(2K - d+1)/2 \leq K(K+1)/2$ parameters, with $Kd - d(d+1)/2$ parameters being estimated for $\mathbf{V}$ (as an element of the Stiefel manifold of orthonormal $d$ frames in $\mathbb{R}^{K}$) and $d$ parameters estimated for $\mathbf{D}$. Therefore, when $d = \mathrm{rk}(\mathbf{B})$ (in addition to $\rho$ and $K$) is also assumed known, another MLE of $\mathbf{B}$ and $\bm{\pi}$ is given by
\begin{equation}
\label{eq:MLE_rank}
 (\hat{\mathbf{B}}^{(M)}, \hat{\bm{\pi}}) = \operatornamewithlimits{argmax}_{\mathbf{B} \in [0,1]^{K \times K},\,\, \mathrm{rk}(\mathbf{B}) = d, \,\, \bm{\pi} \in \mathcal{S}_{K-1}}
L(\mathbf{A}; \mathbf{B}, \bm{\pi}).
\end{equation}
The MLE parametrization in Eq.~\eqref{eq:MLE_naive} is the one that is universally used, see e.g., \cite{Choi2010,Celisse,Snijders1997Estimation,bickel_asymptotic_normality}; variants of MLE estimation such as maximization of the profile likelihood \cite{Bickel2009} or variational inference \cite{daudin} are also based solely on approximating the MLE in Eq.~\eqref{eq:MLE_naive}. In contrast, the MLE parametrization used in Eq.~\eqref{eq:MLE_rank} has, to the best of our knowledge, never been considered heretofore in the literature. We shall refer to $\hat{\mathbf{B}}^{(N)}$ and $\hat{\mathbf{B}}^{(M)}$ as the naive MLE and the true (rank-constrained) MLE, respectively.

The estimator $\hat{\mathbf{B}}^{(N)}$ is asymptotically normal around $\mathbf{B}$; in particular Lemma~1 (and its proof) in \cite{bickel_asymptotic_normality} states that
\begin{theorem}[\cite{bickel_asymptotic_normality}]
\label{thm:bickel_choi}
Let $\mathbf{A}_n \sim \mathrm{SBM}(\mathbf{B}, \bm{\pi})$ for $n \geq 1$ be a sequence of stochastic blockmodel graphs with sparsity factors $\rho_n$.
Let $\hat{\mathbf{B}}^{(N)}$ be the MLE of $\mathbf{B}$ obtained from $\mathbf{A}_n$ with $\rho_n$ assumed known. If $\rho_n \equiv 1$ for all $n$, then
\begin{gather}
\label{eq:naive_clt1}
n (\hat{\mathbf{B}}^{(N)}_{kk} - \mathbf{B}_{kk}) \overset{\mathrm{d}}{\longrightarrow} \mathcal{N}\Bigl(0, \frac{2 \mathbf{B}_{kk} (1 - \mathbf{B}_{kk})}{\pi_k^2}\Bigr), \quad \text{for $k \in [K]$} \\
\label{eq:naive_clt2}
n(\hat{\mathbf{B}}^{(N)}_{kl} - \mathbf{B}_{kl}) \overset{\mathrm{d}}{\longrightarrow} \mathcal{N}\Bigl(0, \frac{\mathbf{B}_{kl} (1 - \mathbf{B}_{kl})}{\pi_k \pi_l}\Bigr), \quad \text{for $k \in [K], l \in [K], k \not = l$}
\end{gather}
as $n \rightarrow \infty$, and that the $K(K+1)/2$ random variables $\{n(\hat{\mathbf{B}}^{(N)}_{kl}- \mathbf{B}_{kl})\}_{k \leq l}$ are asymptotically independent. If, however, $\rho_n \rightarrow 0$ with $n \rho_n = \omega(\log{n})$, then
\begin{gather}
\label{eq:naive_clt3}
n \sqrt{\rho_n} (\hat{\mathbf{B}}^{(N)}_{kk} - \mathbf{B}_{kk}) \overset{\mathrm{d}}{\longrightarrow} \mathcal{N}\Bigl(0, \frac{2 \mathbf{B}_{kk}}{\pi_k^2}
 \Bigr), \quad \text{for $k \in [K]$} \\
\label{eq:naive_clt4}
n \sqrt{\rho_n} (\hat{\mathbf{B}}^{(N)}_{kl} - \mathbf{B}_{kl}) \overset{\mathrm{d}}{\longrightarrow} \mathcal{N}\bigl(0, \frac{\mathbf{B}_{kl}}{\pi_k \pi_l}\Bigr), \quad \text{for $k \in [K], l \in [K], k \not = l$}
\end{gather}
as $n \rightarrow \infty$, and the $K(K+1)/2$ random variables $\{n \sqrt{\rho_n}(\hat{\mathbf{B}}^{(N)}_{kl} - \mathbf{B}_{kl} \}_{k \leq l}$ are asymptotically independent.
\end{theorem}
Furthermore, $\hat{\mathbf{B}}^{(N)}$ is also purported to be optimal; Section~5 of \cite{bickel_asymptotic_normality} states that
\begin{quotation}
These results easily imply that classical optimality properties of these procedures\footnote{The procedures referred to here are the MLE $\hat{\mathbf{B}}^{(N)}$ and its variational approximation as introduced in \cite{daudin}.}, such as achievement of the information bound, hold.
\end{quotation}
We present a few examples in Section~\ref{sec:main_results} illustrating that $\hat{\mathbf{B}}^{(N)}$ is optimal only if $\mathbf{B}$ is invertible, and that in general, for singular $\mathbf{B}$, $\hat{\mathbf{B}}^{(N)}$ is dominated by the MLE $\hat{\mathbf{B}}^{(M)}$.

Another widely used technique, and computationally tractable alternative to $\hat{\mathbf{B}}^{(N)}$, for estimating $\mathbf{B}$ is based on spectral embedding $\mathbf{A}$. More specifically, given $\mathbf{A} \sim \mathrm{SBM}(\mathbf{B}, \bm{\pi})$ with known sparsity factor $\rho$, we consider the following procedure for estimating $\mathbf{B}$:
\begin{enumerate}
\item Assuming $d = \mathrm{rk}(\mathbf{B})$ is known, let $\mathbf{A} = \hat{\mathbf{U}} \hat{\bm{\Lambda}} \hat{\mathbf{U}}^{\top} + \hat{\mathbf{U}}_{\perp} \hat{\bm{\Lambda}}_{\perp} \hat{\mathbf{U}}^{\top}_{\perp}$ be the eigendecomposition of $\mathbf{A}$ where $\hat{\bm{\Lambda}}$ is the diagonal matrix containing the $d$ largest eigenvalues of $\mathbf{A}$ in modulus and $\hat{\mathbf{U}}$ is the $n \times d$ matrix whose columns are the corresponding eigenvectors of $\mathbf{A}$.

\item Assuming $K$ is known, cluster the rows of $\hat{\mathbf{U}}$ into $K$ clusters using $K$-means, obtaining an ``estimate'' $\hat{\bm{\tau}}$ of $\bm{\tau}$.

\item For $k \in [K]$, let $\hat{\bm{s}}_{k}$ be the vector in $\mathbb{R}^{n}$ where the $i$-th entry of $\hat{\bm{s}}_k$ is $1$ if $\hat{\tau}_i = k$ and $0$ otherwise and let $\hat{n}_k = |\{i \colon \hat{\tau}_i = k\}|$ be the number of vertices assigned to block $k$.

\item Estimate $\mathbf{B}_{kl}$ by $\hat{\mathbf{B}}^{(S)}_{kl} = \frac{1}{\hat{n}_k \hat{n}_l \rho∫} \hat{\bm{s}}_k^{\top} \hat{\mathbf{U}} \hat{\bm{\Lambda}} \hat{\mathbf{U}}^{\top} \hat{\bm{s}}_{l}$.
\end{enumerate}
The above procedure assumes $d $ and $K$ are known. When $d$ is unknown, it can be consistently estimated using the following approach: let $\hat{d}$ be the number of eigenvalues of $\mathbf{A}$ exceeding $4 \sqrt{\delta(\mathbf{A})}$ in modulus; here $\delta(\mathbf{A})$ denotes the max degree of $\mathbf{A}$. Then $\hat{d}$ is a consistent estimate of $d$ as $n \rightarrow \infty$. This follows directly from tail bounds for $\|\mathbf{A} - \mathbb{E}[\mathbf{A}]\|$ (see e.g., \cite{rinaldo_2013,oliveira2009concentration}) and Weyl's inequality. The estimation of $K$ is investigated in \cite{bickel13:_hypot,lei2014} among others, and is based on showing that if $\mathbf{A}$ is a $K$-block SBM, then there exists a consistent estimate $\widehat{\mathbb{E}[\mathbf{A}]} = (\hat{\mathbf{A}}_{ij})$ of $\mathbb{E}[\mathbf{A}]$ such that the matrix $\tilde{\mathbf{A}}$ with entries $\tilde{\mathbf{A}}_{ij} = (\mathbf{A}_{ij} - \hat{\mathbf{A}}_{ij})/\sqrt{(n-1)\hat{\mathbf{A}}_{ij} (1 - \hat{\mathbf{A}}_{ij})}$ has a limiting Tracy-Widom distribution, i.e., $n^{2/3}(\lambda_{1}(\tilde{\mathbf{A}}) - 2)$ converges to Tracy-Widom.

The above spectral embedding procedure (and related procedures based on eigendecomposition of other matrices such as the normalized Laplacian) is also well-studied, see e.g., \cite{rinaldo_2013,sussman12,rohe2011spectral,mcsherry,joseph_yu_2015,bickel_sarkar_2013,fishkind2013consistent,athreya2013limit,tang_priebe_16,perfect,coja-oghlan} among others; however, the available results mostly focused on showing that $\hat{\bm{\tau}}$ consistently recovers $\bm{\tau}$. Since $\hat{\bm{\tau}}$ is almost surely an exact recovery of $\bm{\tau}$ in the limit, i.e., $\hat{\tau}_i = \tau_i$ for all $i$ as $n \rightarrow \infty$ (see e.g., \cite{perfect,mcsherry}), the estimate of $\mathbf{B}_{k \ell}$ given by $\tfrac{1}{\hat{n}_k \hat{n}_{\ell} \rho} \hat{\bm{s}}_k^{\top} \mathbf{A} \hat{\bm{s}}_{\ell}$ is a consistent estimate of $\mathbf{B}$, and furthermore, coincides with $\hat{\mathbf{B}}^{(N)}$ as $n \rightarrow \infty$.
The quantity $\hat{\mathbf{U}} \hat{\bm{\Lambda}} \hat{\mathbf{U}}^{\top} - \mathbb{E}[\mathbf{A}]$ is also widely analyzed in the context of matrix and graphon estimation using universal singular values thresholding \cite{chatterjee,gao,xu_spectral,klopp}; the focus there had been in showing the minimax rates of convergence of $\tfrac{1}{n^2}\|\hat{\mathbf{U}} \hat{\bm{\Lambda}} \hat{\mathbf{U}}^{\top} - \mathbb{E}[\mathbf{A}]\|_{F}$ to $0$ as $n \rightarrow \infty$. These rates of convergence, however, do not translate to results on the limiting distribution of $\hat{\mathbf{B}}^{(S)}_{k \ell} - \mathbf{B}_{k \ell} = \tfrac{1}{\hat{n}_k \hat{n}_{\ell} \rho} \hat{\bm{s}}_k^{\top} \hat{\mathbf{U}} \hat{\bm{\Lambda}} \hat{\mathbf{U}}^{\top} \hat{\bm{s}}_{\ell} - \tfrac{1}{n_{k} n_{\ell} \rho} \bm{s}_k^{\top}\mathbb{E}[\mathbf{A}] \bm{s}_{\ell}$.

In summary, formal comparisons between $\hat{\mathbf{B}}^{(N)}$ and $\hat{\mathbf{B}}^{(S)}$ are severely lacking. Our paper addresses this important void in the literature. The contributions of our paper are as follows. For stochastic blockmodel graphs with sparisty factors $\rho_n$ satisfying $n \rho_n = \omega(\sqrt{n})$, we establish asymptotic normality of $n \rho_n^{1/2} (\hat{\mathbf{B}}^{(S)} - \mathbf{B})$ in Theorem~\ref{THM:GEN_D} and Theorem~\ref{THM:GEN_D_SPARSE}. As a corollary of this result, we show that when $\mathbf{B}$ is of full-rank, that $n \sqrt{\rho_n}(\hat{\mathbf{B}}^{(S)} - \mathbf{B})$ has the same limiting distribution as $n \sqrt{\rho_n} (\hat{\mathbf{B}}^{(N)} - \mathbf{B})$ given in Eq.~\eqref{eq:naive_clt1} and Eq.~\eqref{eq:naive_clt2} and that both estimators are asymptotically efficient; the two estimators $\hat{\mathbf{B}}^{(M)}$ and $\hat{\mathbf{B}}^{(N)}$ are identical in this setting. When $\mathbf{B}$ is singular, we show that $n \sqrt{\rho_n}(\hat{\mathbf{B}}^{(S)} - \mathbf{B})$ can have smaller variances than $n \sqrt{\rho_n} (\hat{\mathbf{B}}^{(N)} - \mathbf{B})$, and thus a bias-corrected $\hat{\mathbf{B}}^{(S)}$ can have smaller mean square error than $\hat{\mathbf{B}}^{(N)}$, and furthermore, that the resulting bias-corrected $\hat{\mathbf{B}}^{(S)}$ can be almost as efficient as the asymptotically first-order efficient estimator $\hat{\mathbf{B}}^{(M)}$. Finally, we also provide some justification of the potential necessity of the condition that the average degree satisfies $n \rho_n = \omega(\sqrt{n})$; in essence, as $\rho_n \rightarrow 0$, the bias incurred by the low-rank representation $
\hat{\mathbf{U}} \hat{\bm{\Lambda}} \hat{\mathbf{U}}^{\top}$ of $\mathbf{A}$ overwhelms the reduction in variance resulting from the low-rank representation.

\section{Central limit theorem for $\hat{\mathbf{B}}^{(S)}_{k\ell}$}
\label{sec:main_results}
Let $(\mathbf{A}, \bm{\tau}) \sim \mathrm{SBM}(\mathbf{B}, \bm{\pi}, \rho)$ be a stochastic blockmodel graph
on $n$ vertices with sparsity factor $\rho$ and $\mathrm{rk}(\mathbf{B}) = d$.
We first consider the setting wherein both $\bm{\tau}$ and $d$ are assumed known. The setting wherein $\bm{\tau}$ is unobserved and needs to be recovered will be addressed subsequently in Corollary~\ref{cor:in_practice}, while the setting when $d$ is unknown was previously addressed following the introduction of the estimator $\hat{\mathbf{B}}^{(S)}$ in Section~\ref{sec:intro}. When $\bm{\tau}$ is known, the spectral embedding estimate of $\mathbf{B}_{kl}$ (with $\rho$ assumed known) is $\hat{\mathbf{B}}_{kl}^{(S)} = \tfrac{1}{\rho n_k n_l}
\bm{s}_k^{\top} \hat{\mathbf{U}} \hat{\bm{\Lambda}} \hat{\mathbf{U}}^{\top} \bm{s}_l$ where $\bm{s}_k$ is the vector whose elements $\{s_{ki}\}$ are such that
$s_{ki} = 1$ if vertex $i$ is assigned to block $k$ and $s_{ki} = 0$ otherwise; here $n_k$ denote the number of vertices $v_i$ assigned to block $k$.

We then have the following non-degenerate limiting distribution of $\hat{\mathbf{B}}^{(S)} - \mathbf{B}$. We shall present two variants of this limiting distribution. The first variant, Theorem~\ref{THM:GEN_D}, applies to the setting where the average degree grows linearly with $n$, i.e., the
sparsity factor $\rho_n \rightarrow c > 0$; without loss of generality, we can assume $\rho_n \equiv c = 1$. The second variant applies to the setting where the average degree grows sub-linearly in $n$, i.e., $\rho_n \rightarrow 0$ with $n \rho_n = \omega(\sqrt{n})$. For ease of exposition, these variants (and their proofs) will be presented using the following parametrization of stochastic blockmodel graphs as a sub-class of the more general random dot product graphs model \cite{young2007random,grdpg1}.
\begin{definition}[Generalized random dot product graph]
\label{def:grdpg}
  Let $d$ be a positive integer and $p \geq 1$ and $q \geq 0$ be
  such that $p + q = d$. Let $\mathbf{I}_{p,q}$ denote the
  diagonal matrix whose diagonal elements contains $p$ entries
  equaling $1$ and $q$ entries equaling $-1$.
  Let $\mathcal{X}$ be a subset of
  $\mathbb{R}^{d}$ such $x^{\top} \mathbf{I}_{p,q} y \in[0,1]$ for all $x,y\in
  \mathcal{X}$. Let $F$ be a distribution taking values in $\mathcal{X}$.
  We say $(\mathbf{X},\mathbf{A}) \sim
  \mathrm{GRDPG}_{p,q}(F)$ with sparsity factor $\rho \in (0,1]$ if the following hold. First let $X_1, X_2,
  \dots, X_n \overset{\mathrm{i.i.d}}{\sim} F$ and set
  $\mathbf{X}=[X_1 \mid \cdots \mid X_n]^\top\in \mathbb{R}^{n\times
    d}$. Then $\mathbf{A}\in\{0,1\}^{n\times n}$ is a symmetric matrix
such that, conditioned on $\mathbf{X}$, for all $i \geq j$ the
$A_{ij}$ are independent and
\begin{equation}
 A_{ij} \sim \mathrm{Bernoulli}(\rho X_i^\top \mathbf{I}_{p,q} X_j).
\end{equation}
We therefore have
\begin{equation}
\Pr[\mathbf{A} \mid \mathbf{X}]=\prod_{i \leq j} (\rho X^{\top}_i \mathbf{I}_{p,q}
X_j)^{A_{ij}}(1- \rho X^{\top}_i \mathbf{I}_{p,q} X_j)^{(1-A_{ij})}.
\end{equation}
\end{definition}
It is straightforward to show that any stochastic blockmodel graph
$(\mathbf{A}, \bm{\tau}) \sim \mathrm{SBM}(\bm{\pi}, \mathbf{B})$
can also be represented as a (generalized) random dot product
graph $(\mathbf{X}, \mathbf{A}) \sim \mathrm{GRDPG}_{p,q}(F)$ where $F$ is a
mixture of point masses. Indeed, suppose $\mathbf{B}$ is a $K \times
K$ matrix and let $\mathbf{B} = \mathbf{U}
\bm{\Sigma} \mathbf{U}^{\top}$ be the eigendecomposition of
$\mathbf{B}$. Then, denoting by $\nu_1, \nu_2, \dots, \nu_K$ the rows
of $\mathbf{U} |\bm{\Sigma}|^{1/2}$, we can define $F = \sum_{k=1}^{K} \pi_k
\delta_{\nu_k}$ where $\delta$ is the Dirac delta function; $p$ and $q$ are given by the number of positive and negative
eigenvalues of $\mathbf{B}$, respectively.

\begin{theorem}
\label{THM:GEN_D}
Let $\mathbf{A} \sim \mathrm{SBM}(\bm{\pi}, \mathbf{B}, \rho_n)$ be a $K$-block stochastic blockmodel graph on $n$ vertices with sparsity factor $\rho_n = 1$. Let $\nu_1, \dots, \nu_K$ be point masses in $\mathbb{R}^{d}$ such that $\mathbf{B}_{k \ell} = \nu_k^{\top} \mathbf{I}_{p,q} \nu_{\ell}$ and let $\Delta = \sum_{k} \pi_k \nu_k \nu_k^{\top}$.

For $k \in [K]$ and $\ell \in [K]$, let $\theta_{k \ell}$ be given by
 \begin{equation}
 \label{eq:mu_kl}
 \begin{split}
\theta_{k\ell} &= \sum_{r=1}^{K} \pi_r \bigl(\mathbf{B}_{kr} (1 - \mathbf{B}_{kr}) + \mathbf{B}_{\ell r}(1 - \mathbf{B}_{\ell r})\bigr)\nu_k^{\top} \Delta^{-1} \mathbf{I}_{p,q} \Delta^{-1} \nu_{\ell} \\
& - \sum_{r=1}^{K} \sum_{s=1}^{K} \pi_r \pi_s \mathbf{B}_{sr} (1 - \mathbf{B}_{sr}) \nu_s^{\top} \Delta^{-1} \mathbf{I}_{p,q} \Delta^{-1}
(\nu_{\ell} \nu_k^{\top} + \nu_k \nu_{\ell}^{\top}) \Delta^{-1} \nu_s.

 \end{split}
 \end{equation}
 Now let $\zeta_{k \ell} = \nu_{k}^{\top} \Delta^{-1} \nu_{\ell}$. Define $\sigma_{kk}^2$ for $k \in [K]$ to be
 \begin{equation}
 \begin{split}
 \label{eq:sigma_kk}
 \sigma_{k k}^2 &=
 4 \mathbf{B}_{kk}(1 - \mathbf{B}_{kk}) \zeta_{kk}^2 + 4 \sum_{r} \pi_r \mathbf{B}_{kr} (1 - \mathbf{B}_{kr}) \zeta_{kr}^2 \bigl(\tfrac{1}{\pi_k} - 2 \zeta_{kk}\bigr) \\ &
+ 2 \sum_{r} \sum_{s} \pi_r \pi_s \mathbf{B}_{rs} (1 - \mathbf{B}_{rs}) \zeta_{kr}^2 \zeta_{ks}^2
 \end{split}
 \end{equation}
 and define $\sigma_{k \ell}^{2}$ for $k \in [K], \ell \in [K], k \not = \ell$ to be
 \begin{equation}
 \begin{split}
 \label{eq:sigma_kl2}
 \sigma_{k\ell}^2 &=
 \bigl(\mathbf{B}_{kk} (1 - \mathbf{B}_{kk}) + \mathbf{B}_{\ell \ell}(1 - \mathbf{B}_{\ell \ell}) \bigr)\zeta_{k \ell}^2 +
 2 \mathbf{B}_{k \ell} (1 - \mathbf{B}_{k \ell}) \zeta_{kk} \zeta_{\ell \ell}
\\ &+
\sum_{r}  \pi_{r} \mathbf{B}_{k r}(1 - \mathbf{B}_{kr}) \zeta_{\ell r}^2 \bigl(\tfrac{1}{\pi_{k}} - 2 \zeta_{kk} \bigr)
\\ &+ \sum_{r} \pi_r \mathbf{B}_{\ell r} (1 - \mathbf{B}_{\ell r}) \zeta_{kr}^{2} \bigl(\tfrac{1}{\pi_{\ell}} - 2 \zeta_{\ell \ell} \bigr) \\
& - 2 \sum_{r}  \pi_r \bigl(\mathbf{B}_{k r} (1 - \mathbf{B}_{k r}) + \mathbf{B}_{\ell r} (1 - \mathbf{B}_{\ell r})\bigr) \zeta_{kr} \zeta_{r \ell} \zeta_{k \ell} \\
& + \frac{1}{2} \sum_{r} \sum_{s} \pi_r \pi_s
\mathbf{B}_{rs} (1 - \mathbf{B}_{rs}) (\zeta_{k r} \zeta_{\ell s} + \zeta_{\ell r} \zeta_{k s})^2.
 \end{split}
 \end{equation}
Then for any $k \in [K]$ and $\ell \in [K]$,
\begin{equation}
\label{eq:sbm_normal2}
  n \bigl(\hat{\mathbf{B}}^{(S)}_{k\ell} - \mathbf{B}_{k \ell} - \frac{\theta_{kl}}{n}\bigr) \overset{\mathrm{d}}{\longrightarrow} N(0, \sigma_{k \ell}^2)
\end{equation}
as $n \rightarrow \infty$.
\end{theorem}

\begin{theorem}
\label{THM:GEN_D_SPARSE}
Let $\mathbf{A} \sim \mathrm{SBM}(\bm{\pi}, \mathbf{B}, \rho_n)$ be a $K$-block stochastic blockmodel graph on $n$ vertices with sparsity factor $\rho_n$.
Let $\nu_1, \dots, \nu_K$ be point masses in $\mathbb{R}^{d}$ such that $\mathbf{B}_{k \ell} = \nu_k^{\top} \mathbf{I}_{p,q} \nu_{\ell}$ and let $\Delta = \sum_{k} \pi_k \nu_k \nu_k^{\top}$.
For $k \in [K]$ and $\ell \in [K]$, let $\tilde{\theta}_{k \ell}$ be given by
\begin{equation}
\begin{split}
\label{eq:tilde_theta_kl}
\tilde{\theta}_{k\ell} & = \sum_{r=1}^{K} \pi_r \bigl(\mathbf{B}_{kr} + \mathbf{B}_{\ell r} \bigr)\nu_k^{\top} \Delta^{-1} \mathbf{I}_{p,q} \Delta^{-1} \nu_{\ell}
 \\ &- \sum_{r=1}^{K} \sum_{s=1}^{K} \pi_r \pi_s \mathbf{B}_{sr} \nu_s^{\top} \Delta^{-1} \mathbf{I}_{p,q} \Delta^{-1}
(\nu_{\ell} \nu_k^{\top} + \nu_k \nu_{\ell}^{\top}) \Delta^{-1} \nu_s,
\end{split}
\end{equation}
let $\tilde{\sigma}_{kk}^{2}$ for $k \in [K]$ be
\begin{equation}
\begin{split}
\label{eq:tilde_sigma_kk}
\tilde{\sigma}_{kk}^{2} &=
4 \mathbf{B}_{kk} \zeta_{kk}^2 + 4 \sum_{r} \pi_r \mathbf{B}_{kr} \zeta_{kr}^2 \bigl(\tfrac{1}{\pi_k} - 2 \zeta_{kk}\bigr)^{2}
\\ &+ 2 \sum_{r} \sum_{s} \pi_r \pi_s \mathbf{B}_{rs} \zeta_{kr}^2 \zeta_{ks}^2,
\end{split}
\end{equation}
and let $\tilde{\sigma}_{k\ell}^{2}$ for $k \in [K]$, $\ell \in [K]$, $k \not = \ell$ be
\begin{equation}
 \begin{split}
 \label{eq:tilde_sigma_kl}
 \tilde{\sigma}_{k\ell}^2 &=
 \bigl(\mathbf{B}_{kk}  + \mathbf{B}_{\ell \ell} \bigr)\zeta_{k \ell}^2 +
2 \mathbf{B}_{k \ell} \zeta_{kk} \zeta_{\ell \ell}
 - 2 \sum_{r}  \pi_r \bigl(\mathbf{B}_{k r}  + \mathbf{B}_{\ell r} \bigr) \zeta_{kr} \zeta_{r \ell} \zeta_{k \ell}
\\ &  +
\sum_{r}  \pi_{r} \mathbf{B}_{k r} \zeta_{\ell r}^2 \bigl(\tfrac{1}{\pi_{k}} - 2 \zeta_{kk} \bigr)^{2}
+ \sum_{r} \pi_r \mathbf{B}_{\ell r} \zeta_{kr}^{2} \bigl(\tfrac{1}{\pi_{\ell}} - 2 \zeta_{\ell \ell} \bigr)^{2}
\\ &+ \frac{1}{2} \sum_{r} \sum_{s} \pi_r \pi_s
\mathbf{B}_{rs} (\zeta_{k r} \zeta_{\ell s} + \zeta_{\ell r} \zeta_{k s})^2.
 \end{split}
 \end{equation}
If $\rho_n \rightarrow 0$ and $n \rho_n = \omega(\sqrt{n})$, then for any $k \in [K]$ and $\ell \in [K]$,
\begin{equation}
\label{eq:sbm_normal3}
  n \sqrt{\rho_n} \bigl(\hat{\mathbf{B}}^{(S)}_{k\ell} - \mathbf{B}_{k \ell} - \frac{\tilde{\theta}_{kl}}{n \rho_n}\bigr) \overset{\mathrm{d}}{\longrightarrow} N(0, \tilde{\sigma}_{k \ell}^2)
\end{equation}
as $n \rightarrow \infty$.
\end{theorem}
The proofs of Theorem~\ref{THM:GEN_D} and Theorem~\ref{THM:GEN_D_SPARSE} are given in the appendix. As a corollary of Theorem~\ref{THM:GEN_D} and Theorem~\ref{THM:GEN_D_SPARSE}, we have the following result for the asymptotic efficiency of $\hat{\mathbf{B}}^{(S)}$ whenever $\mathbf{B}$ is invertible (see also Theorem~\ref{thm:bickel_choi}).
\begin{corollary}
\label{cor:full-rank}
Let $\mathbf{A} \sim \mathrm{SBM}(\bm{\pi}, \mathbf{B}, \rho_n)$ be a $K$-block stochastic blockmodel graph
 on $n$ vertices with sparsity factor $\rho_n$. Suppose $\mathbf{B}$ is invertible. Then for all $k \in [K], \ell \in [K]$, $\sigma_{k \ell}$ and $\tilde{\sigma}_{k \ell}$ as defined in Theorem~\ref{THM:GEN_D} and Theorem~\ref{THM:GEN_D_SPARSE} satisfy
 \begin{gather}
 \theta_{k \ell} = 0; \quad \tilde{\theta}_{k \ell} = 0 \\
 \sigma_{kk}^2 = \frac{2 \mathbf{B}_{kk} (1 - \mathbf{B}_{kk})}{\pi_k^2}; \quad
 \sigma_{k\ell}^2 = \frac{\mathbf{B}_{k \ell} (1 - \mathbf{B}_{k\ell})}{\pi_k \pi_{\ell}} \,\, \text{if $k \not = \ell$} \\
 \tilde{\sigma}_{kk} = \frac{2 \mathbf{B}_{kk}}{\pi_k^2}; \quad \tilde{\sigma}_{k \ell} = \frac{2 \mathbf{B}_{k \ell}}{\pi_k \pi_{\ell}} \,\, \text{if $k \not = \ell$}.
 \end{gather}
 Therefore, for all $k \in [K], \ell \in [K]$, if $\rho_n \equiv 1$, then
 \begin{equation}
  n (\hat{\mathbf{B}}^{(S)}_{k\ell} - \mathbf{B}_{k\ell}) \overset{\mathrm{d}}{\longrightarrow} N(0, \sigma_{k\ell}^2).
 \end{equation}
 as $n \rightarrow \infty$. If $\rho_n \rightarrow 0$ and $n \rho_n = \omega(\sqrt{n})$, then
  \begin{equation}
  n \rho_n^{1/2}(\hat{\mathbf{B}}^{(S)}_{k\ell} - \mathbf{B}_{k\ell}) \overset{\mathrm{d}}{\longrightarrow} N(0, \tilde{\sigma}_{k\ell}^2).
 \end{equation}
as $n \rightarrow \infty$. $\hat{\mathbf{B}}^{(S)}_{k \ell}$ is therefore {\em asymptotically efficient} for all $k, \ell$.
\end{corollary}
\begin{proof}[Proof of Corollary~\ref{cor:full-rank}]
The proof follows trivially from the observation that $\zeta_{r s} = \tfrac{1}{\pi_r}$ for $r = s$ and $\zeta_{rs} = 0$ otherwise.
Indeed, $\Delta = \sum_{k} \pi_k \nu_k \nu_k^{\top} = \bm{\nu}^{\top} \mathbf{D} \bm{\nu}$
where $\bm{\nu}$ is a $K \times K$ matrix with $\bm{\nu} \mathbf{I}_{p,q} \bm{\nu}^{\top} = \mathbf{B}$ and $\mathbf{D} = \mathrm{diag}(\bm{\pi})$. Hence
\begin{equation}
\label{key:reduction}
\zeta_{rs} = \nu_r^{\top} \Delta^{-1} \nu_{s} = \nu_r^{\top} \bm{\nu}^{-1} \mathbf{D}^{-1} (\bm{\nu}^{-1})^{\top} \nu_s = \frac{1}{\pi_r} \mathbbm{1}\{r = s\}.
\end{equation}
As an example, the expression for $\theta_{k \ell}$ in Eq.~\eqref{eq:mu_kl} reduces to
\begin{equation*}
\begin{split} \theta_{k\ell} &= \sum_{r=1}^{K} \pi_r \bigl(\mathbf{B}_{kr} (1 - \mathbf{B}_{kr}) + \mathbf{B}_{\ell r}(1 - \mathbf{B}_{\ell r})\bigr)\nu_k^{\top} \Delta^{-1} \mathbf{I}_{p,q} \Delta^{-1} \nu_{\ell}  \\ &- \sum_{r=1}^{K} \sum_{s=1}^{K} \pi_r \pi_s \mathbf{B}_{sr} (1 - \mathbf{B}_{sr}) \nu_s^{\top} \Delta^{-1} \mathbf{I}_{p,q} \Delta^{-1}
 \Bigl(\tfrac{\mathbbm{1}\{s = k\}}{\pi_k} \nu_{\ell} + \tfrac{\mathbbm{1}\{s = \ell\}}{\pi_{\ell}} \nu_{k} \Bigr)
\\ &= 0
\end{split}
\end{equation*}
for all $k \in [K], \ell \in [K]$. The expression for $\sigma_{kk}^{2},\tilde{\sigma}_{kk}^2$, $\sigma_{k \ell}^2$ and $\tilde{\sigma}_{k\ell}^2$ also follows directly from Eq.~\eqref{key:reduction}.
\end{proof}

\begin{remark}
As a special case of Theorem~\ref{THM:GEN_D}, we consider the two blocks stochastic blockmodel with block probability matrix $\mathbf{B} = \Bigl[\begin{smallmatrix} p^2 & pq \\ pq & q^2 \end{smallmatrix} \Bigr]$ and block assignment probabilities $\bm{\pi} = (\pi_p, \pi_q), \pi_p + \pi_q = 1.$ Then
$\Delta = \pi_p p^2 + \pi_q q^2$ and Eq.~\eqref{eq:mu_kl} reduces to
\begin{gather*}
\theta_{11} = \tfrac{2 \pi_q p^2 q^2}{\Delta^3} \bigl(\pi_p p^2 (1 - p^2) + (\pi_q - \pi_p) p q (1 - pq) -  \pi_q q^2 (1 - q^2) \bigr), \\
\theta_{12} = \tfrac{pq}{\Delta^3} \bigl(\pi_p p^2 ( 1- p^2) (\pi_q q^2 - \pi_p p^2) + (\pi_p - \pi_q) pq (1 - pq) (\pi_p q^2 - \pi_q p^2) \\ + \pi_q q^2 ( 1- q^2) (\pi_p p^2 - \pi_q q^2) \bigr),	 \\
\theta_{22} = \tfrac{2 \pi_p p^2 q^2}{\Delta^3} \bigl(\pi_q q^2 (1 - q^2) + (\pi_p - \pi_q) p q (1 - pq) - \pi_p p^2 (1 - p^2) \bigr).
\end{gather*}
Meanwhile, we also have
\begin{gather*}
\sigma^{2}_{11} = \tfrac{8 p^6 (1 - p^2)}{\Delta^2} \bigl(1 - \tfrac{\pi_p p^2}{2 \Delta}\bigr)^2 + \tfrac{4 \pi_q p^3 q^3 (1 - pq)}{\pi_p \Delta^2} \bigl(1 - \tfrac{\pi_p p^2}{\Delta}\bigr)^2 + \tfrac{2 \pi_q^2 p^4 q^6 (1 - q^2)}{\Delta^4} \\
\sigma^{2}_{12} = \tfrac{2 \pi_q^2 p^4 q^6 (1 - p^2)}{\Delta^4} + \tfrac{\pi_p \pi_q pq (1 - pq)}{\Delta^4} \bigl(\tfrac{\pi_q q^4}{\pi_p} + \tfrac{\pi_p p^4}{\pi_q}\bigr)^2 + \tfrac{2 \pi_p^2 p^6 q^4 (1 - q^2)}{\Delta^4} \\
\sigma^2_{22} = \tfrac{8 q^6 (1 - q^2)}{\Delta^2} \bigl(1 - \tfrac{\pi_q q^2}{2 \Delta}\bigr)^2 + \tfrac{4 \pi_p p^3 q^3 (1 - pq)}{\pi_q \Delta^2} \bigl(1 - \tfrac{\pi_q q^2}{\Delta}\bigr)^2 + \tfrac{2 \pi_p^2 q^4 p^6 (1 - p^2)}{\Delta^4}.
\end{gather*}
The naive (MLE) estimator $\hat{\mathbf{B}}^{(N)}$ has asymptotic variances
\begin{gather*}
\mathrm{Var}[\hat{\mathbf{B}}^{(N)}_{11}] = \tfrac{2 p^2 (1 - p^2)}{\pi_p^2}; \quad \mathrm{Var}[\hat{\mathbf{B}}^{(N)}_{12}] = \tfrac{pq (1 - pq)}{\pi_p \pi_q}; \quad \mathrm{Var}[\hat{\mathbf{B}}^{(N)}_{22}] = \tfrac{2 q^2 (1 - q^2)}{\pi_q^2}.
\end{gather*}
We now evaluate the asymptotic variances for the true (rank-constrained) MLE. Suppose for simplicity that $n \pi_p$ vertices are assigned to block $1$ and $n \pi_q$ vertices are assigned to block $2$. Let $n_{11} = \tbinom{(n \pi_p + 1)}{2}$, $n_{12} = n^2 \pi_p \pi_q$ and $n_{22} = \tbinom{(n \pi_p + 1)}{2}$.
Let $\mathbf{A}$ be given and assume for the moment that $\bm{\tau}$ is observed. Then the log-likelihood for $\mathbf{A}$ is equivalent to the log-likelihood for observing $m_{11} \sim \mathrm{Bin}(n_{11}, p^2)$,  $m_{12} \sim \mathrm{Bin}(n_{12}, pq)$ and $m_{22} \sim \mathrm{Bin}(n_{22}, q^2)$ with $m_{11}$, $m_{12}$, and $m_{22}$ mutually independent. More specifically, ignoring terms of the form $\tbinom{n_{ij}}{m_{ij}}$, we have
\begin{equation*}
\begin{split}
 \ell(\mathbf{A} \mid p,q) &= m_{11} \log p^2 + (n_{11} - m_{11}) \log (1 - p^2) + m_{12} \log pq
 \\ &+ (n_{12} - m_{12}) \log (1 - pq) + m_{22} \log q^2 + (n_{22} - m_{22}) \log (1 - q^2)
 \end{split}
 \end{equation*}
 We therefore have
 \begin{equation*}
 \begin{split}
 \mathrm{Var}\Bigl(\tfrac{\partial \ell}{\partial p}\Bigr) &= \mathrm{Var}\Bigl(\tfrac{2m_{11}p}{p^2} - \tfrac{2(n_{11} - m_{11})p}{1 - p^2} +
\tfrac{m_{12}q}{pq} - \tfrac{(n_{12} - m_{12})q}{1 - pq}\Bigr)
= \tfrac{4n_{11}}{1 - p^2} + \tfrac{n_{12} pq}{p^2(1 - pq)}.
 \end{split}
 \end{equation*}
 Similarly,
 \begin{gather*}

 \mathrm{Var}\Bigl(\tfrac{\partial \ell}{\partial q}\Bigr) = \tfrac{n_{12}pq}{q^2(1-pq)} + \tfrac{4n_{22}}{1 - q^2}; \quad
 \mathrm{Cov}\Bigl(\tfrac{\partial \ell}{\partial p}, \tfrac{\partial \ell}{\partial q} \Bigr) = \tfrac{n_{12}}{1 - pq}.
 \end{gather*}
 Next we note that $n_{11}/n^2 \rightarrow \pi_p^2/2$,
$n_{12}/n^2 \rightarrow \pi_p \pi_q$ and
 $n_{22}/n^2 \rightarrow \pi_q^2/2$. We therefore have
 $$\frac{1}{n^2} \mathrm{Var}\Bigl[\Bigl(\tfrac{\partial \ell}{\partial p}, \tfrac{\partial \ell}{\partial p}\Bigr) \Bigr] \overset{\mathrm{a.s.}}{\longrightarrow} \mathcal{I} := \begin{bmatrix} \frac{2 \pi_p^2}{1 - p^2} + \frac{\pi_p \pi_q q}{p(1 - pq)} & \frac{\pi_p \pi_q}{1 - pq} \\
 \frac{\pi_p \pi_q}{1 - pq} & \frac{2 \pi_q^2}{1 - q^2} + \frac{\pi_p \pi_q p}{q(1 - pq)} \end{bmatrix} $$
 as $n \rightarrow \infty$. Let $(\hat{p}, \hat{q})$ be the MLE of $(p,q)$ (we emphasize that there is no close form formula for the MLE $\hat{p}$ and $\hat{q}$ in this setting) and let $\mathcal{J}$ be the Jacobian of $\mathbf{B}$ with respect to $p$ and $q$, i.e.,
$ \mathcal{J}^{\top} = \Bigl[\begin{smallmatrix} 2p & q & 0 \\ 0 & p & 2q \end{smallmatrix}\Bigr].$
 The classical theory of maximum likelihood estimation (see e.g., Theorem~5.1 of \cite{point_estimation}) implies that
 $$ n\bigl( \mathrm{vech}(\hat{\mathbf{B}}^{(M)} - \mathbf{B}) \bigr) \overset{\mathrm{d}}{\longrightarrow} \mathrm{MVN}\bigl(\bm{0}, \mathcal{J} \mathcal{I}^{-1} \mathcal{J}^{\top} \bigr)$$
 as $n \rightarrow \infty$, i.e., $\hat{\mathbf{B}}^{(M)}$ is asymptotically (first-order) efficient.

\begin{figure}[tp!]
\center
\subfloat[$\mathrm{MSE}(\hat{\mathbf{B}}^{(S)})/\mathrm{MSE}(\hat{\mathbf{B}}^{(N)})$]{
\includegraphics[width=0.48\textwidth]{ratio_mse_rank1.pdf}}
\subfloat[$\mathrm{MSE}(\hat{\mathbf{B}}^{(S)})/\mathrm{MSE}(\hat{\mathbf{B}}^{(M)})$]{
\includegraphics[width=0.48\textwidth]{ratio_mle.pdf}}
\caption{The ratios $\mathrm{MSE}(\hat{\mathbf{B}}^{(S)})/\mathrm{MSE}(\hat{\mathbf{B}}^{(N)})$
and $\mathrm{MSE}(\hat{\mathbf{B}}^{(S)})/\mathrm{MSE}(\hat{\mathbf{B}}^{(M)})$ for values of
$p \in [0.1, 0.9]$ and $q \in [0.1, 0.9]$ in a $2$-blocks rank $1$ SBM.
The labeled lines in each plot are the contour lines for the $\mathrm{MSE}$ ratios.
The $\mathrm{MSE}(\hat{\mathbf{B}}^{(S)})$ is computed using the bias-adjusted estimates $\{\hat{\mathbf{B}}^{(S)}_{ij} - \hat{\theta}_{ij}\}_{i \leq j}$ (see Corollary~\ref{cor:in_practice}).
}
\label{fig:ratio-plot}
\end{figure}
We now compare the naive estimator $\hat{\mathbf{B}}^{(N)}$, the ASE estimator $\hat{\mathbf{B}}^{(S)}$ and the true MLE $\hat{\mathbf{B}}^{(M)}$ for the case when $\pi_p  = 0.5$ and $\rho_n \equiv 1$.
Plots of the (ratio) of the MSE (equivalently the sum of variances $\sigma_{11}^2 + \sigma_{12}^2 + \sigma_{22}^2$)
for $\hat{\mathbf{B}}^{(S)}$ (adjusted for the bias terms $\theta_{k \ell}$; see Corollary~\ref{cor:in_practice}) against the MSE (equivalently the sum of variances) of $\hat{\mathbf{B}}^{(N)}$ and the MSE of $\hat{\mathbf{B}}^{(M)}$ for
$p \in [0.1,0.9]$ and $q \in [0.1,0.9]$ are given in Figure~\ref{fig:ratio-plot}. For this example,  $\hat{\mathbf{B}}^{(S)}$ have smaller mean squared error than $\hat{\mathbf{B}}^{(N)}$ over the whole range of $p$ and $q$. In addition $\hat{\mathbf{B}}^{(S)}$ has mean squared error almost as small as that of $\hat{\mathbf{B}}^{(M)}$ for a large range of $p$ and $q$.
\end{remark}

\begin{remark}
We next compare the three estimators $\hat{\mathbf{B}}^{(S)}$, $\hat{\mathbf{B}}^{(M)}$ and $\hat{\mathbf{B}}^{(N)}$ in the setting of stochastic blockmodels with $3 \times 3$ block probability matrix $\mathbf{B}$ where $\mathbf{B}$ is positive semidefinite and $\mathrm{rk}(\mathbf{B}) = 2$. The minimal parametrization of $\mathbf{B}$ requires $5$ parameters $(r_1,r_2,r_3,\theta,\gamma)$, namely
\begin{gather*}
\mathbf{B}_{11} = r_1^2; \quad \mathbf{B}_{22} = r_2^2; \quad \mathbf{B}_{33} = r_3^2; \\ \mathbf{B}_{12} = r_1 r_2 \cos \theta; \quad \mathbf{B}_{13} = r_1 r_3 \cos \gamma ; \quad \mathbf{B}_{23} = r_2 r_3 \cos(\theta - \gamma).
\end{gather*}
Now let $\bm{\pi} = (\pi_1, \pi_2, \pi_3)$ be the block assignment probability vector. Let $\mathbf{A} \sim \mathrm{SBM}(\mathbf{B}, \bm{\pi})$ be a graph on $n$ vertices and suppose for simplicity that the number of vertices in block $i$ is $n_i = n \pi_i$. Let $n_{ii} = n_i^{2}/2$ for $i=1,2,3$
and $n_{ij} = n_i n_j$ if $i \not = j$. Let $m_{ij}$ for $i \leq j$ be independent random variables with $m_{ij} \sim \mathrm{Bin}(n_{ij}, \mathbf{B}_{ij})$.
Then, assuming $\bm{\tau}$ is known, the log-likelihood for $\mathbf{A}$ is
\begin{equation*}
\begin{split}
\ell(\mathbf{A}) &= m_{11} \log(r_1^2) + (n_{11} - m_{11}) \log(1 - r_1^2) + m_{22} \log (r_2^2) + (n_{22} - m_{22}) \log (1 - r_2^2) \\ &+ m_{33} \log (r_3^2) + (n_{33} - m_{33}) \log (1 - r_3^2) + m_{12} \log (r_1 r_2 \cos \theta) \\ &+ (n_{12} - m_{12}) \log (1 - r_1 r_2 \cos \theta) + m_{13} \log (r_1 r_3 \cos \gamma) \\ & + (n_{13} - m_{13}) \log (1 - r_1 r_3 \cos \gamma) + m_{23} \log (r_2 r_3 \cos (\theta - \gamma)) \\ &+ (n_{23} - m_{23}) \cos(1 - r_2 r_3 \cos(\theta - \gamma))
\end{split}
\end{equation*}
The Fisher information matrix $\mathcal{I}$ for $(r_1, r_2, r_3, \theta, \gamma)$ in this setting is straightforward, albeit tedious, to derive.

\begin{figure}[tp!]
\center
\subfloat[$\mathrm{MSE}(\hat{\mathbf{B}}^{(S)})/\mathrm{MSE}(\hat{\mathbf{B}}^{(N)})$]{
\includegraphics[width=0.48\textwidth]{ratio_3blocks_naive.pdf}}
\subfloat[$\mathrm{MSE}(\hat{\mathbf{B}}^{(S)})/\mathrm{MSE}(\hat{\mathbf{B}}^{(M)})$]{
\includegraphics[width=0.48\textwidth]{ratio_3blocks_mle.pdf}}
\caption{The ratios $\mathrm{MSE}(\hat{\mathbf{B}}^{(S)})/\mathrm{MSE}(\hat{\mathbf{B}}^{(N)})$ and $\mathrm{MSE}(\hat{\mathbf{B}}^{(S)})/\mathrm{MSE}(\hat{\mathbf{B}}^{(M)})$ for values of
$r_2 \in [0.1, 0.9]$ and $\theta \in [0.1, \tfrac{\pi}{2} - 0.1]$ in a $3$-block rank $2$ SBM.
The labeled lines in each plot are the contour lines for the $\mathrm{MSE}$ ratios.
The $\mathrm{MSE}(\hat{\mathbf{B}}^{(S)})$ is computed using the bias-adjusted estimates $\{\hat{\mathbf{B}}^{(S)}_{ij} - \hat{\theta}_{ij}\}_{i \leq j}$ (see Corollary~\ref{cor:in_practice}).
}
\label{fig:ratio-plot2}
\end{figure}
Plots of the MSE (equivalently the sum of variances
) for the bias-adjusted estimates $\hat{\mathbf{B}}^{(S)}$ against the MSE (equivalently the sum of variances) of the naive estimates $\hat{\mathbf{B}}^{(N)}$ and the true MLE estimates $\hat{\mathbf{B}}^{(M)}$ for a subset of the parameters of a $3$-blocks SBM with $\rho_n \equiv 1$ given in Figure~\ref{fig:ratio-plot2}. In Figure~\ref{fig:ratio-plot2}, we fix $\bm{\pi} = (1/3,1/3,1/3)$, $r_3 = 0.7$, $\gamma = 0.5$, $r_1 = 1- r_2$, letting $r_2$ and $\theta$ vary in the intervals $[0.1,0.9]$ and $[0.1, \tfrac{\pi}{2} - 0.1]$, respectively. Once again, $\hat{\mathbf{B}}^{(S)}$ has smaller mean squared error than $\hat{\mathbf{B}}^{(N)}$ over the whole range of $r_2$ and $\theta$, and has mean squared error almost as small as that of $\hat{\mathbf{B}}^{(M)}$ for a large range of $r_2$ and $\theta$.

\end{remark}

We emphasize that the rank assumptions placed on $\mathbf{B}$ in the previous two examples are natural assumptions, i.e., there is no primafacie reason why $\mathbf{B}$ needs to be invertible, and hence procedures that can both estimate $\mathrm{rk}(\mathbf{B})$ and incorporate it in the subsequent estimation of $\mathbf{B}$ are equally flexible and generally more efficient.  This is in contrast to other potentially more restrictive assumptions such as assuming that $\mathbf{B}$ is of the form $q \bm{1} \bm{1}^{\top} + (p - q) \mathbf{I}$ for $p > q$ (i.e., the planted-partitions model). Indeed, a $K$-block SBM from the planted-partitions model is parametrized by two parameters, irrespective of $K$ and as such the three estimators considered in this paper are provably sub-optimal for estimating the parameters of the planted partitions model.
\section{Discussions}
Theorem~\ref{THM:GEN_D} and Theorem~\ref{THM:GEN_D_SPARSE} were presented in the context wherein the vertices to block assignments $\bm{\tau}$ are assumed known. For unknown $\bm{\tau}$, Lemma~\ref{LEM:PERFECT} (presented below) implies that $\hat{\bm{\tau}}$ obtained using $K$-means (or Gaussian mixture modeling) on the rows of $\hat{\mathbf{U}}$ is an exact recovery of $\bm{\tau}$, provided that $n \rho_n \ = \omega(\log n)$. The lemma implies Corollary~\ref{cor:in_practice} showing that we can replace the quantities $\theta_{k \ell}$ and $\tilde{\theta}_{k \ell}$ in Eq.~\eqref{eq:sbm_normal2} and Eq.~\eqref{eq:sbm_normal3} of Theorem~\ref{THM:GEN_D} and Theorem~\ref{THM:GEN_D_SPARSE} by consistent estimates $\hat{\theta}_{k \ell}$ without changing the resulting limiting distribution. We emphasize that it is essential for Corollary~\ref{cor:in_practice} that $\hat{\bm{\tau}}$ is an exact recovery of $\bm{\tau}$ in order for the limiting distributions in Eq.~\eqref{eq:sbm_normal2} and Eq.~\eqref{eq:sbm_normal3} to remain valid when $\hat{\theta}_{k \ell}$ is substituted for $\theta_{k \ell}$ and $\tilde{\theta}_{k \ell}$. Indeed, if there is even a single vertex that is mis-clustered by $\hat{\bm{\tau}}$, then $\hat{\theta}_{k \ell}$ as defined will introduce an additional (random) bias term in the limiting distribution of Eq.~\eqref{eq:sbm2_inpractice}.

\begin{remark}
For ease of exposition, bounds in this paper are often written as holding ``with high probability''. A random variable $\xi \in \mathbb{R}$ is $O_{\mathbb{P}}(f(n))$ if, for any positive constant $c > 0$ there exists a $n_0 \in \mathbb{N}$ and a constant $C > 0$ (both of which possibly depend on $c$) such that for all $n \geq n_0$, $|\xi| \leq C f(n)$ with probability at least $1 - n^{-c}$; moreover, a random variable $\xi \in \mathbb{R}$ is $o_{\mathbb{P}}(f(n))$ if for any positive constant $c > 0$ and any $\epsilon > 0$ there exists a $n_0 \in \mathbb{N}$ such that for all $n \geq n_0$, $|\xi| \leq \epsilon f(n)$ with probability at least $1 - n^{-c}$. Similarly, when $\xi$ is a random vector in $\mathbb{R}^{d}$ or a random matrix in $\mathbb{R}^{d_1 \times d_2}$, $\xi = O_{\mathbb{P}}(f(n))$ or $\xi = o_{\mathbb{P}}(f(n))$ if $\|\xi\| = O_{\mathbb{P}}(f(n))$  or $\|\xi\| = o_{\mathbb{P}}(f(n))$, respectively. Here $\|x\|$ denotes the Euclidean norm of $x$ when $x$ is a vector and the spectral norm of $x$ when $x$ is a matrix. We write $\xi = \zeta + O_{\mathbb{P}}(f(n))$ or $\xi = \zeta + o_{\mathbb{P}}(f(n))$ if $\xi - \zeta = O_{\mathbb{P}}(f(n))$ or $\xi - \zeta = o_{\mathbb{P}}(f(n))$, respectively.
\end{remark}

\begin{lemma}
\label{LEM:PERFECT}
Let $(\mathbf{A}_n, \mathbf{X}_n) \sim \mathrm{GRDPG}_{p,q}(F)$ be a generalized random dot product graph on $n$ vertices
with sparsity factor $\rho_n$.
Let $\hat{\mathbf{U}}_n(i)$ and $\mathbf{U}_n(i)$ be the $i$-th row of $\hat{\mathbf{U}}_n$ and $\mathbf{U}_n$, respectively. Here $\hat{\mathbf{U}}_n$ and $\mathbf{U}_n$ are the eigenvectors of $\mathbf{A}_n$ and
$\mathbf{X}_n \mathbf{X}_n^{\top}$ corresponding to the $p + q$ largest eigenvalues (in modulus) of $\mathbf{A}_n$ and $\mathbf{X}_n \mathbf{X}_n^{\top}$. Then there exist a (universal) constant $c > 0$ and a $d \times d$
orthogonal matrix $\mathbf{W}_n$ such that, for $n \rho_n = \omega(\log^{2c}(n)$,
\begin{equation}
\label{eq:perfect}
\max_{i \in [n]} \|\mathbf{W}_n \hat{\mathbf{U}}_n(i) - \mathbf{U}_n(i)\| = O_{\mathbb{P}}\Bigl(\frac{\log^{c}{n}}{n \sqrt{\rho_n}}\Bigr).
\end{equation}
\end{lemma}
The proof of Lemma~\ref{LEM:PERFECT} is given in the appendix.
If $\mathbf{A}_n \sim \mathrm{SBM}(\mathbf{B}, \bm{\pi})$ with sparsity factor $\rho_n$ and $\mathbf{B}$ is $K \times K$, then the rows of $\mathbf{U}_n$ take on at most $K$ possible distinct values. Moreover, for any vertices $i$ and $j$ with $\tau_i \not = \tau_j$, $\|\mathbf{U}_n(i) - \mathbf{U}_n(j)\| \geq Cn^{-1/2}$ for some constant $C$ depending only on $\mathbf{B}$. Now if $n \rho_n = \omega(\log^{2c}(n))$, then Lemma~\ref{LEM:PERFECT} implies, for sufficiently large $n$,
$$\|\mathbf{W}_n \hat{\mathbf{U}}_n(i) - \mathbf{U}_n(i)\| < \min_{j \colon \tau_j \not = i}\|\mathbf{W}_n \hat{\mathbf{U}}_n(i) - \mathbf{U}_n(j)\|; \quad \text{for all $i \in [n]$}.$$
Hence, since $\mathbf{W}_n$ is an orthogonal matrix, $K$-means clustering of the rows of $\hat{\mathbf{U}}_n$ yield an assignment $\hat{\bm{\tau}}$ that is indeed, up to a permutation of the block labels, an exact recovery of $\bm{\tau}$ as $n \rightarrow \infty$. We note that Lemma~\ref{LEM:PERFECT} is an extension of our earlier results on bounding the perturbation $\hat{\mathbf{U}}_n - \mathbf{U}_n \mathbf{W}$ using the $2 \to \infty$ matrix norm \cite{perfect,lyzinski15_HSBM,ctp_2_to_infty}. Lemma~\ref{LEM:PERFECT} is very similar in flavor to a few recent results by other researchers \cite{abbe2,mao_sarkar,belkin_2_infty} where eigenvector perturbations of $\mathbf{A}_n$ (compared to the eigenvectors of $\mathbf{X}_n \mathbf{X}_n^{\top}$)
in the $\ell_{\infty}$ norm is established in the regime where $n \rho_n = \omega(\log^{c}(n))$ for some constant $c > 0$.
\begin{corollary}
\label{cor:in_practice}
Assume the setting and notations of Theorem~\ref{THM:GEN_D}. Assume $K$ known, let $\hat{\bm{\tau}} \colon [n] \mapsto [K]$ be the vertex to cluster assignments when the rows of $\hat{\mathbf{U}}$ are clustered into $K$ clusters. For $k \in [K]$, let $\hat{\bm{s}}_k \in \{0,1\}^{n}$ where the $i$-th entry of $\hat{\bm{s}}_k$ is $1$ if $\hat{\tau}_i = k$ and $0$ otherwise. Let $\hat{n}_k = |\{i \colon \hat{\tau}_i = k \}|$ and let $\hat{\pi}_k = \tfrac{\hat{n}_k}{n}$. For $k \in [K]$, let $\hat{\nu}_k = \tfrac{1}{\hat{n}_k} \hat{\bm{s}}_k^{\top} \hat{\mathbf{U}} \hat{\bm{\Lambda}}^{1/2}$,
let $\hat{\mathbf{B}}_{k \ell} = \hat{\mathbf{B}}_{k \ell}^{(S)} = \hat{\nu}_k^{\top} \mathbf{I}_{p,q} \hat{\nu}_{\ell}$, and
let $\hat{\Delta} = \sum_{k} \hat{\pi}_k \hat{\nu}_k \hat{\nu}_k^{\top}$. For $k \in [K]$ and $\ell \in [K]$, let $\hat{\theta}_{k \ell}$ be given by
\begin{equation}
 \label{eq:hat.mu_kl}
 \begin{split}
\theta_{k\ell} &= \sum_{r=1}^{K} \hat{\pi}_r \bigl(\hat{\mathbf{B}}_{kr} (1 - \hat{\mathbf{B}}_{kr}) + \hat{\mathbf{B}}_{\ell r}(1 - \hat{\mathbf{B}}_{\ell r})\bigr)\hat{\nu}_k^{\top} \hat{\Delta}^{-1} \mathbf{I}_{p,q} \hat{\Delta}^{-1} \nu_{\ell} \\
& - \sum_{r=1}^{K} \sum_{s=1}^{K} \hat{\pi}_r \hat{\pi}_s \hat{\mathbf{B}}_{rs} (1 - \hat{\mathbf{B}}_{sr}) \hat{\nu}_s^{\top} \hat{\Delta}^{-1} \mathbf{I}_{p,q} \hat{\Delta}^{-1}
(\hat{\nu}_{\ell} \hat{\nu}_k^{\top} + \hat{\nu}_k \hat{\nu}_{\ell}^{\top}) \hat{\Delta}^{-1} \hat{\nu}_s.

 \end{split}
 \end{equation}
 Then there exists a (sequence of) permutation(s) $\psi \equiv \psi_n$ on $[K]$ such that for any $k \in [K]$ and $\ell \in [K]$,
 \begin{equation}
 \label{eq:sbm2_inpractice}
 n(\hat{\mathbf{B}}^{(S)}_{\psi(k), \psi(\ell)} - \mathbf{B}_{k \ell} - \tfrac{\hat{\theta}_{k \ell}}{n}) \overset{\mathrm{d}}{\longrightarrow} \mathcal{N}(0, \sigma_{k \ell}^2)
 \end{equation}
as $n \rightarrow \infty$.
\end{corollary}
An almost identical result hold in the setting when $\rho_n \rightarrow 0$. More specifically, assume the setting and notations of Theorem~\ref{THM:GEN_D_SPARSE} and
let $\hat{\nu}_k$, $\hat{\Delta}$ and $\hat{\mathbf{B}} = \hat{\mathbf{B}}^{(S)}$ be as defined in Corllary~\ref{cor:in_practice}. Now let $\hat{\theta}_{k \ell}$ be given by
\begin{equation}
 \label{eq:hat.mu_kl2}
 \begin{split}
\hat{\theta}_{k\ell} &= \sum_{r=1}^{K} \hat{\pi}_r \bigl(\hat{\mathbf{B}}_{kr} + \hat{\mathbf{B}}_{\ell r} \bigr)\hat{\nu}_k^{\top} \hat{\Delta}^{-1} \mathbf{I}_{p,q} \hat{\Delta}^{-1} \nu_{\ell} \\
& - \sum_{r=1}^{K} \sum_{s=1}^{K} \hat{\pi}_r \hat{\pi}_s \hat{\mathbf{B}}_{rs} \hat{\nu}_s^{\top} \hat{\Delta}^{-1} \mathbf{I}_{p,q} \hat{\Delta}^{-1}
(\hat{\nu}_{\ell} \hat{\nu}_k^{\top} + \hat{\nu}_k \hat{\nu}_{\ell}^{\top}) \hat{\Delta}^{-1} \hat{\nu}_s.

 \end{split}
 \end{equation}
 Then there exists a (sequence of) permutation(s) $\psi \equiv \psi_n$ on $[K]$ such that for any $k \in [K]$ and $\ell \in [K]$,
 \begin{equation}
 n \rho_n^{1/2} (\hat{\mathbf{B}}_{\psi(k), \psi(\ell)} - \mathbf{B}_{k \ell} - \tfrac{\hat{\theta}_{k \ell}}{n \rho_n}) \overset{\mathrm{d}}{\longrightarrow} \mathcal{N}(0, \tilde{\sigma}_{k \ell}^2)
 \end{equation}
as $n \rightarrow \infty$, $\rho_n \rightarrow 0$ and $n \rho_n = \omega(\sqrt{n}).$

Finally, we provide some justification on the necessity of the assumption $n \rho_n = \omega(\sqrt{n})$ in the statement of Theorem~\ref{THM:GEN_D_SPARSE}, even though Lemma~\ref{LEM:PERFECT} implies that $\hat{\bm{\tau}}$ is an exact recovery of $\bm{\tau}$ for $n \rho_n = \omega(\log^{2c}(n))$.
Consider the case of $\mathbf{A}$ being an Erd\H{o}s-R\'{e}nyi graph on $n$ vertices with edge probability $p$.
The estimate $\hat{p}$ obtained from the spectral embedding in this setting is $\tfrac{1}{n^2} \hat{\lambda} (\bm{1}^{\top} \hat{\bm{u}})^2$ where $\hat{\lambda}$ is the largest eigenvalue of $\mathbf{A}$, $\bm{1}$ is the all ones vector, and $\hat{\bm{u}}$ is the associated (unit-norm) eigenvector. Let $\bm{e} = n^{-1/2} \bm{1}$.
We then have
\begin{equation*}
\begin{split}
n (\hat{p} - p) &= \tfrac{1}{n} \hat{\lambda} (\bm{1}^{\top} \hat{\bm{u}})^2 - np = \hat{\lambda} \bigl((\bm{e}^{\top} \hat{\bm{u}})^2 - 1\bigr) + \hat{\lambda} - np.
\end{split}
\end{equation*}
When $p$ remains constant as $n$ changes, then the results of \cite{furedi1981eigenvalues} implies $\bm{e}^{\top} \hat{\bm{u}} = 1 - \tfrac{1-p}{2np} + O_{\mathbb{P}}(n^{-3/2})$ and $\hat{\lambda} - np = \bm{e}^{\top} (\mathbf{A} - \mathbb{E}[\mathbf{A}]) \bm{e} + (1-p) + O_{\mathbb{P}}(n^{-1/2})$, from which we infer
\begin{equation*}
\begin{split}
n (\hat{p} - p) &= - (1-p) \tfrac{\hat{\lambda}}{np} + \bm{e}^{\top} (\mathbf{A} - \mathbb{E}[\mathbf{A}]) \bm{e} + (1-p) + O_{\mathbb{P}}(n^{-1/2}) \\ &= \bm{e}^{\top} (\mathbf{A} - \mathbb{E}[\mathbf{A}]) \bm{e} + O_{\mathbb{P}}(n^{-1/2}) \overset{\mathrm{d}}{\longrightarrow} \mathcal{N}(0, 2p(1-p))
\end{split}
\end{equation*}
since $\bm{e}^{\top} (\mathbf{A} - \mathbb{E}[\mathbf{A}]) \bm{e}$ is a sum of $n(n+1)/2$ independent mean $0$ random variables with variance $p(1-p)$.
On the other hand, if $p \rightarrow 0$ as $n$ increases, then Theorem~6.2 of \cite{erdos} (more specifically Eq.~(6.9) and Eq.~(6.26) of \cite{erdos}) implies
\begin{gather}
\label{eq:et_hatu}
\bm{e}^{\top} \hat{\bm{u}} = 1 - \tfrac{1-p}{2np} + O_{\mathbb{P}}\bigl((np)^{-3/2} + \tfrac{\log^{c}n}{n \sqrt{p}}\bigr)
\end{gather}
and
\begin{equation}
\label{eq:lambda_ER}
\begin{split}
 \hat{\lambda} - np &= \bm{e}^{\top} (\mathbf{A} - \mathbf{E}[\mathbf{A}]) \bm{e} + \tfrac{\bm{e}^{\top} (\mathbf{A} - \mathbb{E}[\mathbf{A}])^2 \bm{e}}{np} + O_{\mathbb{P}}\bigl((np)^{-1} + \tfrac{\log^{c}{n}}{n \sqrt{p}}) \\
 &= \bm{e}^{\top} (\mathbf{A} - \mathbf{E}[\mathbf{A}]) \bm{e} + (1 - p) + O_{\mathbb{P}}\bigl((np)^{-1} + \tfrac{\log^{c}{n}}{n \sqrt{p}}).
 \end{split}
\end{equation}
The second equality in Eq.~\eqref{eq:lambda_ER} follows from Lemma~6.5 of \cite{erdos} which states that $\bm{e}^{\top} (\mathbf{A} - \mathbf{E}[\mathbf{A}])^{k} \bm{e} = \bm{e}^{\top} \mathbb{E}[(\mathbf{A} - \mathbb{E}[\mathbf{A}])^{k}] \bm{e} + O_{\mathbb{P}}(\tfrac{(np)^{k/2} \log^{kc}(n)}{\sqrt{n}})$ for some universal constant $c > 0$ provided that $n p = \omega(\log{n})$. Hence
\begin{equation}
n(\hat{p} - p) = - (1-p) \tfrac{\hat{\lambda}}{np} + \bm{e}^{\top} (\mathbf{A} - \mathbf{E}[\mathbf{A}]) \bm{e} + (1-p) + O_{\mathbb{P}}((np)^{-1/2}).
\end{equation}
Once again $\bm{e}^{\top} (\mathbf{A} - \mathbf{E}[\mathbf{A}]) \bm{e}$ is a sum of $n(n+1)/2$ independent mean $0$ random variables with variance $p(1-p)$, but since $p \rightarrow 0$, the individual variance also vanishes as $n \rightarrow \infty$. In order to obtain a non-degenerate limiting distribution for $\bm{e}^{\top} (\mathbf{A} - \mathbf{E}[\mathbf{A}]) \bm{e}$, it is necessary that we consider $p^{-1/2} \bm{e}^{\top} (\mathbf{A} - \mathbf{E}[\mathbf{A}]) \bm{e}$. This, however, lead to non-trivial technical difficulties. In particular,
\begin{equation*}
\begin{split}
n p^{-1/2}(\hat{p} - p) &= p^{-1/2} \bm{e}^{\top} (\mathbf{A} - \mathbf{E}[\mathbf{A}]) \bm{e} + (1-p)(\tfrac{\hat{\lambda} - np}{np^{3/2}}) + O_{\mathbb{P}}(n^{-1/2} p^{-1}) \\
&= p^{-1/2} \bm{e}^{\top} (\mathbf{A} - \mathbf{E}[\mathbf{A}]) \bm{e} \bigl(1 + \tfrac{1-p}{np}\bigr) + O_{\mathbb{P}}(n^{-1/2} p^{-1})
\end{split}
\end{equation*}
upon iterating the term $(\hat{\lambda} - np)$. To guarantee that $O_{\mathbb{P}}(n^{-1/2} p^{-1})$ vanishes in the above expression,
it might be necessary to require $n p = \omega(\sqrt{n})$. That is to say, the expansions for $\bm{e}^{\top} \hat{\bm{u}}$ and $\hat{\lambda} - np$ in Eq.~\eqref{eq:et_hatu} and Eq.~\eqref{eq:lambda_ER} is not sufficiently refined.

We surmise that to extend Theorem~\ref{THM:GEN_D_SPARSE}, even in the context of Erd\H{o}s-R\'{e}nyi graphs, to the setting wherein $n p = o(\sqrt{n})$, it is necessary to consider higher order expansion for $\bm{e}^{\top} \hat{\bm{u}}$ and $\hat{\lambda} - np$. But this necessitates evaluating $\bm{e}^{\top} \mathbb{E}[(\mathbf{A} - \mathbb{E}[\mathbf{A}])^{k}] \bm{e}$ for $k \geq 3$, a highly non-trivial task; in particular $n p = \omega(\log{n})$ potentially require evaluating $\mathbb{E}[\bm{e}^{\top}(\mathbf{A} - \mathbb{E}[\mathbf{A}])^{k} \bm{e}]$ for $k = O(\log{n})$. In a slightly related vein,
\cite{zongmingma} evaluates $\mathrm{tr}[\mathbb{E}[(\mathbf{A} - \mathbb{E}[\mathbf{A}])^{k}$ in the case of Erd\H{o}s-R\'{e}nyi graphs and two-blocks planted partition SBM graphs.

Exact recovery of $\bm{\tau}$ via $\hat{\bm{\tau}}$ is therefore not sufficient to guarantee control of $\hat{\mathbf{B}}^{(S)}_{k \ell} - \mathbf{B}_{k \ell} = \bm{s}_k^{\top} (\hat{\mathbf{U}} \hat{\bm{\Lambda}} \hat{\mathbf{U}}^{\top} - \mathbb{E}[\mathbf{A}]) \bm{s}_{\ell}$. In essence, as $\rho_n \rightarrow 0$, the bias incurred by the low-rank approximation $
\hat{\mathbf{U}} \hat{\bm{\Lambda}} \hat{\mathbf{U}}^{\top}$ of $\mathbf{A}$ overwhelms the reduction in variance resulting from the low-rank approximation.


\section{Introduction}
\label{sec:intro}

Diachronic \textit{Lexical Semantic Change (LSC)} detection, i.e., the automatic detection of word sense changes over time, is a flourishing new field within NLP \citep[i.a.]{Frermann:2016,Hamilton:2016,schlechtweg-EtAl:2017:CoNLL}.\footnote{An example for diachronic LSC is the German noun \textit{Vorwort} \citep{Paul02XXI}, which was mainly used in the meaning of `preposition' before $\approx$1800. Then \textit{Vorwort} rapidly acquired a new meaning `preface', which after 1850 has nearly exclusively been used.} Yet, it is hard to compare the performances of the various models, and optimal parameter choices remain unclear, because up to now most models have been compared on different evaluation tasks and data. Presently, we do not know which model performs best under which conditions, and if more complex model architectures gain performance benefits over simpler models. This situation hinders advances in the field and favors unfelicitous drawings of statistical laws of diachronic LSC \citep{dubossarsky2017}.

In this study, we provide the first large-scale evaluation of an extensive number of approaches. Relying on an existing German LSC dataset we compare models regarding different combinations of semantic representations, alignment techniques and detection measures, while exploring various pre-processing and parameter settings. Furthermore, we introduce \textit{Word Injection} to LSC, a modeling idea drawn from term extraction, that overcomes the problem of vector space alignment. Our comparison of state-of-the-art approaches identifies best models and optimal parameter settings, and it suggests modifications to existing models which consistently show superior performance.

Meanwhile, the detection of lexical sense divergences across time-specific corpora is not the only possible application of LSC detection models. In more general terms, they have the potential to detect sense divergences between corpora of any type, not necessarily time-specific ones. We acknowledge this observation and further explore a \textit{synchronic LSC detection} task: identifying domain-specific changes of word senses in comparison to general-language usage, which is addressed, e.g., in term identification and automatic term extraction \cite{drouin2004,perez2016measuring,Haetty18LaymenStudy}, and in determining social and dialectal language variations \cite{del2017semantic,hovy/purschke:emnlp18}.\footnote{An example for domain-specific synchronic LSC is the German noun \textit{Form}. In general-language use, \textit{Form} means `shape'/`form', while in the cooking domain the predominant meaning is the domain-specific `baking tin'.}

For addressing the synchronic LSC task, we present a recent sense-specific term dataset \cite{haettySurel:2019} that we created analogously to the existing diachronic dataset, and we show that the diachronic models can be successfully applied to the synchronic task as well. This two-fold evaluation assures robustness and reproducibility of our model comparisons under various conditions.

\section{Related Work}
\label{sec:previous}

\paragraph{Diachronic LSC Detection.} Existing approaches for diachronic LSC detection are mainly based on three types of meaning representations: (i) semantic vector spaces, (ii) topic distributions, and (iii) sense clusters. In (i), semantic vector spaces, each word is represented as two vectors reflecting its co-occurrence statistics at different periods of time \citep[][]{Gulordava11,Kim14,Xu15,Eger:2016,HamiltonShiftDrift,Hamilton:2016,Hellrich16p2785,RosenfeldE18}. LSC is typically measured by the cosine distance (or some alternative metric) between the two vectors, or by differences in contextual dispersion between the two vectors \citep{kisselewetal:2016,schlechtweg-EtAl:2017:CoNLL}.
(ii) Diachronic topic models infer a probability distribution for each word over different word senses (or topics), which are in turn modeled as a distribution over words \citep[][]{Wang06,Bamman11p1,Wijaya11p35,Lau12p591,Mihalcea12,Cook14p1624,Frermann:2016}. LSC of a word is measured by calculating a novelty score for its senses based on their frequency of use.
(iii) Clustering models assign all uses of a word into sense clusters based on some contextual property \citep{Mitra15p773}. Word sense clustering models are similar to topic models in that they map uses to senses. Accordingly, LSC of a word is measured similarly as in (ii). For an overview on diachronic LSC detection, see \citet{2018arXiv181106278T}.

\paragraph{Synchronic LSC Detection.} We use the term synchronic LSC to refer to NLP research areas with a focus on how the meanings of words vary across domains or communities of speakers.
Synchronic LSC per se is not widely researched; for meaning shifts across domains, there is strongly related research which is concerned with domain-specific word sense disambiguation \cite{maynard1998term, chen2006context, Taghipour15, daille2016ambiguity} or term ambiguity detection \cite{Baldwin13TermAmbiguity,wang2013automatic}. The only notable work for explicitly measuring across domain meaning shifts is \citet{ferrari2017detecting}, which is based on semantic vector spaces and cosine distance.
Synchronic LSC across communities has been investigated as meaning variation in online communities, leveraging the large-scale data which has become available thanks to online social platforms \cite{del2017semantic,rotabi2017competition}.

\paragraph{Evaluation.} Existing evaluation procedures for LSC detection can be distinguished into evaluation on (i) empirically observed data, and (ii) synthetic data or related tasks. (i) includes case studies of individual words \citep[][]{Sagi09p104,Jatowt:2014,HamiltonShiftDrift}, stand-alone comparison of a few hand-selected words \citep[][]{Wijaya11p35,Hamilton:2016,del2017semantic}, comparison of hand-selected changing vs. semantically stable words \citep[][]{Lau12p591,Cook14p1624}, and post-hoc evaluation of the predictions of the presented models \citep[][]{Cook10,Kulkarni14,del2016tracing,Eger:2016,ferrari2017detecting}.
\citet{schlechtweg-EtAl:2017:CoNLL} propose a small-scale annotation of diachronic metaphoric change.

Synthetic evaluation procedures (ii) include studies that simulate LSC \citep{Cook10,Kulkarni14,RosenfeldE18}, evaluate sense assignments in WordNet \citep{Mitra15p773,Frermann:2016}, identify text creation dates, \citep[][]{Mihalcea12,Frermann:2016}, or predict the log-likelihood of textual data \citep[][]{Frermann:2016}.

Overall, the various studies use different evaluation tasks and data, with little overlap. Most evaluation data has not been annotated. Models were rarely compared to previously suggested ones, especially if the models differed in meaning representations. Moreover, for the diachronic task, synthetic datasets are used which do not reflect actual diachronic changes.

\section{Task and Data}
\label{sec:task}

Our study makes use of the evaluation framework proposed in \citet{Schlechtwegetal18}, where diachronic LSC detection is defined as a comparison between word uses in two time-specific corpora. We further applied the framework to create an analogous synchronic LSC dataset that compares word uses across general-language and domain-specific corpora. The common, meta-level task in our diachronic+synchronic setup is, given two corpora $C_a$ and $C_b$, to rank the targets in the respective datasets according to their degree of relatedness between word uses in $C_a$ and $C_b$.

\subsection{Corpora}
\label{subsec:corpora}

\paragraph{\textsc{DTA}} \citep{dta2017} is a freely available lemmatized, POS-tagged and spelling-normalized diachronic corpus of German containing texts from the 16th to the 20th century.

\vspace{-1mm}
\paragraph{\textsc{Cook}} is a domain-specific corpus. We crawled cooking-related texts from several categories (recipes, ingredients, cookware and cooking techniques) from the German cooking recipes websites \textit{kochwiki.de} and \textit{Wikibooks Kochbuch}\footnote{\url{de.wikibooks.org/wiki/Kochbuch}}.

\vspace{-1mm}
\paragraph{\textsc{SdeWaC}} \citep{faasseckart2012} is a cleaned version of the web-crawled corpus \textsc{deWaC} \citep{baroni2009wacky}. We reduced \textsc{SdeWaC} to \nicefrac{1}{8}th of its original size by selecting every 8th sentence for our general-language corpus.
\vspace{+2mm}\\
Table \ref{tab:corpSizes} summarizes the corpus sizes after applying pre-processing.
See Appendix \ref{sec:parameter} for pre-processing details.

\begin{table}[]
\centering
\begin{adjustbox}{width=0.45\textwidth}
\small
\begin{tabular}{lrrrr}
& \multicolumn{2}{c}{\textbf{Times}}&\multicolumn{2}{c}{\textbf{Domains}} \\
\hline
        & \multicolumn{1}{c}{\textsc{Dta18}} & \multicolumn{1}{c}{\textsc{Dta19}}
        & \multicolumn{1}{c}{\textsc{SdeWaC}} & \multicolumn{1}{c}{\textsc{Cook}} \\
\hline
\textsc{L\textsubscript{all}}  & 26M  &  40M & 109M   & 1M     \\
\textsc{L/P\textsubscript{}}  & 10M  & 16M & 47M & 0.6M     \\
\end{tabular}
\end{adjustbox}
\caption{Corpora and their approximate sizes.}\label{tab:corpSizes}
\end{table}

\subsection{Datasets\footnote{The datasets are available in Appendix \ref{sec:supplemental} and at \url{https://github.com/Garrafao/LSCDetection}.} and Evaluation}
\label{subsec:datasets}

\paragraph{Diachronic Usage Relatedness (DURel).} DURel is a gold standard for diachronic LSC consisting of 22 target words with varying degrees of LSC \citep{Schlechtwegetal18}. Target words were chosen from a list of attested changes in a diachronic semantic dictionary \citep{Paul02XXI}, and for each target a random sample of use pairs from the DTA corpus was annotated for meaning relatedness of the uses on a scale from 1 (unrelated meanings) to 4 (identical meanings), both within and across the time periods 1750--1799 and 1850--1899. The annotation resulted in an average Spearman's $\rho=0.66$ across five annotators and 1,320 use pairs. For our evaluation of diachronic meaning change we rely on the ranking of the target words according to their mean usage relatedness across the two time periods.

\paragraph{Synchronic Usage Relatedness (SURel).} SURel is a recent gold standard for synchronic LSC \citep{haettySurel:2019} using the same framework as in DURel. The 22 target words were chosen such as to exhibit different degrees of domain-specific meaning shifts, and use pairs were randomly selected from \textsc{SdeWaC} as general-language corpus and from \textsc{Cook} as domain-specific corpus. The annotation for usage relatedness across the corpora resulted in an average Spearman's $\rho=0.88$ across four annotators and 1,320 use pairs. For our evaluation of synchronic meaning change we rely on the ranking of the target words according to their mean usage relatedness between general-language and domain-specific uses.

\paragraph{Evaluation.} The gold LSC ranks in the DURel and SURel datasets are used to assess the correctness of model predictions by applying Spearman's rank-order correlation coefficient $\rho$ as evaluation metric, as done in similar previous studies \citep{Gulordava11,schlechtweg-EtAl:2017:CoNLL,SchlechtwegWalde18}. As corpus data underlying the experiments we rely on the corpora from which the annotated use pairs were sampled: \textsc{DTA} documents from 1750--1799 as $C_a$ and documents from 1850--1899 as $C_b$ for the diachronic experiments, and the \textsc{SdeWaC} corpus as $C_a$ and the \textsc{Cook} corpus as $C_b$ for the synchronic experiments.

\section{Meaning Representations\footnote{Find the hyperparameter settings in Appendix \ref{sec:parameter}. The scripts for vector space creation, alignment, measuring LSC and evaluation are available at \url{https://github.com/Garrafao/LSCDetection}.}}
\label{sec:repres}

Our models are based on two families of distributional meaning representations:  semantic vector spaces (Section~\ref{subsec:spaces}), and topic distributions (Section~\ref{subsec:topic}). All representations are bag-of-words-based, i.e. each word representation reflects a weighted bag of context words. The contexts of a target word $w_i$ are the words surrounding it in an $n$-sized window: $w_{i-n}, ..., w_{i-1}, w_{i+1}, ..., w_{i+n}$.

\subsection{Semantic Vector Spaces}
\label{subsec:spaces}

A semantic vector space constructed from a corpus $C$ with vocabulary $V$ is a matrix $M$, where each row vector represents a word $w$ in the vocabulary $V$ reflecting its co-occurrence statistics \citep{turney2010frequency}. We compare two state-of-the-art approaches to learn these vectors from co-occurrence data, (i) counting and (ii) predicting, and construct vector spaces for each time period and domain.

\subsubsection{Count-based Vector Spaces}

In a count-based semantic vector space the matrix $M$ is high-dimensional and sparse. The value of each matrix cell $M_{i,j}$ represents the number of co-occurrences of the word $w_i$ and the context $c_j$, $\#(w_i,c_j)$. In line with \citet{Hamilton:2016} we apply a number of transformations to these raw co-occurrence matrices, as previous work has shown that this improves results on different tasks \cite{Bullinaria2012,Levy2015}.

\vspace{+1mm}
\paragraph{Positive Pointwise Mutual Information (PPMI).} In PPMI representations the co-occurrence counts in each matrix cell $M_{i,j}$ are weighted by the positive mutual information of target $w_i$ and context $c_j$ reflecting their degree of association. The values of the transformed matrix are

\vspace{-3mm}
\small
\begin{equation*}
{ M}^{\textrm{PPMI}}_{i,j} = \max\left\lbrace\log\left(\frac{\#(w_i,c_j)\sum_c \#(c)^{\alpha}}{ \#(w_i)\#(c_j)^{\alpha}}\right)-\log(k),0\right\rbrace,
\end{equation*}
\normalsize
where $k > 1$ is a prior on the probability of observing an actual occurrence of $(w_i, c_j)$ and $0 < \alpha < 1$ is a smoothing parameter reducing PPMI's bias towards rare words \citep{Levy:2014,Levy2015}.

\vspace{+1mm}
\paragraph{Singular Value Decomposition (SVD).} Truncated SVD finds the optimal rank $d$ factorization of matrix $M$ with respect to L2 loss \citep{Eckart1936}. We use truncated SVD to obtain low-dimensional approximations of the PPMI representations by factorizing ${ M}^{\textrm{PPMI}}$ into the product of the three matrices ${ U}{ \Sigma}{ V}^\top$. We keep only the top $d$ elements of $\Sigma$ and obtain

\vspace{-2mm}
\begin{equation*}
{M}^{\textrm{SVD}} = { U_d}{ \Sigma^{p}_{d}},
\end{equation*}
where $p$ is an eigenvalue weighting parameter \cite{Levy2015}. The $i$th row of ${ M}^{\textrm{SVD}}$ corresponds to $w_i$'s $d$-dimensional representation.

\vspace{+1mm}
\paragraph{Random Indexing (RI).} RI is a dimensionality reduction technique based on the Johnson-Lindenstrauss lemma according to which points in a vector space can be mapped into a randomly selected subspace under approximate preservation of the distances between points, if the subspace has a sufficiently high dimensionality \citep{Johnson1984,Sahlgren2004}. We reduce the dimensionality of a count-based matrix $M$ by multiplying it with a random matrix $R$:

\vspace{-2mm}
\begin{equation*}
{M}^{\textrm{RI}} = {M}{ R^{|\mathcal{V}| \times d}},
\end{equation*}
where the $i$th row of ${M}^{\textrm{RI}}$ corresponds to $w_i$'s $d$-dimensional semantic representation. The choice of the random vectors corresponding to the rows in $R$ is important for RI. We follow previous work \citep{Basile2015} and use sparse ternary random vectors with a small number $s$ of randomly distributed $-1$s and $+1$s, all other elements set to 0, and we apply subsampling with a threshold $t$.

\vspace{+1mm}
\subsubsection{Predictive Vector Spaces}

\vspace{+1mm}
\paragraph{Skip-Gram with Negative Sampling (SGNS)} differs from count-based techniques in that it directly represents each word $w \in V$ and each context $c \in V$ as a $d$-dimensional vector by implicitly factorizing $M=WC^\top$ when solving

\vspace{-3mm}
\small
\begin{equation*}
\arg\max_\theta \sum_{(w,c)\in D} \log \sigma(v_c \cdot v_w) + \sum_{(w,c) \in D'} \log \sigma (-v_c \cdot v_w),
\end{equation*}
\normalsize
where $\sigma(x) = \frac{1}{1+e^{-x}}$, $D$ is the set of all observed word-context pairs and $D'$ is the set of randomly generated negative samples \citep{Mikolov13a,Mikolov13b,GoldbergL14}. The optimized parameters $\theta$ are $v_{c_i}=C_{i*}$ and $v_{w_i}=W_{i*}$ for $w,c\in V$, $i\in 1,...,d$. $D'$ is obtained by drawing $k$ contexts from the empirical unigram distribution $P(c) = \frac{\#(c)}{|D|}$ for each observation of (w,c), cf. \citet{Levy2015}. SGNS and PPMI representations are highly related in that the cells of the implicitly factorized matrix $M$ are PPMI values shifted by the constant $k$ \citep{Levy:2014}. Hence, SGNS and PPMI share the hyper-parameter $k$. The final SGNS matrix is given by

\vspace{-2mm}
\begin{equation*}
{M}^{\textrm{SGNS}} = W,
\end{equation*}
where the $i$th row of ${M}^{\textrm{SGNS}}$ corresponds to $w_i$'s $d$-dimensional semantic representation. As in RI we apply subsampling with a threshold $t$. SGNS with particular parameter configurations has shown to outperform transformed count-based techniques on a variety of tasks \cite{marcobaroni2014predict,Levy2015}.

\vspace{+1mm}
\subsubsection{Alignment}

\vspace{+1mm}
\paragraph{Column Intersection (CI).} In order to make the matrices $A$ and $B$ from time periods $a < b$ (or domains $a$ and $b$) comparable, they have to be aligned via a common coordinate axis. This is rather straightforward for count and PPMI representations, because their columns correspond to context words which often occur in both $A$ and $B$ \citep{Hamilton:2016}. In this case, the alignment for $A$ and $B$ is

\begin{equation*}
\begin{split}
A_{*j}^{\textrm{CI}} = A_{*j}\textrm{~~ for all } c_j \in V_{a} \cap V_{b},\\
B_{*j}^{\textrm{CI}} = B_{*j}\textrm{~~ for all } c_j \in V_{a} \cap V_{b},\
\end{split}
\end{equation*}
where $X_{*j}$ denotes the $j$th column of $X$.

\vspace{+1mm}
\paragraph{Shared Random Vectors (SRV).} RI offers an elegant way to align count-based vector spaces and reduce their dimensionality at the same time \citep{Basile2015}. Instead of multiplying count matrices $A$ and $B$ each by a separate random matrix $R_A$ and $R_B$ they may be multiplied both by the same random matrix $R$ representing them in the same low-dimensional random space. Hence, $A$ and $B$ are aligned by
\begin{equation*}
\begin{split}
A^{\textrm{SVR}} = A R,\\
B^{\textrm{SVR}} = B R.
\end{split}
\end{equation*}
We follow \citeauthor{Basile2015} and adopt a slight variation of this procedure: instead of multiplying both matrices by exactly the same random matrix (corresponding to an intersection of their columns) we first construct a shared random matrix and then multiply $A$ and $B$ by the respective sub-matrix.

\vspace{+1mm}
\paragraph{Orthogonal Procrustes (OP).} In the low-dimensional vector spaces produced by SVD, RI and SGNS the columns may represent different coordinate axes (orthogonal variants) and thus cannot directly be aligned to each other. Following \citet{Hamilton:2016} we apply OP analysis to solve this problem. We represent the dictionary as a binary matrix D, so that $D_{i,j} = 1$ if $w_i \in V_b$ (the $i$th word in the vocabulary at time $b$) corresponds to $w_j \in V_a$. The goal is then to find the optimal mapping matrix $W^{*}$ such that the sum of squared Euclidean distances between $B$'s mapping
$B_{i*}W$ and $A_{j*}$ for the dictionary entries $D_{i,j}$ is minimized:
\begin{equation*}
W^{*} = \arg\min_{W} \sum_i\sum_j D_{i,j}\| B_{i*}W-A_{j*} \|^{2}.
\end{equation*}
Following standard practice we length-normalize and mean-center $A$ and $B$ in a pre-processing step \citep{artetxe2017acl}, and constrain $W$ to be orthogonal, which preserves distances within each time period. Under this constraint, minimizing the squared Euclidean distance becomes equivalent to maximizing the dot product when finding the optimal rotational alignment \citep{Hamilton:2016,artetxe2017acl}.
The optimal solution for this problem is then given by $W^{*}=UV^{\top}$, where $B^{\top} DA=U\Sigma V^{\top}$ is the SVD of $B^{\top} DA$. Hence, $A$ and $B$ are aligned by
\begin{equation*}
\begin{split}
&A^{\textrm{OP}} = A,\\
&B^{\textrm{OP}} = B W^{*},
\end{split}
\end{equation*}
where $A$ and $B$ correspond to their preprocessed versions. We also experiment with two variants: $\textrm{OP}_{-}$ omits mean-centering \citep{Hamilton:2016}, which is potentially harmful as a better solution may be found after mean-centering. $\textrm{OP}_{+}$ corresponds to OP with additional pre- and post-processing steps and has been shown to improve performance in research on bilingual lexicon induction \citep{artetxe2018aaai,artetxe2018acl}. We apply all OP variants only to the low-dimensional matrices.

\vspace{+1mm}
\paragraph{Vector Initialization (VI).} In VI we first learn $A^{\textrm{VI}}$ using standard SGNS and then initialize the SGNS model for learning $B^{\textrm{VI}}$ on $A^{\textrm{VI}}$ \citep{Kim14}. The idea is that if a word is used in similar contexts in $a$ and $b$, its vector will be updated only slightly, while more different contexts lead to a stronger update.

\vspace{+1mm}
\paragraph{Word Injection (WI).} Finally, we use the word injection approach by \citet{ferrari2017detecting} where target words are substituted by a placeholder in one corpus before learning semantic representations, and a single matrix $M^{\textrm{WI}}$ is constructed for both corpora after mixing their sentences. The advantage of this approach is that all vector learning methods described above can be directly applied to the mixed corpus, and target vectors are constructed directly in the same space, so no post-hoc alignment is necessary.

\vspace{+1mm}
\subsection{Topic Distributions}
\label{subsec:topic}

\vspace{+1mm}
\paragraph{Sense ChANge (SCAN).} SCAN models LSC of word senses via smooth and gradual changes in associated topics \citep{Frermann:2016}. The semantic representation inferred for a target word $w$ and time period $t$ consists of a $K$-dimensional distribution over word senses $\phi^{t}$ and a $V$-dimensional distribution over the vocabulary $\psi^{t,k}$ for each word sense $k$, where $K$ is a predefined number of senses for target word $w$. SCAN places parametrized logistic normal priors on $\phi^{t}$ and $\psi^{t,k}$ in order to encourage a smooth change of parameters, where the extent of change is controlled through the precision parameter $K^{\phi}$, which is learned during training.

Although $\psi^{t,k}$ may change over time for word sense $k$, senses are intended to remain thematically consistent as controlled by word precision parameter $K^{\psi}$. This allows comparison of the topic distribution across time periods. For each target word $w$ we infer a SCAN model for two time periods $a$ and $b$ and take $\phi^{a}_w$ and $\phi^{b}_w$ as the respective semantic representations.

\vspace{+1mm}
\section{LSC Detection Measures}
\label{sec:measures}

LSC detection measures predict a degree of LSC from two time-specific semantic representations of a word $w$. They either capture the contextual similarity (Section~\ref{sec:similarity}) or changes in the contextual dispersion (Section~\ref{subsec:dispersion}) of $w$'s representations.\footnote{Find an overview of which measure was applied to which representation type in Appendix \ref{sec:parameter}.}

\vspace{+1mm}
\subsection{Similarity Measures}
\label{sec:similarity}

\vspace{+2mm}
\paragraph{Cosine Distance (CD).} CD is based on cosine similarity which measures the cosine of the angle between two non-zero vectors $\vec{x},\vec{y}$ with equal magnitudes \cite{salton1986introduction}:
	\begin{equation*}
	cos(\vec{x},\vec{y}) = \frac{\vec{x} \cdot \vec{y}}{\sqrt{\vec{x} \cdot \vec{x}} \sqrt{\vec{y} \cdot \vec{y}}}.
	\end{equation*}
The cosine distance is then defined as
	\begin{equation*}
	CD(\vec{x},\vec{y})=1-cos(\vec{x},\vec{y}).
	\end{equation*}
CD's prediction for a degree of LSC of $w$ between time periods $a$ and $b$ is obtained by $CD(\vec{w}_a,\vec{w}_b)$.

\vspace{+1mm}
\paragraph{Local Neighborhood Distance (LND).} LND computes a second-order similarity for two non-zero vectors $\vec{x},\vec{y}$ \citep{HamiltonShiftDrift}. It measures the extent to which $\vec{x}$ and $\vec{y}$~'s distances to their shared nearest neighbors differ. First the cosine similarity of $\vec{x},\vec{y}$ with each vector in the union of the sets of their $k$ nearest neighbors $N_k(\vec{x})$ and $N_k(\vec{y})$ is computed and represented as a vector $s$ whose entries are given by
    \begin{equation*}
    \begin{split}
    s(j) = \text{cos}(\vec{x},\vec{z}_j) \quad \forall \vec{z}_j\in N_{k}(\vec{x}) \cup N_{k}(\vec{y}).
    \end{split}
    \end{equation*}

LND is then computed as cosine distance between the two vectors:
	\begin{equation*}
    LND(\vec{x},\vec{y}) = CD(\vec{s_x}, \vec{s_y}).
	\end{equation*}
LND does not require matrix alignment, because it measures the distances to the nearest neighbors in each space separately. It was claimed to capture changes in paradigmatic rather than syntagmatic relations between words \citep{HamiltonShiftDrift}.

\paragraph{Jensen-Shannon Distance (JSD).} JSD computes the distance between two probability distributions $\phi_x,\phi_y$ of words $w_x, w_y$ \citep{Lin1991,DonosoS17}. It is the symmetrized square root of the Kullback-Leibler divergence:

\small
\begin{equation*}
JSD(\phi_x||\phi_y)=\sqrt{\frac{D_{KL}(\phi_x||M)+D_{KL}(\phi_y||M)}{2}}\,,
\end{equation*}
\normalsize
where $M=(\phi_x+\phi_y)/2$. JSD is high if $\phi_x$ and $\phi_y$ assign different probabilities to the same events.

\begin{table*}[]
	\center
	\small
	\hspace*{-10pt}
	\begin{tabular}{ c | c  S  c  c  c  c | c }
		\hline
		\textbf{Dataset} & \textbf{Preproc} & \textbf{Win} & \specialcell{\textbf{Space}} & \specialcell{\textbf{Parameters}\\}
		& \specialcell{\textbf{Align}\\} & \specialcell{\textbf{Measure}}  &
		\textbf{Spearman  m (h, l)} \\ \hline

		\multirow{5}{*}{\textbf{DURel}} & \multirow{1}{*}{\textsc{L\textsubscript{all}}} &10& SGNS & k=1,t=None & $\textrm{OP}$ & CD & \textbf{0.866} \tiny (0.914, 0.816) \\
		&\multirow{1}{*}{\textsc{L\textsubscript{all}}} &10& SGNS & k=5,t=None & $\textrm{OP}$ & CD & 0.857 \tiny  (0.891, 0.830) \\
		&\multirow{1}{*}{\textsc{L\textsubscript{all}}} &5& SGNS & k=5,t=0.001 & $\textrm{OP}$ & CD & 0.835 \tiny (0.872, 0.814) \\
		&\multirow{1}{*}{\textsc{L\textsubscript{all}}} &10& SGNS & k=5,t=0.001 & $\textrm{OP}$ & CD & 0.826 \tiny (0.863, 0.768)  \\
		&\multirow{1}{*}{\textsc{L/P}} &2& SGNS & k=5,t=None & $\textrm{OP}$ & CD & 0.825 \tiny  (0.826, 0.818) \\
		\hline
		\multirow{5}{*}{\textbf{SURel}} & \textsc{L/P} &2& SGNS & k=1,t=0.001 & $\textrm{OP}$ & CD & \textbf{0.851} \tiny  (0.851, 0.851)  \\
		& \textsc{L/P} &2& SGNS & k=5,t=None & $\textrm{OP}$ & CD & 0.850 \tiny  (0.850, 0.850)
		  \\
		& \textsc{L/P} &2& SGNS & k=5,t=0.001 & $\textrm{OP}$ & CD & 0.834 \tiny  (0.838, 0.828)
		  \\
		 & \textsc{L/P} &2& SGNS & k=5,t=0.001 & $\textrm{OP}_{-}$ & CD & 0.831 \tiny  (0.836, 0.817)
		  \\
		  & \textsc{L/P} &2& SGNS & k=5,t=0.001 & $\textrm{OP}$ & CD & 0.829 \tiny  (0.832, 0.823)
		  \\  \hline
	\end{tabular}
	\caption{Best results of $\rho$ scores (Win=Window Size, Preproc=Preprocessing, Align=Alignment, k=negative sampling, t=subsampling, Spearman m(h,l): mean, highest and lowest results).}
	\label{tab:bestResults}
\end{table*}

\vspace{+2mm}
\subsection{Dispersion Measures}
\label{subsec:dispersion}

\vspace{+2mm}
\paragraph{Frequency Difference (FD).} The log-transformed relative frequency of a word $w$ for a corpus $C$ is defined by
	\begin{equation*}
    F(w,C)=\log \frac{|w\in C|}{|C|}
	\end{equation*}
FD of two words $x$ and $y$ in two corpora $X$ and $Y$ is then defined by the absolute difference in F:
	\begin{equation*}
    FD(x,X,y,Y)= |F(x,X)-F(y,Y)|
	\end{equation*}
FD's prediction for $w$'s degree of LSC between time periods $a$ and $b$ with corpora $C_a$ and $C_b$ is computed as $FD(w,C_a,w,C_b)$ (parallel below).

\vspace{+1mm}
\paragraph{Type Difference (TD).} TD is similar to FD, but based on word vectors $\vec{w}$ for words $w$. The normalized log-transformed number of context types of a vector $\vec{w}$ in corpus $C$ is defined by
	\begin{equation*}
    T(\vec{w},C) = \log \frac{\sum_{i=1} 1 \quad \textrm{if $\vec{w}_i \neq 0$}}{|C_T|} ,
	\end{equation*}
where $|C_T|$ is the number of types in corpus $C$. The TD of two vectors $\vec{x}$ and $\vec{y}$ in two corpora $X$ and $Y$ is the absolute difference in T:
	\begin{equation*}
    TD(\vec{x},X,\vec{y},Y)= |T(\vec{x},X)-T(\vec{y},Y)|.
	\end{equation*}

\vspace{+1mm}
\paragraph{Entropy Difference (HD).} HD relies on vector entropy as suggested by \citet{Santus:2014}. The entropy of a non-zero word vector $\vec{w}$ is defined by
	\begin{equation*}
    VH(\vec{w}) = -\sum_{i=1} \frac{\vec{w}_i}{\sum_{j=1} \vec{w}_j} \; \log \frac{\vec{w}_i}{\sum_{j=1} \vec{w}_j}.
	\end{equation*}
VH is based on Shannon's entropy \citep{Shannon:1948}, which measures the unpredictability of $w$'s co-occurrences \citep{schlechtweg-EtAl:2017:CoNLL}. HD is defined as
	\begin{equation*}
    HD(\vec{x},\vec{y}) = |VH(\vec{x})-VH(\vec{y})|.
	\end{equation*}
We also experiment with differences in H between topic distributions $\phi^{a}_w,\phi^{b}_w$, which are computed in a similar fashion, and with normalizing VH by dividing it by $\log(VT(\vec{w}))$, its maximum value.

\vspace{+2mm}
\section{Results and Discussions}
\label{sec:results}

First of all, we observe that nearly all model predictions have a strong positive correlation with the gold rank. Table \ref{tab:bestResults} presents the overall best results across models and parameters.\footnote{For models with randomness we computed the average results of five iterations.} With $\rho=0.87$ for diachronic LSC (DURel) and $\rho=0.85$ for synchronic LSC (SURel), the models reach comparable and unexpectedly high performances on the two distinct datasets. The overall best-performing model is Skip-Gram with orthogonal alignment and cosine distance (SGNS+OP+CD). The model is robust in that it performs best on both datasets and produces very similar, sometimes the same results across different iterations.

\vspace{+1mm}
\paragraph{Pre-processing and Parameters.} Regarding pre-processing, the results are less consistent: \textsc{L\textsubscript{all}} (all lemmas) dominates in the diachronic task, while \textsc{L/P} (lemma:pos of content words) dominates in the synchronic task. In addition, \textsc{L/P} pre-processing, which is already limited on content words, prefers shorter windows, while \textsc{L\textsubscript{all}} (pre-processing where the complete sentence structure is maintained) prefers longer windows. Regarding the preference of \textsc{L/P} for SURel, we blame noise in the \textsc{Cook} corpus, which contains a lot of recipes listing ingredients and quantities with numerals and abbreviations, to presumably contribute little information about context words. For instance, \textsc{Cook} contains $4.6\%$ numerals, while \textsc{DTA} only contains $1.2\%$ numerals.

Looking at the influence of subsampling, we find that it does not improve the mean performance for Skip-Gram (SGNS) (with $\rho=0.506$, without $\rho=0.517$), but clearly for Random Indexing (RI) (with $\rho=0.413$, without $\rho=0.285$). \citet{Levy2015} found that SGNS prefers numerous negative samples ($k>1$), which is confirmed here: mean $\rho$ with $k=1$ is $0.487$, and mean $\rho$ with $k=5$ is $0.535$.\footnote{For PPMI we observe the opposite preference, mean $\rho$ with $k=1$ is $0.549$ and mean $\rho$ with $k=5$ is $0.439$.} This finding is also indicated in Table \ref{tab:bestResults}, where $k=5$ dominates the 5 best results on both datasets; yet, $k=1$ provides the overall best result on both datasets.

\paragraph{Semantic Representations.} Table \ref{tab:spaces} shows the best and mean results for different semantic representations. SGNS is clearly the best vector space model, even though its mean performance does not outperform other representations as clearly as its best performance. Regarding count models, PPMI and SVD show the best results.

SCAN performs poorly, and its mean results indicate that it is rather unstable. This may be explained by the particular way in which SCAN constructs context windows: it ignores asymmetric windows, thus reducing the number of training instances considerably, in particular for large window sizes.

\vspace{+1mm}
\begin{table}[htp]
	\center
	\small
	\begin{adjustbox}{width=0.8\linewidth}
	\begin{tabular}{ c | l  c  c }
		\hline
	\textbf{Dataset}&\textbf{Representation} &\textbf{best}&\textbf{mean}\\
		\hline
		\multirow{6}{*}{\textbf{DURel}} & raw count & 0.639 & 0.395\\
		&  PPMI & 0.670 & 0.489\\
		&  SVD & 0.728 & 0.498\\
		&  RI & 0.601  & 0.374 \\
		&  SGNS & \textbf{0.866} & \textbf{0.502}\\
		& SCAN & 0.327 & 0.156 \\
		\hline
		\multirow{6}{*}{\textbf{SURel}} & raw count & 0.599 & 0.120\\
		&  PPMI & 0.791 & 0.500\\
		&  SVD & 0.639 & 0.300\\
		&  RI & 0.622 & 0.299 \\
		&  SGNS & \textbf{0.851} & \textbf{0.520} \\
		& SCAN & 0.082 & -0.244 \\
		\hline
	\end{tabular}
	\end{adjustbox}
	\caption{Best and mean $\rho$ scores across similarity measures (CD, LND, JSD) on semantic representations.}
	\label{tab:spaces}
\end{table}

\paragraph{Alignments.} The fact that our modification of \citet{Hamilton:2016} (SGNS+OP) performs best across datasets confirms our assumption that column-mean centering is an important preprocessing step in Orthogonal Procrustes analysis and should not be omitted.

Additionally, the mean performance in Table \ref{tab:CompOP} shows that OP is generally more robust than its variants. $\textrm{OP}_{+}$ has the best mean performance on DURel, but performs poorly on SURel. \citet{artetxe2018aaai} show that the additional pre- and post-processing steps of $\textrm{OP}_{+}$ can be harmful in certain conditions. We tested the influence of the different steps and identified the non-orthogonal whitening transformation as the  main reason for a performance drop of $\approx$20\%.

In order to see how important the alignment step is for the low-dimensional embeddings (SVD/RI/SGNS), we also tested the performance without alignment (`None' in Table \ref{tab:CompOP}). As expected, the mean performance drops considerably. However, it remains positive, which suggests that the spaces learned in the models are not random but rather slightly rotated variants.

Especially interesting is the comparison of Word Injection (WI) where one common vector space is learned against the OP-models where two separately learned vector spaces are aligned. Although WI avoids (post-hoc) alignment altogether, it is consistently outperformed by OP, which is shown in Table \ref{tab:CompOP} for low-dimensional embeddings.\footnote{We see the same tendency for WI against random indexing with a shared random space (SRV), but instead variable results for count and PPMI alignment (CI). This contradicts the findings in \citet{Dubossarskyetal19}, using, however, a different task and synthetic data.} We found that OP profits from mean-centering in the pre-processing step: applying mean-centering to WI matrices improves the performance by 3\% on WI+SGNS+CD.

The results for Vector Initialization (VI) are unexpectedly low (on DURel mean $\rho=-0.017$, on SURel mean $\rho=0.082$). An essential parameter choice for VI is the number of training epochs for the initialized model. We experimented with 20 epochs instead of 5, but could not improve the performance. This contradicts the results obtained by \citet{Hamilton:2016} who report a ``negligible'' impact of VI when compared to $\textrm{OP}_{-}$. We reckon that VI is strongly influenced by frequency. That is, the more frequent a word is in corpus C$_b$, the more its vector will be updated after initialization on C$_a$. Hence, VI predicts more change with higher frequency in C$_b$.

\vspace{+1mm}
\begin{table}[t]
	\centering
	\begin{adjustbox}{width=0.45\textwidth}
	\begin{tabular}{c|r r r |r |r }
	    \hline
\textbf{Dataset}&\textbf{$\textrm{OP}$}&\textbf{$\textrm{OP}_{-}$}&\textbf{$\textrm{OP}_{+}$}&\textbf{WI}&\textbf{None} \\
            \hline
            \textbf{DURel} &0.618&0.557&\textbf{0.621}&0.468&0.254\\
            \textbf{SURel} &\textbf{0.590}&0.514&0.401&0.492&0.285\\
		\hline
	\end{tabular}
	\end{adjustbox}
	\caption{Mean $\rho$ scores for CD across the alignments. Applies only to RI, SVD and SGNS.}
	\label{tab:CompOP}
\end{table}

\vspace{-1mm}
\paragraph{Detection Measures.} Cosine distance (CD) dominates Local Neighborhood Distance (LND) on all vector space and alignment types (e.g., mean $\rho$ on DURel with SGNS+OP is $0.723$ for CD vs. $0.620$ for LND) and hence should be generally preferred if alignment is possible. Otherwise LND or a variant of WI+CD should be used, as they show lower but robust results.\footnote{JSD was not included here, as it was only applied to SCAN and its performance thus strongly depends on the underlying meaning representation.} Dispersion measures in general exhibit a low performance, and previous positive results for them could not be reproduced \citep{schlechtweg-EtAl:2017:CoNLL}. It is striking that, contrary to our expectation, dispersion measures on SURel show a strong negative correlation (max. $\rho=-0.79$). We suggest that this is due to frequency particularities of the dataset: SURel's gold LSC rank has a rather strong negative correlation with the targets' frequency rank in the \textsc{Cook} corpus ($\rho=-0.51$). Moreover, because \textsc{Cook} is magnitudes smaller than \textsc{SdeWaC} the normalized values computed in most dispersion measures in \textsc{Cook} are much higher. This gives them also a much higher weight in the final calculation of the absolute differences. Hence, the negative correlation in \textsc{Cook} propagates to the final results. This is supported by the fact that the only measure not normalized by corpus size (HD) has a positive correlation. As these findings show, the dispersion measures are strongly influenced by frequency and very sensitive to different corpus sizes.

\paragraph{Control Condition.} As we saw, dispersion measures are sensitive to frequency. Similar observations have been made for other LSC measures \citep{dubossarsky2017}. In order to test for this influence within our datasets we follow \citet{dubossarsky2017} in adding a control condition to the experiments for which sentences are randomly shuffled across corpora (time periods). For each target word we merge all sentences from the two corpora $C_a$ and $C_b$ containing it, shuffle them, split them again into two sets while holding their frequencies from the original corpora approximately stable and merge them again with the original corpora. This reduces the target words' mean degree of LSC between $C_a$ and $C_b$ significantly. Accordingly, the mean degree of LSC predicted by the models should reduce significantly if the models measure LSC (and not some other controlled property of the dataset such as frequency). We find that the mean prediction on a result sample (\textsc{L/P}, win=2) indeed reduces from $0.5$ to $0.36$ on DURel and from $0.53$ to $0.44$ on SURel. Moreover, shuffling should reduce the correlation of individual model predictions with the gold rank, as many items in the gold rank have a high degree of LSC, supposedly being canceled out by the shuffling and hence randomizing the ranking. Testing this on a result sample (SGNS+OP+CD, \textsc{L/P}, win=2, k=1, t=None), as shown in Table \ref{tab:shuffle}, we find that it holds for DURel with a drop from $\rho=0.816$ (ORG) to $0.180$ on the shuffled (SHF) corpora, but not for SURel where the correlation remains stable ($0.767$ vs. $0.763$). We hypothesize that the latter may be due to SURel's frequency properties and find that downsampling all target words to approximately the same frequency in both corpora ($\approx 50$) reduces the correlation (+DWN). However, there is still a rather high correlation left ($0.576$). Presumably, other factors play a role: (i) Time-shuffling may not totally randomize the rankings because words with a high change still end up having slightly different meaning distributions in the two corpora than words with no change at all. Combined with the fact that the SURel rank is less uniformly distributed than DURel this may lead to a rough preservation of the SURel rank after shuffling. (ii) For words with a strong change the shuffling creates two equally polysemous sets of word uses from two monosemous sets. The models may be sensitive to the different variances in these sets, and hence predict stronger change for more polysemous sets of uses.
Overall, our findings demonstrate that much more work has to be done to understand the effects of time-shuffling as well as sensitivity effects of LSC detection models to frequency and polysemy.

\begin{table}[t]
	\centering
	\begin{adjustbox}{width=0.65\linewidth}
	\begin{tabular}{c|r| rr }
	    \hline
             \textbf{Dataset} & \textbf{ORG}  & \textbf{SHF} & \textbf{+DWN}  \\ \hline
            \textbf{DURel} & \textbf{0.816} & 0.180 & 0.372  \\
            \textbf{SURel} & \textbf{0.767} & 0.763 & 0.576  \\ \hline
	\end{tabular}
	\end{adjustbox}
	\caption{$\rho$ for SGNS+OP+CD (\textsc{L/P}, win=2, k=1, t=None) before (ORG) and after time-shuffling (SHF) and downampling them to the same frequency (+DWN).}
	\label{tab:shuffle}
\end{table}

\vspace{+1mm}
\section{Conclusion}
\label{sec:conclusion}

We carried out the first systematic comparison of a wide range of LSC detection models on two datasets which were reliably annotated for sense divergences across corpora. The diachronic and synchronic evaluation tasks we introduced were solved with impressively high performance and robustness. We introduced \textit{Word Injection} to overcome the need of (post-hoc) alignment, but find that Orthogonal Procrustes yields a better performance across vector space types.

The overall best performing approach on both data suggests to learn vector representations for different time periods (or domains) with SGNS, to align them with an orthogonal mapping, and to measure change with cosine distance. We further improved the performance of the best approach with the application of mean-centering as an important pre-processing step for rotational vector space alignment.

\section*{Acknowledgments}

The first author was supported by the Konrad Adenauer Foundation and the CRETA center funded by the German Ministry for Education and Research (BMBF) during the conduct of this study. We thank Haim Dubossarsky, Simon Hengchen, Andres Karjus, Barbara McGillivray, Cennet Oguz, Sascha Schlechtweg, Nina Tahmasebi and the three anonymous reviewers for their valuable comments. We further thank Michael Dorna and Bingqing Wang for their helpful advice. We also thank Lea Frermann for providing the code of SCAN and helping to set up the implementation.