\subsection{Sample Questions \label{sec:Sample-Questions}}
\begin{itemize}
\item \textbf{Q1} (\emph{Ordinal}): How would you describe your day today\textemdash{}has
it been a typical day, a particularly good day, or a particularly
bad day?
\item \textbf{Q7} (\emph{Binary}): Now thinking about our country, overall,
are you satisfied or dissatisfied with the way things are going in
our country today?
\item \textbf{Q5} (\emph{Multicategorical}): What do you think is the most
important problem facing you and your family today? \{Economic problems
/ Housing / Health / Children and education/Work/Social relations
/ Transportation / Problems with government / Crime / Terrorism and
war / No problems / Other / Don't know / Refused\}
\item \textbf{Q10,11} (\emph{Category-ranking}): In your opinion, which
one of these poses the greatest/second greatest threat to the world:
\{the spread of nuclear weapons / religious and ethnic hatred/AIDS
and other infectious diseases / pollution and other environmental
problems / or the growing gap between the rich and poor\}?
\item \textbf{Q74} (\emph{Continuous}): How old were you at your last birthday?
\item \textbf{Q91} (\emph{Categorical}): Are you currently married or living
with a partner, widowed, divorced, separated, or have you never been
married?
\end{itemize}

\subsection{Mean-field Approximation \label{sub:Mean-field-Approximation}}

We present here a simplification of $P\left(v_{i}\mid\vb_{\neg C}\right)$
in Eq.~(\ref{eq:predictive-distribution}) using the mean-field approximation.
Recall that $P\left(v_{i}\mid\vb_{\neg C}\right)=\sum_{\hb}P\left(v_{i},\hb\mid\vb_{\neg C}\right)$,
where $P\left(v_{i},\hb\mid\vb_{\neg C}\right)$ is defined in Eq.~(\ref{eq:output-hidden-prob}).
We approximate $P\left(v_{i},\hb\mid\vb_{\neg C}\right)$ by a fully
factorised distribution
\[
Q\left(v_{i},\hb\mid\vb_{\neg C}\right)=Q\left(\v_{i}\mid\vb_{\neg C}\right)\prod_{k}Q\left(\h_{k}\mid\vb_{\neg C}\right).
\]
The approximate distribution $Q\left(v_{i},\hb\mid\vb_{\neg C}\right)$
is obtained by minimising the Kullback-Leibler divergence
\[
\mathcal{D}_{KL}\left(Q\left(v_{i},\hb\mid\vb_{\neg C}\right)\parallel P\left(v_{i},\hb\mid\vb_{\neg C}\right)\right)=\sum_{\v_{i}}\sum_{\hb}Q\left(v_{i},\hb\mid\vb_{\neg C}\right)\log\frac{Q\left(v_{i},\hb\mid\vb_{\neg C}\right)}{P\left(v_{i},\hb\mid\vb_{\neg C}\right)}
\]
with respect to $Q\left(\v_{i}\mid\vb_{\neg C}\right)$ and $\left\{ Q\left(\h_{k}\mid\vb_{\neg C}\right)\right\} _{k=1}^{K}$.
This results in the following recursive relations:

\begin{eqnarray*}
Q\left(\v_{i}\mid\vb_{\neg C}\right) & \propto & \exp\left\{ G_{i}(\v_{i})+\sum_{k}H_{ik}(\v_{i})Q\left(\h_{k}\mid\vb_{\neg C}\right)\right\} ,\\
Q\left(\h_{k}\mid\vb_{\neg C}\right) & = & \frac{1}{1+\exp\{-w_{k}-\sum_{\v_{i}}H_{ik}(\v_{i})Q\left(\v_{i}\mid\vb_{\neg C}\right)-\sum_{j\in\neg C}H_{ik}(\v_{j})\}}.
\end{eqnarray*}

Now we make a further assumption that $\left|\sum_{\v_{i}}H_{ik}(\v_{i})Q\left(\v_{i}\mid\vb_{\neg C}\right)\right|\ll\left|\sum_{j\in\neg C}H_{ik}(\v_{j})\right|$,
e.g., when the set $\neg C$ is sufficiently large. This results in
$Q\left(\h_{k}\mid\vb_{\neg C}\right)\approx P\left(\h_{k}\mid\vb_{\neg C}\right)$
and
\[
Q\left(\v_{i}\mid\vb_{\neg C}\right)\propto\exp\left\{ G_{i}(\v_{i})+\sum_{k}H_{ik}(\v_{i})P\left(\h_{k}^{1}\mid\vb_{\neg C}\right)\right\} ,
\]
which is essentially the data model $P\left(\v_{i}\mid\hb\right)$
in Eq.~(\ref{eq:data-model}) with $h_{k}$ being replaced by $P\left(\h_{k}^{1}\mid\vb_{\neg C}\right)$.

The overall complexity of computing $Q\left(\v_{i}\mid\vb_{\neg C}\right)$
is the same as that of evaluating $P\left(\v_{i}\mid\vb_{\neg C}\right)$
in Eq.~(\ref{eq:predictive-distribution}). However, the approximation
is often numerically faster, and in the case of continuous variables,
it has the simpler functional form.

\subsection{Setting}

In this experiment, we run the $\MODEL$ on a large-scale survey of
the general world opinion, which was published by the Pew Global Attitudes
Project%
\footnote{http://pewglobal.org/datasets/%
} in the summer of $2002$. The survey was based on interviewing with
people in $44$ countries in the period of $2001$--$2002$. Some
sample questions are listed in Appendix~\ref{sec:Sample-Questions}.
After some pre-processing, we obtain a dataset of $38,263$ people,
each of whom provides answers to a subset of $189$ questions over
multiple topics ranging from globalization, democracy to terrorism.
Many answers are deliberately left empty because it may be inappropriate
to ask certain type of questions in a certain area or ethnic group.
Of all answers, $43$ are binary, $12$ are categorical, $3$ are
multicategorical, $125$ are ordinal, $2$ are category-ranking, and
$3$ are continuous. To suppress the scale difference in continuous
responses, we normalise them to zeros means and
unit variances~\footnote{It may be desirable to learn the variance structure, but we keep it simple
by fixing to unit variance. For more sophisticated variance learning, we refer to a recent paper \cite{le2011learning}
for more details.}.

We evaluate each data type separately. In particular, let $u$ be
the user index, $\hat{\v}{}_{i}$ be the predicted value of the $i$-th
variable, and $N_{t}$ is the number of variables of type $t$ in
the test data, we compute the prediction errors as follows:

\begin{tabular}{ll}
\noalign{\vskip0.2cm}
--Binary & :~~$\frac{1}{N_{bin}}\sum_{u}\sum_{i}\mathbb{I}\left[\v_{i}^{(u)}\ne\hat{\v}_{i}^{(u)}\right]$,\tabularnewline[0.2cm]
\noalign{\vskip0.2cm}
--Categorical & :~~$\frac{1}{N_{cat}}\sum_{u}\sum_{i}\mathbb{I}\left[\v_{i}^{(u)}\ne\hat{\v}_{i}^{(u)}\right]$,\tabularnewline[0.2cm]
\noalign{\vskip0.2cm}
--Multicategorical & :~~$1-\nicefrac{2\mathrm{R}\mathrm{P}}{\left(\mathrm{R}+\mathrm{P}\right)}$,\tabularnewline[0.2cm]
\noalign{\vskip0.2cm}
--Continuous & :~~$\sqrt{\frac{1}{D_{cont}}\sum_{u}\sum_{i}\left(\v_{i}^{(u)}-\hat{\v}_{i}^{(u)}\right)^{2}}$,\tabularnewline[0.2cm]
\noalign{\vskip0.2cm}
--Ordinal & :~~$\frac{1}{N_{ord}}\sum_{u}\sum_{i}\frac{1}{M_{i}-1}\left|\v_{i}^{(u)}-\hat{\v}_{i}^{(u)}\right|$,\tabularnewline[0.2cm]
\noalign{\vskip0.2cm}
--Category-ranking & :~~$\frac{1}{D_{rank}}\sum_{u}\sum_{i}\frac{2}{M_{i}(M_{i}-1)}\sum_{l,m>l}\mathbb{I}\left[(\pi_{il}^{(u)}-\pi_{im}^{(u)})(\hat{\pi}_{il}^{(u)}-\hat{\pi}_{im}^{(u)})<0\right]$,\tabularnewline[0.2cm]
\end{tabular}\\
where $\mathbb{I}\left[\cdot\right]$ is the identity function, $\pi_{im}\in\{1,2,...,M_{i}\}$
is the rank of the $m$-th category of the $i$-th variable, $\mathrm{R}$
is the recall rate and $\mathrm{P}$ is the precision. The recall
and precision are defined as:
\[
\mathrm{R}=\frac{\sum_{u}\sum_{i}\frac{1}{M_{i}}\sum_{m=1}^{M_{i}}\mathbb{I}\left[a_{im}^{(u)}=\hat{a}_{im}^{(u)}\right]}{\sum_{u}\sum_{i}\frac{1}{M_{i}}\sum_{m=1}^{M_{i}}a_{im}^{(u)}};\quad\quad\quad\mathrm{P}=\frac{\sum_{u}\sum_{i}\frac{1}{M_{i}}\sum_{m=1}^{M_{i}}\mathbb{I}\left[a_{im}^{(u)}=\hat{a}_{im}^{(u)}\right]}{\sum_{u}\sum_{i}\frac{1}{M_{i}}\sum_{m=1}^{M_{i}}\hat{a}_{im}^{(u)}},
\]
where $a_{im}\in\{0,1\}$ is the $m$-th component of the $i$-th
multicategorical variable. Note that the summation over $i$ for each
type only consists of relevant variables.

To create baselines, we use the $\MODEL$ without the hidden layer,
i.e., by assuming that variables are independent\footnote{To the best of our
knowledge, there has been no totally comparable work addressing the issues we study
in this paper. Existing survey analysis methods are suitable
for individual tasks such as measuring
pairwise correlation among variables, or building individual
regression models where complex co-variates are coded into binary variables.}.

\subsection{Results}

\subsubsection{Feature Extraction and Visualisation}

\begin{table*}
\begin{centering}
\begin{tabular}{|l|c|ccccc|}
\hline
 & Baseline  & $K=20$ & $K=50$ & $K=100$ & $K=200$ & $K=500$\tabularnewline
\hline
Binary  & 32.9 & 23.6 & 20.1 & 16.3 & 13.2 & 9.8\tabularnewline
\hline
Categorical  & 52.3 & 29.8 & 22.0 & 17.0 & 13.2 & 7.1\tabularnewline
\hline
Multicategorical  & 49.6 & 46.6 & 42.2 & 36.9 & 29.2 & 23.8\tabularnewline
\hline
Continuous({*})  & 100.0 & 89.3 & 84.1 & 78.4 & 69.5 & 65.5\tabularnewline
\hline
Ordinal  & 25.2 & 19.5 & 16.2 & 13.5 & 10.9 & 7.7\tabularnewline
\hline
Category ranking  & 19.3 & 11.7 & 6.0 & 5.0 & 3.2 & 2.3\tabularnewline
\hline
\end{tabular}
\par\end{centering}

\caption{Error rates (\%) when reconstructing data from posteriors. The baseline
is essentially the $\MODEL$ without hidden layer (i.e., assuming
variables are independent). ({*}) The continuous variables have been
normalised to account for different scales between items, thus the
baseline error will be $1$ (i.e., the unit variance).\label{tab:Reconstruction-error-rates}}
\end{table*}

Recall that our $\MODEL$ can be used as a feature extraction tool
through the posterior projection. The projection converts a multimodal
input into a real-valued vector of the form $\hat{\hb}=\left(\hat{\h}_{1},\hat{\h}_{2},...,\hat{\h}_{K}\right)$,
where $\hat{\h}_{k}=P\left(\h_{k}=1\mid\vb\right)$. Clearly, numerical
vectors are much easier to process further than the original data,
and in fact the vectorial form is required for the majority of modern data handling
tools (e.g., for transformation, clustering, comparison and visualisation).
To evaluate the faithfulness of the new representation, we reconstruct
the original data using $\hat{\v}_{i}=\arg\max_{\v_{i}}P\left(\v_{i}\mid\hat{\hb}\right)$,
that is, in Eq.~(\ref{eq:data-model}), the binary vector $\hb$
is replaced by $\hat{\hb}$. The use of $P\left(\v_{i}\mid\hat{\hb}\right)$
can be reasoned through the mean-field approximation framework presented
in Appendix~\ref{sub:Mean-field-Approximation}. Table~\ref{tab:Reconstruction-error-rates}
presents the reconstruction results. The trends are not surprising:
with more hidden units, the model becomes more flexible and accurate
in capturing the data content.

\begin{figure}
\begin{centering}
\includegraphics[height=0.5\linewidth]{Pew_02_44_nations_country_projection.pdf}
\par\end{centering}

\caption{t-SNE projection of posteriors ($K=50$) with country information
removed. Each point is a person from one of the $10$ countries: Angola,
Argentina, Bangladesh, Bolivia, Brazil, Bulgaria, Canada, China, Czech
Republic, and Egypt. Each colour represents a country. Best viewed
in colour. \label{fig:t-SNE-projection}}
\end{figure}

For visualisation, we first learn our $\MODEL$ (with $K=50$ hidden
units) using randomly chosen $3,830$ users, with the country information
removed. Then we use the t-SNE \cite{van2008visualizing} to project
the posteriors further into 2D. Figure~\ref{fig:t-SNE-projection}
shows the distribution of people's opinions in $10$ countries (Angola,
Argentina, Bangladesh, Bolivia, Brazil, Bulgaria, Canada, China, Czech
Republic, and Egypt). It is interesting to see how opinions cluster
geographically and culturally: Europe \& North America (Bulgaria,
Canada \& Czech Republic), South America (Argentina, Bolivia, Brazil),
East Asia (China), South Asia (Bangladesh), North Africa (Egypt) and
South Africa (Angola).

\subsubsection{Data Completion}

In this task, we need to fill missing answers for each survey response.
Missing answers are common in real survey data because the respondents
may forget to answer or simply ignore the questions. We create an
evaluation test by randomly removing a portion $\rho\in(0,1)$ of
answers for each person. The $\MODEL$ is then trained on the remaining
answers in a generative fashion (Section~\ref{sub:Estimating-Data-Distribution}).
Missing answers are then predicted as in Section~\ref{sub:Prediction}.
The idea here is that missing answers of a person can be interpolated
from available answers by other persons. This is essentially a multimodal
generalisation of the so-called collaborative filtering problem. Table~\ref{tab:Completion-errors-full}
reports the completion results for a subset of the data.

\begin{table*}
\begin{centering}
\begin{tabular}{|l|c|ccccc|}
\hline
 & Baseline  & $K=20$ & $K=50$ & $K=100$ & $K=200$ & $K=500$\tabularnewline
\hline
Binary  & 32.7 & 26.0 & 24.2 & 23.3 & 22.7 & 22.3\tabularnewline
\hline
Categorical  & 52.1 & 34.3 & 30.0 & 28.2 & 27.5 & 27.1\tabularnewline
\hline
Multicategorical  & 49.5 & 48.3 & 45.7 & 43.6 & 42.4 & 42.0\tabularnewline
\hline
Continuous({*})  & 101.6 & 93.5 & 89.9 & 87.9 & 87.3 & 87.9\tabularnewline
\hline
Ordinal  & 25.1 & 20.7 & 19.3 & 18.6 & 18.2 & 17.9\tabularnewline
\hline
Category ranking  & 19.3 & 15.4 & 14.7 & 14.2 & 14.1 & 13.9\tabularnewline
\hline
\end{tabular}
\par\end{centering}

\caption{Completion error rates (\%) $\rho=0.2$ answers missing at random.
({*}) See Table~\ref{tab:Reconstruction-error-rates}.\label{tab:Completion-errors-full} }
\end{table*}

\subsubsection{Learning Predictive Models}

We study six predictive problems, each of which is representative
for a data type. This means six corresponding variables are reserved
as outputs and the rest as input co-variates. The predictive problems
are: (i) satisfaction with the country (\emph{binary}), (ii) country
of origin (\emph{categorical}, of size $44$), (iii) problems facing
the country (\emph{multicategorical}, of size $11$), (iv) age of
the person (\emph{continuou}s), (v) ladder of life (\emph{ordinal},
of size $11$), and (vi) rank of dangers of the world (\emph{category-ranking},
of size $5$). All models are trained discriminatively (see Section~\ref{sub:Learning-Predictive-Models}).
We randomly split the users into a training subset and a testing subset.
The predictive results are presented in Table~\ref{tab:Predictive-errors-full}.
It can be seen that learning predictive models requires far less number
of hidden units than the tasks of reconstruction and completion. This
is because in discriminative training, the hidden layer acts as an
information filter that allows relevant amount of bits passing from
the input to the output. Since there is only one output per prediction
task, the number of required bits, therefore number of hidden units,
is relatively small. In reconstruction and completion, on the other
hand, we need many bits to represent all the available information.

\begin{table*}
\begin{centering}
\begin{tabular}{|l|c|cccccc|}
\hline
 & Baseline & $K=3$ & $K=5$ & $K=10$ & $K=15$ & $K=20$ & $K=50$\tabularnewline
\hline
Satisfaction (\emph{bin}.)  & 26.3 & 18.0 & 17.7 & 17.7 & 17.8 & 18.0 & 18.0\tabularnewline
\hline
Country (\emph{cat}.) & 92.0 & 70.2 & 61.0 & 21.6 & 11.0 & 9.9 & 5.9\tabularnewline
\hline
Probs. (\emph{multicat}.)  & 49.6 & 47.6 & 41.9 & 39.2 & 38.8 & 39.1 & 39.2\tabularnewline
\hline
Age (\emph{cont}.{*})  & 99.8 & 67.3 & 67.6 & 66.3 & 66.4 & 65.8 & 66.3\tabularnewline
\hline
Life ladder (\emph{ord}.) & 16.9 & 12.2 & 12.2 & 11.9 & 11.9 & 12.2 & 11.8\tabularnewline
\hline
Dangers (\emph{cat.-rank})  & 31.2 & 27.1 & 24.6 & 24.0 & 23.2 & 23.0 & 22.5\tabularnewline
\hline
\end{tabular}
\par\end{centering}

\caption{Predictive error rates (\%) with $80/20$ train/test split. ({*})
See Table~\ref{tab:Reconstruction-error-rates}. \label{tab:Predictive-errors-full}}
\end{table*}

\subsection{Parameter Learning \label{sub:Learning}}

We now present parameter estimation for $\{w_{k},U_{i},U_{im},V_{ik},V_{imk}\}$,
which clearly depend on the specific applications.

\subsubsection{Estimating Data Distribution \label{sub:Estimating-Data-Distribution}}

The problem of estimating a distribution from data is typically performed
by maximising the data likelihood $\mathcal{L}_{1}=\sum_{\vb}\tilde{P}(\vb)\log P(\vb)$,
where $\tilde{P}(\vb)$ denotes the empirical distribution of the
visible variables, and $P(\vb)=\sum_{\hb}P(\vb,\hb)$ is the model
distribution. Since the $\MODEL$ belongs to the exponential family,
the gradient of $\mathcal{L}_{1}$ with respect to parameters takes
the form of difference of expectations. For example, in the case of
binary variables, the gradient reads
\[
\frac{\partial\mathcal{L}_{1}}{\partial V_{ik}}=\left\langle v_{i}h_{k}\right\rangle _{\tilde{P}(v_{i},h_{k})}-\left\langle v_{i}h_{k}\right\rangle _{P(v_{i},h_{k})}
\]
where $\tilde{P}(h_{k},v_{i})=P(h_{k}|\vb)\tilde{P}(v_{i})$ is the
empirical distribution, and $P(h_{k},v_{i})=P(h_{k}|\vb)P(v_{i})$
the model distribution. Due to space constraint, we omit the derivation
details here.

The empirical expectation $\left\langle v_{i}h_{k}\right\rangle _{\tilde{P}(v_{i},h_{k})}$
is easy to estimate due to the factorisation in Eq.~(\ref{eq:posterior}).
However, the model expectation $\left\langle v_{i}h_{k}\right\rangle _{P(v_{i},h_{k})}$
is intractable to evaluate exactly, and thus we must resort to approximate
methods. Due to the factorisations in Eqs.~(\ref{eq:posterior},\ref{eq:data-model}),
Markov Chain Monte Carlo samplers are efficient to run. More specifically,
the sampler is alternating between $\left\{ \widehat{h}_{k}\sim P(h_{k}|\vb)\right\} _{k=1}^{K}$
and $\left\{ \widehat{\v}_{i}\sim P(\v_{i}|\hb)\right\} _{i=1}^{N}$.
Note that in the case of multicategorical variables, make use of the
factorisation in Eq.~(\ref{eq:multilabel-data-model}) and sample
$\{a_{im}\}_{m=1}^{M_{i}}$ simultaneously. On the other hand, in
the case of category-ranked variables, we do not sample directly from
$P(\v_{i}|\hb)$ but from its relaxation $\left\{ P_{i}(c_{il}\triangleright c_{im}|\hb)\right\} _{l,m>l}$
- which have the form of multinomial distributions. To speed up, we
follow the method of Contrastive Divergence (CD) \cite{Hinton02},
in which the MCMC is restarted from the observed data $\vb$ and stopped
after just a few steps for every parameter update. This has been known
to introduce bias to the model estimate, but it is often fast and
effective for many applications.

For the data completion application, in the data we observed only
some variables and others are missing. There are two ways to handle
a missing variable during training time: one is to treat it as hidden,
and the other is to ignore it. In this paper, we follows the latter
for simplicity and efficiency, especially when the data is highly
sparse\footnote{Ignoring missing data may be inadequate if
the missing patterns are not at random. However, treating missing data as
zero observations (e.g., in the case of binary variables) may not be accurate
either since it may introduce bias to the data marginals.}.

\subsubsection{Learning Predictive Models \label{sub:Learning-Predictive-Models}}

In our $\MODEL$, a predictive task can be represented by an output
variable conditioned on input variables. Denote by $v_{i}$ the $i$-th
output variable, and $\vb_{\neg i}$ the set of input variables, that
is, $\vb=(\v_{i},\vb_{\neg i})$. The learning problem is translated
into estimating the conditional distribution $P(\v_{i}\mid\vb_{\neg i})$.

There are three general ways to learn a predictive model. The \emph{generative}
method first learns the joint distribution $P(\v_{i},\vb_{\neg i})$
as in the problem of estimating data distribution. The \emph{discriminative}
method, on the other hand, effectively ignores $P(\vb_{\neg i})$
and concentrates only on $P(\v_{i}\mid\vb_{\neg i})$. In the latter,
we typically maximise the conditional likelihood $\mathcal{L}_{2}=\sum_{\v_{i}}\sum_{\vb_{\neg i}}\tilde{P}(\v_{i},\vb_{\neg i})\log P(\vb_{i}\mid\vb_{\neg i})$.
This problem is inherently easier than the former because we do not
have to make inference about $\vb_{\neg i}$. The learning strategy
is almost identical to that of the generative counterpart, except
that we \emph{clamp} the input variables $\vb_{\neg i}$ to their
observed values. For tasks whose size of the output space is small
(e.g., standard binary, ordinal, categorical variables) we can perform
exact evaluations and use any non-linear optimisation methods for
parameter estimation. The conditional distribution $P(\v_{i}\mid\vb_{\neg i})$
can be computed as in Eq.~(\ref{eq:predictive-distribution}). We
omit the likelihood gradient here for space limitation.

It is often argued that the discriminative method is more preferable
since there is no waste of effort in learning $P(\vb_{\neg i})$,
which we do not need at test time. In our setting, however, learning
$P(\vb_{\neg i})$ may yield a more faithful representation%
\footnote{As we do not need labels to learn $P(\vb_{\neg i})$, this is actually
a form of \emph{semi-supervised learning}.%
} of the data through the posterior $P(\hb\mid\vb_{\neg i})$. This
suggests a third, \emph{hybrid} method: combining the generative and
discriminative objectives. One way is to optimise a hybrid likelihood:

\[
\mathcal{L}_{3}=\lambda\sum_{\vb_{\neg i}}\tilde{P}(\vb_{\neg i})\log P(\vb_{\neg i})+(1-\lambda)\sum_{\vb_{i}}\sum_{\vb_{\neg i}}\tilde{P}(\vb_{i},\vb_{\neg i})\log P(\vb_{i}\mid\vb_{\neg i}),
\]
where $\lambda\in(0,1)$ is the hyper-parameter controlling the relative
contribution of generative and discriminative components. Another
way is to use a \textbf{$2$}-stage procedure: first we \emph{pre-train}
the model $P(\vb_{\neg i})$ in an unsupervised manner, and then \emph{fine-tune}
the predictive model%
\footnote{We can also avoid tuning parameters associated with $\vb_{\neg i}$
by using the posteriors as features and learn $P\left(\v_{i}\mid\hat{\hb}\right)$,
where $\hat{\h}_{k}=P\left(h_{k}^{1}\mid\vb_{\neg i}\right).$%
} $P(\vb_{i}\mid\vb_{\neg i})$.

\subsection{Prediction \label{sub:Prediction}}

Once the model has been learnt, we are ready to perform prediction.
We study two predictive applications: completing missing data, and
output labels in predictive modelling. The former leads to the inference
of $P(\vb_{C}\mid\vb_{\neg C})$, where $\vb_{\neg C}$ is the set
of observed variables, and $\vb_{C}$ is the set of unseen variables
to be predicted. Ideally, we should predict all unseen variables simultaneously
but the inference is likely to be difficult. Thus, we resort to estimating
$P(v_{i}|\vb_{\neg C})$, for $i\in C$. The prediction application
requires the estimation of $P(v_{i}|\vb_{\neg i})$, which is clearly
a special case of $P(v_{i}|\vb_{\neg C})$, i.e., when $C=\{i\}$.
The output is predicted as follows
\begin{align}
\hat{\v}_{i} & =\arg\max_{\v_{i}}P(v_{i}|\vb_{\neg C})=\arg\max_{\v_{i}}\sum_{\hb}P(v_{i},\hb|\vb_{\neg C});\quad\mbox{where}\label{eq:prediction}\\
P(v_{i},\hb|\vb_{\neg C}) & =\frac{1}{Z(\vb_{\neg C},\hb)}\exp\left\{ G_{i}(v_{i})+\sum_{k}w_{k}h_{k}+\sum_{j\in\{\neg C,i\}}\sum_{k}H_{jk}(v_{j})h_{k}\right\} ,\label{eq:output-hidden-prob}
\end{align}
where $Z(\vb_{\neg C})$ is the normalising constant. Noting that
$\h_{k}\in\{0,1\}$, the computation of $P(v_{i}|\vb_{\neg C})$ can
be simplified as
\begin{align}
P(v_{i}|\vb_{\neg C}) & =\frac{1}{Z(\vb_{\neg C})}\exp\{G_{i}(v_{i})\}\prod_{k}\left[1+\frac{\exp\{H_{ik}(v_{i})\}}{1/P(h_{k}^{1}|\vb_{\neg C})-1}\right]\label{eq:predictive-distribution}
\end{align}
where $P(h_{k}^{1}|\vb_{\neg C})$ is computed using Eq.~(\ref{eq:posterior})
as
\[
P(h_{k}^{1}|\vb_{\neg C})=\frac{1}{1+\exp\{-w_{k}-\sum_{j\in\neg C}H_{jk}(v_{j})\}}.
\]

For the cases of binary, categorical and ordinal outputs, the estimation
in Eq.~(\ref{eq:prediction}) is straightforward using Eq.~(\ref{eq:predictive-distribution}).
However, for other output types, suitable simplification must be made:
\begin{itemize}
\item For multicategorical and category-ranking variables, we do not enumerate
over all possible assignments of $v_{i}$, but rather in an indirect
manner:

\begin{itemize}
\item For multiple categories (Section~\ref{sub:Multi-categorical-Variables}),
we first estimate $\left\{ P_{i}(a_{im}=1|\vb_{\neg i})\right\} _{m=1}^{M_{i}}$
and then output $a_{im}=1$ if $P_{i}(a_{im}=1|\vb_{\neg i})\ge\nu$
for some threshold%
\footnote{Raising the threshold typically leads to better precision at the expense
of recall. Typically we choose $\nu=0.5$ when there is no preference
over recall nor precision.%
} $\nu\in(0,1)$.
\item For category-ranking (Section~\ref{sub:Category-ranking-Variables}),
we first estimate $\left\{ P_{i}(c_{il}\succ c_{im}|\vb_{\neg i})\right\} _{l,m>l}$.
The complete ranking over the set $\{c_{i1},c_{i2},...,c_{iM_{i}}\}$
can be obtained by aggregating over probability pairwise relations.
For example, the score for $c_{im}$ can be estimated as $s(c_{im})=\sum_{l\ne m}P_{i}(c_{im}\succ c_{il}|\vb_{\neg i})$,
which can be used for sorting categories%
\footnote{Note that we do not estimate the event of ties during prediction.%
}.
\end{itemize}
\item For continuous variables, the problem leads to a non-trivial nonlinear
optimisation: even for the case of Gaussian variables, $P(v_{i}|\vb_{\neg C})$
in Eq.~(\ref{eq:predictive-distribution}) is no longer Gaussian.
For efficiency and simplicity, we can take a \emph{mean-field} approximation
by substituting $\hat{\h}_{k}=P(h_{k}^{1}|\vb_{\neg C})$ for $\h_{k}$.
For example, in the case of Gaussian outputs, we then obtain a simplified
expression for $P(v_{i}|\vb_{\neg C})$:
\[
P(v_{i}|\vb_{\neg C})\propto\exp\left\{ -\frac{\v_{i}^{2}}{2\sigma_{i}^{2}}+U_{i}\v_{i}+\sum_{k}V_{ik}v_{i}\hat{\h}_{k}\right\} ,
\]
which is also a Gaussian. Thus the optimal value is the mean itself:
$\hat{\v}_{i}=\sigma_{i}^{2}\left(U_{i}+\sum_{k}V_{ik}\hat{\h}_{k}\right).$
Details of the mean-field approximation is presented in Appendix~\ref{sub:Mean-field-Approximation}.\end{itemize}

\subsection{Model Definition}

Denote by $\vb=(\v_{1},\v_{2},...,\v_{N})$ the set of \emph{mixed-variate}
visible variables where each $\v_{i}$ can be one of the following
types: \emph{binary}, \emph{categorical}, \emph{multicategorical},
\emph{continuous}, \emph{ordinal} or \emph{category-ranked}. Let $\vb_{disc}$
be the joint set of discrete elements and $\vb_{cont}$ be the continuous
set, and thus $\vb=(\vb_{disc},\vb_{cont})$. Denoting by $\hb=(h_{1},h_{2},...,h_{K})\in\{0,1\}^{K}$
the hidden variables, the model distribution of $\MODEL$ is defined
as
\begin{equation}
P(\vb,\hb)=\frac{1}{Z}\exp\{-E(\vb,\hb)\},\label{eq:model-def}
\end{equation}
where $E(\vb,\hb)$ is the model energy, $Z$ is the normalisation
constant. The model energy is further decomposed into a sum of singleton
and pairwise energies:
\[
E(\vb,\hb)=\sum_{i=1}^{N}E_{i}(\v_{i})+\sum_{k=1}^{K}E_{k}(\h_{k})+\sum_{i=1}^{N}\sum_{k=1}^{K}E_{ik}(\v_{i},\h_{k}),
\]
where $E_{i}(\v_{i})$ depends only on the $i$-th visible unit, $E_{k}(\h_{k})$
on the $k$-th hidden unit, and $E_{ik}(\v_{i},\h_{k})$ on the interaction
between the $i$-th visible and $k$-hidden units. The $\MODEL$ is
thus a $2$-layer mixed-variate Markov random field with pairwise
connectivity across layers.

For the distribution in Eq.~(\ref{eq:model-def}) to be properly
specified, we need to keep the normalisation constant finite. In other
words, the following integration

\[
Z=\int_{\vb_{cont}}\left(\sum_{\vb_{disc}}\sum_{\hb}\exp\{-E(\vb_{disc},\vb_{cont},\hb)\}\right)d(\vb_{cont})
\]
must be bounded from above. One way is to choose appropriate continuous
variable types with bounded moments, e.g., Gaussian. Another way is
to explicitly bound the continuous variables to some finite ball,
i.e., $\left\Vert \vb_{cont}\right\Vert \le R$.

In our $\MODEL$, we further assume that the energies have the following
form:
\begin{equation}
E_{i}(\v_{i})=-G_{i}(\v_{i});\quad\quad\quad\quad E_{k}(h_{k})=-w_{k}h_{k};\quad\quad\quad\quad E_{ik}(\v_{i},\h_{k})=-H_{ik}(v_{i})h_{k},\label{eq:energies}
\end{equation}
where $w_{k}$ is the bias parameter for the $k$-th hidden unit,
and $G_{i}(v_{i})$ and $H_{ik}(v_{i})$ are functions to be specified
for each data type. An important consequence of this energy decomposition
is the factorisation of the \emph{posterior}:

\begin{eqnarray}
P(\hb\mid\vb) & = & \prod_{k}P(h_{k}\mid\vb);\quad\quad\quad\quad P(h_{k}^{1}\mid\vb)=\frac{1}{1+\exp\{-w_{k}-\sum_{i}H_{ik}(\v_{i})\}},\label{eq:posterior}
\end{eqnarray}
where $h_{k}^{1}$ denotes the assignment $h_{k}=1$. This posterior
is efficient to evaluate, and thus the vector $\left(P(h_{k}^{1}\mid\vb),P(h_{k}^{2}\mid\vb),...,P(h_{k}^{K}\mid\vb)\right)$
can be used as extracted features for mixed-variate input $\vb$.

Similarly, the \emph{data model} $P(\vb|\hb)$ has the following factorisation

\begin{equation}
P(\vb\mid\hb)=\prod_{i}P_{i}(\v_{i}\mid\hb);\quad\quad\quad\quad P_{i}(\v_{i}\mid\hb)=\frac{1}{Z(\hb)}\exp\{G_{i}(\v_{i})+\sum_{k}H_{ik}(\v_{i})h_{k}\},\label{eq:data-model}
\end{equation}
where $Z(\hb)=\sum_{\v_{i}}\exp\{G_{i}(\v_{i})+\sum_{k}H_{ik}(\v_{i})h_{k}\}$
if $\v_{i}$ is discrete and $Z(\hb)=\int_{\v_{i}}\exp\{G_{i}(\v_{i})+\sum_{k}H_{ik}(\v_{i})h_{k}\}d(\v_{i})$
if $\v_{i}$ is continuous, assuming that the integration exists.
Note that we deliberately use the subscript index $i$ in $P_{i}(\cdot\mid\hb)$
to emphasize the heterogeneous nature of the input variables.

\subsection{Type-specific Data Models}

We now specify $P_{i}(\v_{i}|\mathbf{h})$ in Eq.~(\ref{eq:data-model}),
or equivalently, the functionals $G_{i}(\v_{i})$ and $H_{ik}(\v_{i})$.
Denote by $\mathbb{S}_{i}=(c_{i1},c_{i2},...,c_{iM_{i}})$ the set
of categories in the case of discrete variables. In this section,
for continuous types, we limit to Gaussian variables as they are the
by far the most common. Interested readers are referred to \cite{le2011learning} for Beta variables
in the context of image modelling. The data model and related functionals for
binary, Gaussian and categorical data types are well-known, and thus we provide
a summary here:

\begin{flushleft}
\begin{tabular}{lccc}
 & $G_{i}(\v_{i})$ & $H_{ik}(\v_{i})$ & $P_{i}(\v_{i}|\hb)$\tabularnewline
\hline
\noalign{\vskip0.2cm}
--Binary & $U_{i}v_{i}$ & $V_{ik}v_{i}$ & $\frac{\exp\{U_{i}v_{i}+\sum_{k}V_{ik}h_{k}v_{i}\}}{1+\exp\{U_{i}+\sum_{k}V_{ik}h_{k}\}}$\tabularnewline
\noalign{\vskip0.2cm}
--Gaussian & $-\nicefrac{\v_{i}^{2}}{2\sigma_{i}^{2}}+U_{i}v_{i}$ & $V_{ik}v_{i}$ & $\mathcal{N}\left(\sigma_{i}^{2}\left(U_{i}+\sum_{k}V_{ik}h_k\right);\sigma_{i}\right)$\tabularnewline
\noalign{\vskip0.2cm}
--Categorical & $\sum_{m}U_{im}\delta_{m}[v_{i}]$ & $\sum_{m,k}V_{imk}\delta_{m}[v_{i}]$ & $\frac{\exp\{\sum_{m}U_{im}\delta_{m}[v_{i}]+\sum_{m,k}V_{imk}\delta_{m}[v_{i}]h_k\}}{\sum_{l}\exp\{U_{il}+\sum_{k}V_{ilk}h_{k}\}}$\tabularnewline[0.4cm]
\end{tabular}\\
where $m=1,2,...,M_{i}$; $U_{i},V_{ik},U_{im},V_{imk}$ are model
parameters; and $\delta_{m}[v_{i}]=1$ if $v_{i}=c_{im}$ and $0$
otherwise.
\par\end{flushleft}

The cases of multicategorical, ordinal and category-ranking variables
are, however, much more involved, and thus some further simplification
may be necessary. In what follows, we describe the specification details
for these three cases.

\subsubsection{Multicategorical Variables \label{sub:Multi-categorical-Variables}}

An assignment to a multicategorical variable has the form of a subset
from a set of categories. For example, a person may be interested
in \textsf{games} and \textsf{music} from a set of offers: \textsf{games},
\textsf{sports}, \textsf{music}, and \textsf{photography}. More formally,
let $\mathbb{S}_{i}$ be the set of categories for the $i$-th variable,
and $\mathcal{P}_{i}=2^{\mathbb{S}_{i}}$ be the power set of $\mathbb{S}_{i}$
(the set of all possible subsets of $\mathbb{S}_{i}$). Each variable
assignment consists of a non-empty element of $\mathcal{P}_{i}$,
i.e. $\v_{i}\in\left\{ \mathcal{P}_{i}\backslash\emptyset\right\} $.
Since there are $2^{M_{i}}-1$ possible ways to select a non-empty
subset, directly enumerating $P_{i}(\v_{i}|\hb)$ proves to be highly
difficult even for moderate sets. To handle this state explosion,
we first assign each category $c_{im}$ with a binary indicator $a_{im}\in\{0,1\}$
to indicate whether the $m$-th category is active, that is $\v_{i}=\left(a_{i1},a_{i2},...,a_{iM_{i}}\right)$.
We then assume the following factorisation:
\begin{equation}
P_{i}(\v_{i}|\hb)=\prod_{m=1}^{M_{i}}P_{i}(a_{im}|\hb).\label{eq:multilabel-data-model}
\end{equation}
Note that this does not says that binary indicators are independent
in their own right but given the knowledge of the hidden variables
$\hb$. Since they hidden variables are never observed, binary indicators
are therefore interdependent. Now, the probability for activating
a binary indicator is defined as

\begin{equation}
P_{i}(a_{im}=1|\hb)=\frac{1}{1+\exp(-U_{im}-\sum_{k}V_{imk}\h_{k})}.\label{eq:multicat-model}
\end{equation}

Note that this specification is equivalent to the following decomposition
of the functionals $G_{i}(v_{i})$ and $H_{ik}(\v_{i})$ in Eq.~(\ref{eq:energies}):
\begin{align*}
G_{i}(v_{i}) & =\sum_{m=1}^{M_{i}}U_{im}a_{im};\quad\quad\quad\quad H_{ik}(\v_{i})=\sum_{m=1}^{M_{i}}V_{imk}a_{im}.
\end{align*}

\subsubsection{Ordinal Variables}

An ordinal variable receives individual values from an ordinal set
$S_{i}=\{c_{i1}\prec c_{i2}\prec...,\prec c_{iM_{i}}\}$ where $\prec$
denotes the order in some sense. For example, $c_{im}$ can be a numerical
rating from a review, or it can be sentimental expression such as
\textsf{love}, \textsf{neutral} and \textsf{hate}. There are two straightforward
ways to treat an ordinal variable: (i) one is simply ignoring the
order, and considering it as a multinomial variable, and (ii) another
way is to convert the ordinal expression into some numerical scale,
for example, $\{-1,0,+1\}$ for the triple \{\textsf{love}\textsf{\emph{,}}\textsf{neutral}\textsf{\emph{,}}\textsf{hate}\}
and then proceed as if it is a continuous variable. However, in the
first treatment, substantial ordinal information is lost, and in the
second treatment, there is no satisfying interpretation using numbers.

In this paper, we adapt the Stereotype Ordered Regression Model (SORM)
by \cite{anderson1984rao}. More specifically, the SORM defines the
conditional distribution as follows
\[
P(v_{i}=m\mid\hb)=\frac{\exp\{U_{im}+\sum_{d=1}^{D}\sum_{k=1}^{K}V_{idk}\phi_{id}(m)h_{k}\}}{\sum_{l}\exp\{U_{il}+\sum_{d=1}^{D}\sum_{k=1}^{K}V_{idk}\phi_{id}(l)h_{k}\}}
\]
where $U_{im},V_{idk}$ are free parameters, $D\le M_{i}$ is the
dimensionality of the ordinal variable%
\footnote{This should not be confused with the dimensionality of the whole data
$\vb$.%
} $\v_{i}$, and $\phi_{id}(m)$ is the monotonically increasing function
of $m$:
\[
\phi_{id}(1)<\phi_{id}(2)<...<\phi_{id}(M_{i})
\]

A shortcoming of this setting is that when $\hb=\boldsymbol{0}$,
the model reduces to the standard multiclass logistic, effectively
removing the ordinal property. To deal with this, we propose to make
the input bias parameters order dependent:
\begin{equation}
P(v_{i}=m\mid\hb)\propto\exp\left\{ \sum_{d=1}^{D}\phi_{id}(m)\left(U_{id}+\sum_{k=1}^{K}V_{idk}h_{k}\right)\right\} \label{eq:ordinal-model}
\end{equation}
where $U_{id}$ is the newly introduced parameter. Here we choose
$D=M_{i}$, and $\phi_{id}(m)=\nicefrac{\left(m-d\right)}{\left(M_{i}-1\right)}$.

\subsubsection{Category-ranking Variables \label{sub:Category-ranking-Variables}}

In category ranking, a variable assignment has the form of a ranked
list of a set of categories. For example, from a set of offers namely
\textsf{games}, \textsf{sports}, \textsf{music}, and \textsf{photography},
a person may express their preferences in a particular decreasing
order: \textsf{sports} $\succ$ \textsf{music} $\succ$ \textsf{games} $\succ$ \textsf{photography}.
Sometimes, they may like sports and music equally, creating a situation
known as \emph{ties} in ranking, or \emph{indifference} in preference.
When there are no ties, we can say that the rank is \emph{complete}.

More formally, from a set of categories $\mathbb{S}_{i}=\{c_{i1},c_{i2},...,c_{iM_{i}}\}$,
a variable assignment without ties is then a permutation of elements
of $\mathbb{S}_{i}$. Thus, there are $M_{i}!$ possible complete
rank assignments. When we allow ties to occur, however, the number
of possible assignments is extremely large. To see how, let us group
categories of the same rank into a partition. Orders within a partition
are not important, but orders between partitions are. Thus, the problem
of rank assignment turns out to be choosing from a set of all possible
schemes for partitioning and ordering a set. The number of such schemes
is known in combinatorics as the \emph{Fubini's number} \cite[pp. 396--397]{muresan2008concrete},
which is extremely large even for small sets. For example, $\mbox{Fubini}\left(1\right)=1$,
$\mbox{Fubini}\left(3\right)=13$, $\mbox{Fubini}\left(5\right)=541$
and $\mbox{Fubini}\left(10\right)=102,247,563$. Directly modelling
ranking with ties proves to be intractable.

We thus resort to approximate methods. One way is to model just pairwise
comparisons: we treat each pair of categories separately \emph{when
conditioned on the hidden layer}. More formally, denote by $c_{il}\succ c_{im}$
the preference of category $c_{il}$ over $c_{im}$, and by $c_{il}\simeq c_{im}$
the indifference. We replace the data model $P_{i}(\v_{i}|\hb)$ with
a product of pairwise comparisons $\prod_{l}\prod_{m>l}P_{i}(c_{il}\triangleright c_{im}|\hb)$,
where $\triangleright$ denotes preference relations (i.e., $\succ$,
$\prec$ or $\simeq$). This effectively translates the original problem
with Fubini's number complexity to $M_{i}(M_{i}-1)/2$ pairwise sub-problems,
each of which has only three preference choices. The drawback is that
this relaxation loses the guarantee of \emph{transitivity }(i.e.,
$c_{il}\succeq c_{im}$ and $c_{im}\succeq c_{in}$ would entail $c_{il}\succeq c_{in}$,
where $\succeq$ means \emph{better} or \emph{equal-to}). The hope
is that the hidden layer is rich enough to absorb this property, that
is, the probability of preserving the transitivity is sufficiently
high.

Now it remains to specify $P_{i}(c_{il}\triangleright c_{im}|\hb)$
in details. In particular, we adapt the Davidson's model \cite{davidson1970extending}
of pairwise comparison:
\begin{eqnarray}
P_{i}(c_{il}\succ c_{im}|\hb) & = & \frac{1}{Z_{iml}(\hb)}\varphi(c_{il},\hb)\nonumber \\
P_{i}(c_{il}\simeq c_{im}|\hb) & = & \frac{1}{Z_{iml}(\hb)}\gamma\sqrt{\varphi(c_{il},\hb)\varphi(c_{im},\hb)}\label{eq:label-rank-model}\\
P_{i}(c_{il}\prec c_{im}|\hb) & = & \frac{1}{Z_{iml}(\hb)}\varphi(c_{im},\hb)\nonumber
\end{eqnarray}
where $Z_{ilm}(\hb)=\varphi(c_{il},\hb)+\varphi(c_{im},\hb)+\gamma\sqrt{\varphi(c_{il},\hb)\varphi(c_{im},\hb)}$,
$\gamma>0$ is the tie parameter, and

\[
\varphi(c_{im},\hb)=\exp\{\frac{1}{M_{i}}(U_{im}+\sum_{k}V_{imk}h_{k})\}.
\]
The term $\nicefrac{1}{M_{i}}$ normalises the occurrence frequency
of a category in the model energy, leading to better numerical stability.

\section{Introduction}

\input{intro.tex}

\section{Mixed-Variate Restricted Boltzmann Machines \label{sec:Mixed-State-Restricted}}

\input{model.tex}

\section{Learning and Inference \label{sec:Learning-and-Inference}}

\input{learning.tex}

\section{A Case Study: World Attitudes \label{sub:Filling-Missing-Data}}

\input{experiments.tex}

\section{Related Work}

The most popular use of RBMs is in modelling of individual types,
for example, binary variables \cite{freund1994unsupervised}, Gaussian
variables \cite{hinton2006rdd,ranzato2010modeling}, categorical
variables \cite{Salakhutdinov-et-alICML07}, rectifier linear units
\cite{nair2010rectified}, Poisson variables \cite{gehler2006rap},
counts \cite{salakhutdinov2009replicated} and Beta variables \cite{le2011learning}. When RBMs are used
for classification \cite{larochelle2008classification}, categorical
variables might be employed for labeling in additional to the features.
Other than that, there has been a model called Dual-Wing RBM for modelling
both continuous and binary variables \cite{xing2005mining}. However,
there have been no attempts to address all \emph{six} data types in
a single model as we do in the present paper.

The literature on ordinal variables is sufficiently rich in
statistics, especially after the seminal work of \cite{mccullagh1980rmo}.
In machine learning, on the other hand, the literature is quite sparse
and recent (e.g. see \cite{shashua2002tlm,yu2006collaborative}) and it is often limitted to single
ordinal output (given numerical input co-variates). An RBM-based modelling of ordinal variables
addressed in \cite{Truyen:2009a} is similar to ours, except that
our treatment is more general and principled.

Mixed-variate modelling has been previously studied in statistics, under
a variety of names such as \emph{mixed outcomes}, \emph{mixed data},
or \emph{mixed responses} \cite{sammel1997latent,dunson2000bayesian,shi2000latent,mcculloch2008joint}.
Most papers focus on the mix of ordinal, Gaussian and binary variables under the
\emph{latent variable} framework. More specifically, each observed variable is
assumed to be generated from one or more underlying continuous latent variables.
Inference becomes complicated since we need to integrate out
these correlated latent variables, making it difficult to handle hundreds of variables
and large-scale datasets.

In machine learning, the problem of predicting a single multicategorical
variable is also known as multilabel learning (e.g., see \cite{tsoumakas2007multi}).
Previous ideas that we have adapted into our context including the
shared structure among labels \cite{ji2008extracting}. In our model,
the sharing is captured by the hidden layer in a probabilistic manner
and we consider many multicategorical variables at the same time.
Finally, the problem of predicting a single category-ranked variable
is also known as label-ranking (e.g., see \cite{dekel2003llm,hullermeier2008lrl}).
The idea we adopt is the pairwise comparison between categories. However,
the previous work neither considered the hidden correlation between
those pairs nor attempted multiple category-ranked variables.

\section{Conclusion}

We have introduced Mixed-Variate Restricted Boltzmann Machines ($\MODEL$)
as a generalisation of the RBMs for modelling correlated variables
of multiple modalities and types. Six types considered were: binary,
categorical, multicategorical, continuous information, ordinal, and
category-ranking. We shown that the $\MODEL$ is capable of handling
a variety of machine learning tasks including feature exaction, dimensionality
reduction, data completion, and label prediction. We demonstrated
the capacity of the model on a large-scale world-wide survey.

We plan to further the present work in several directions. First,
the model has the capacity to handle multiple related predictive models
simultaneously by learning a shared representation through hidden
posteriors, thereby applicable to the setting of multitask learning.
Second, there may exist strong interactions between variables which
the RBM architecture may not be able to capture. The theoretical question
is then how to model inter-type dependencies directly without going
through an intermediate hidden layer. Finally, we plan to enrich the
range of applications of the proposed model.

\paragraph{Acknowledgment:}
We thank anonymous reviewers for insightful comments.


\section{Abstract}

Quantum entanglement, one of the most counterintuitive effects in quantum mechanics \cite{Einstein35}, plays an essential role in quantum information and communication technology.
Thus far, many efforts have been taken to create multipartite entanglement in photon polarization \cite{Walther2005, Lu2007, Gao2010, Wieczorek2009, Prevedel2009}, quadrature amplitude \cite{Furusawa2009}, and ions \cite{Haffner2005}, for demonstration and precise operation of quantum protocols.
These efforts have mainly concentrated on the generation of pure entangled states, such as GHZ \cite{Greenberger1989}, W \cite{Dur2000}, and cluster \cite{Briegel2001} states.
By contrast, bound entanglement \cite{Horodecki1998} could not be distilled into pure entangled states, and had been considered useless for quantum information protocols such as quantum teleportation \cite{Boumeester1997, Furusawa1998}.
However, it is interesting that some bound entanglement can be distilled by certain procedures \cite{Smolin2001} or interaction with auxiliary systems \cite{Acin2004, Shor2003}.
These properties provide new quantum communication schemes, for instance, remote information concentration \cite{Murao2001}, secure quantum key distribution \cite{Horodecki2005}, super-activation \cite{Shor2003}, and convertibility of pure entangled states \cite{Ishizaka2004}.

Recently, a distillation protocol from the bound entangled state, so called `unlocking' \cite{Smolin2001}, has been experimentally demonstrated \cite{Amselem2009,Lavoie2010}.
In this protocol, as depicted in Fig. 1 (a), four-party bound entanglement in the Smolin state \cite{Smolin2001} can be distilled into two parties (e.g., A and D) when the other two parties (e.g., B and C) come together and make joint measurements on their qubits.
The unlocking protocol, in principle, can distill pure and maximal entanglement into the two qubits. However, it does not belong to the category of LOCC, since the two parties have to meet to distill the entanglement.

\begin{tiny}
\begin{figure}[t!]
   \includegraphics[width= \columnwidth, clip]{activation_unlock.eps}
   \label{ }
   \caption{Principle of the distillation of the bound entanglement in the Smolin state $\rho_s$. Each circle represents a qubit, and the blue line and squares show the entanglement of parties. BSM, Bell state measurement; CC, classical channel. (a) Unlocking. (b) Activation of the bound entanglement. Both of them can distill entanglement from the Smolin state; however, the activation protocol can be carried out under LOCC, while the unlocking needs two parties coming together and making joint measurements.     }
\end{figure}
\end{tiny}

The activation of bound entanglement that we demonstrate here is another protocol by which one can distill the Smolin-state bound entanglement by means of LOCC. The principle of the activation is sketched in Fig. 1 (b).
Consider four parties, A, B, C, and D each of which has a qubit in the Smolin state.
The Smolin state is a statistically equal mixture of pairs of the four Bell states, and its density matrix $\rho_s$ is given by

\begin{align}
\rho_s &= \sum_{i=1}^{4} |\phi^i\rangle \langle \phi^i |_{AB}\otimes |\phi^i\rangle \langle \phi^i |_{CD} \notag \\
       &= \sum_{i=1}^{4} |\phi^i\rangle \langle \phi^i |_{AC}\otimes |\phi^i\rangle \langle \phi^i |_{BD} \notag \\
       &= \sum_{i=1}^{4} |\phi^i\rangle \langle \phi^i |_{AD}\otimes |\phi^i\rangle \langle \phi^i |_{BC},
\end{align}
where $|\phi^i\rangle \in \{ |\phi^{\pm} \rangle ,|\psi^{\pm} \rangle \}$ are the two-qubit Bell states given by
\begin{align}
|\phi^{\pm} \rangle &= \frac{1}{\sqrt{2}} \left( |00 \rangle \pm |11 \rangle \right) \notag \\
|\psi^{\pm} \rangle &= \frac{1}{\sqrt{2}} \left( |01 \rangle \pm |10 \rangle \right),
\end{align}
where  $\ket{0}$ and $\ket{1}$ are the qubit bases.
Since the $\rho_s$ state is symmetric with respect to the exchange of any two parties, $\rho_s$ is separable in any two-two bipartite cuts.
This implies that there is no distillable entanglement in any two-two bipartite cuts:
$D_{AB|CD}(\rho_s)=D_{AC|BD}(\rho_s)=D_{AD|BC}(\rho_s)=0$, where $D_{i|j}(\rho)$ is the distillable entanglement of $\rho$ in an $i|j$ bipartite cut.
In addition to the Smolin state $\rho_s$, two of the parties (e.g., B and C) share distillable entanglement in the two-qubit Bell state (e.g.,  $\ket{\psi^+}_{B'C'}$).
Hence, the initial state $\rho_I$ is given by
\begin{align}
\rho_I &= \rho_s \otimes \ket{\psi^+}\bra{\psi^+}_{B'C'} \notag \\
       &= \sum_{i=1}^{4} \ket{\phi^i}\bra{\phi^i}_{AD}\otimes \ket{\phi^i}\bra{\phi^i} _{BC} \otimes \ket{\psi^+}\bra{\psi^+}_{B'C'}.
\end{align}
The state $\rho_s$ or $\ket{\psi^+}_{B'C'}$ gives no distillable entanglement into A and D: $D_{A|D}(\rho_s) = D_{A|D}(\ket{\psi^+}_{B'C'})=0$, since $D_{A|D}(\rho_s) \leq D_{AB|CD}(\rho_s)=0$.
To distill entanglement into A and D, the Bell state measurements (BSMs),
the projection measurements into the Bell bases,
are taken for the qubits B-B' and C-C', and the results are informed A and D via classical channels.
Due to the property that A-D and B-C share the same Bell states $\ket{\phi^i}_{AD}\otimes\ket{\phi^i}_{BC}$ in $\rho_s$, the result of the BSMs in each $\ket{\phi^i}_{BC}\otimes \ket{\psi^+}_{B'C'}$ can tell the type of the Bell state shared by A and D.
Table I shows the list of the resulting states shared by A and D for all the possible combinations of the result of the BSM of B-B' and C-C'.
Given this information one can determine the state shared by A and D and then convert any $\ket{\phi^i}_{AD}$ into $\ket{\psi^-}_{AD}$ by local unitary operations.
Hence, the activation protocol can distill entanglement from the Smolin state by four parties' LOCC with the help of the auxiliary two-qubit Bell state.
This is in strong contrast to the unlocking protocol (see Fig. 1 (a)), which requires non-local joint BSM between the two parties (B and C).
It is noteworthy that in our activation protocol, the distillable entanglement between A and D is superadditive:
\begin{equation}
D_{A|D}(\rho_I)  > D_{A|D}(\rho_s) +D_{A|D}(\ket{\psi^+}_{B'C'}) = 0.
\end{equation}
This superadditivity means that the bound entanglement is activated with the help of the auxiliary distillable entanglement, although A and D share no distillable entanglement for $\rho_s$ or $\ket{\psi^+}_{B'C'}$ alone.
This protocol can also be regarded as an entanglement transfer from B-C to A-D.
In this context, it is interesting that two parties (B-C) can transfer the Bell state to the other two parties (A-D) despite being separated: $D_{AD|BC}(\rho_s) = 0$.
In other words, entanglement can be transferred by the mediation of the undistillable, bound entanglement.
This unique feature is quite different from two-stage entanglement swapping \cite{Goebel2008}, which needs distillable entanglement shared by senders and receivers to transfer the Bell states.

\begin{table}[b!]
\caption{Relationship between the combination of the results of the BSM of B-B' and C-C', and the state of AD. Each combination of the BSMs tells the state of A-D. }
\label{table}
\begin{center}\small
\def\arraystretch{1.8}
\begin{tabular}{lcccccc}\hline
                     & &  & $\ket{\phi^+}_{BB'}$ & $\ket{\phi^-}_{BB'}$ & $\ket{\psi^+}_{BB'}$& $\ket{\psi^-}_{BB'}$ \\ \hline
$\ket{\phi^+}_{CC'}$ & &  & $\ket{\psi^+}_{AD}$  & $\ket{\psi^-}_{AD}$  & $\ket{\phi^+}_{AD}$ & $\ket{\phi^-}_{AD} $ \\
$\ket{\phi^-}_{CC'}$ & &  & $\ket{\psi^-}_{AD}$  & $\ket{\psi^+}_{AD}$  & $\ket{\phi^-}_{AD}$ & $\ket{\phi^+}_{AD} $ \\
$\ket{\psi^+}_{CC'}$ & &  & $\ket{\phi^+}_{AD}$  & $\ket{\phi^-}_{AD}$  & $\ket{\psi^+}_{AD}$ & $\ket{\psi^-}_{AD} $ \\
$\ket{\psi^-}_{CC'}$ & &  & $\ket{\phi^-}_{AD}$  & $\ket{\phi^+}_{AD}$  & $\ket{\psi^-}_{AD}$ & $\ket{\psi^+}_{AD} $ \\ \hline
\end{tabular}
\end{center}
\vspace*{-4mm}
\end{table}

\begin{tiny}
\begin{figure}[t!]
   \includegraphics[width=\columnwidth, clip]{activation_exsetup05.eps}
   \label{ }
   \caption{Scheme of the activation of the bound entanglement.
Each source of spontaneous parametric down-conversion (SPDC) produces $\ket{\psi^+}$.
The four-photon states emitted from SPDC1 and SPDC2 pass through liquid crystal variable retarders (LCVRs) to be transformed into the Smolin states.
The polarization of two-photon states in mode A and D are analyzed on the condition that the pair of the Bell state, $\ket{\phi^+}_{BB'}\otimes\ket{\phi^+}_{CC'}$ is detected. P, a polarizer.
 }
\end{figure}
\end{tiny}

Figure 2 illustrates the experimental scheme of our activation protocol.
In our experiment, the physical qubits are polarized photons, having horizontal $\ket{H}$ and vertical $\ket{V}$ polarizations as the state bases.
By using three sources of spontaneous parametric down-conversion (SPDC) \cite{Kwiat95}, we produced three $\ket{\psi^+}$ states simultaneously.
The state $\rho(\psi^+) \equiv \ket{\psi^+}\bra{\psi^+}_{AB}\otimes \ket{\psi^+}\bra{\psi^+}_{CD}$ emitted from the SPDC1 and SPDC2 was transformed into the Smolin state by passing through the synchronized liquid-crystal variable retarders (LCVRs, see Supplementary Material).
The state $\ket{\psi^+}_{B'C'}$ emitted from SPDC3 was used as the auxiliary Bell state for the activation protocol.
A polarizing beam splitter (PBS) and a $\ket{+}_{B}\ket{+}_{B'}$ ($\ket{+}_{C}\ket{+}_{C'}$) coincidence event at modes B and B' (C and C') allow the projection onto the Bell state $\ket{\phi^+}_{BB'}$ ($\ket{\phi^+}_{CC'}$),
where $\ket{+}_{i}= (\ket{H}_i+\ket{V}_i)/\sqrt{2}$  \cite{Goebel2008}.
Given the simultaneous BSMs at B-B' and C-C' we post-selected the events of detecting $\ket{\phi^+}_{BB'} \otimes \ket{\phi^+}_{CC'}$ out of the 16 combinations (Table I).
The result, i.e., the state after the activation, was expected to be $\ket{\psi^+}_{AD}$.
To characterize the experimentally obtained Smolin state $\rho_s^{exp}$ and the state
after the activation process  $\rho_{AD}$,
the maximum likelihood state tomography \cite{James2001} was performed (see Supplementary Material).

\begin{tiny}
\begin{figure}[t!]
   \includegraphics[width=0.7\columnwidth, clip]{2010112701mlre.eps}
   \caption{ Real part of the measured Smolin state $\rho_s^{exp}$.    }
\end{figure}
\end{tiny}

Figure 3 shows the real part of the reconstructed density matrix of the Smolin state $\rho_s^{exp}$ we obtained.
The fidelity to the ideal Smolin state $F_s$ = $ \left({\rm Tr} \sqrt{\sqrt{\rho_s} \rho_s^{exp} \sqrt{\rho_s} }\right)^2$ was calculated to be 82.2$\pm$0.2$\%$.
From $\rho_s^{exp}$, we evaluated the separability of the generated state across the bipartite cuts AB$|$CD, AC$|$BD, and AD$|$BC in terms of the logarithmic negativity (LN) \cite{Vidal2002}, which is an entanglement monotone quantifying the upper bound of the distillable entanglement under LOCC.
The LN of the density matrix $\rho$ composed of the two subsystems $i$ and $j$ is given by
\begin{equation}
LN _{i|j}(\rho ) =\log_2 ||\rho^{T_{i}}||,
\end{equation}
where $\rho^{T_{i}}$ represents the partial transpose with respect to the subsystem $i$, and $||\rho^{T_{i}}||$ is the trace norm of $\rho^{T_{i}}$.
The LN values thus obtained for the three bipartite cuts of $\rho_s^{exp}$ are presented in Table II, together with those of the Smolin state $\rho_s$ and the state $\rho(\psi^+)$.
The state $\rho_s$ has zero negativity for all three bipartite cuts, while the state $\rho(\psi^+)$ has finite values, i.e., finite distillable entanglement originally from the two Bell states, for AC$|$BD and AD$|$BC cuts.
For $\rho_s^{exp}$, the LN values are all close to zero, indicating that $\rho_s^{exp}$ has a very small amount, if any, of distillable entanglement.

\begin{table}[b!]
\caption{Logarithmic negativities (LNs) for the two-two bipartite cuts. }
\label{table}
\begin{center}\small
\def\arraystretch{1.3}
\begin{tabular}{lccc}\hline
                   & $\rho_s^{exp}$      & $\rho_s$ & $\rho(\psi^+)$ \\ \hline
$LN_{AB|CD}(\rho)$ & $0.076 \pm 0.012 $ & 0        &   0 \\
$LN_{AC|BD}(\rho)$ & $0.183 \pm 0.012 $ & 0        &   2 \\
$LN_{AD|BC}(\rho)$ & $0.178 \pm 0.012 $ & 0        &   2 \\\hline
\end{tabular}
\end{center}
\vspace*{-4mm}
\end{table}
To test other separabilities, we calculated the entanglement witness Tr$\{W \rho_s^{exp} \}$, which shows negative values for the non-separable states, and non-negative values for separable ones.
The witness for our four-qubit states is $W = I^{\otimes 4}-\sigma_x^{\otimes 4}-\sigma_y^{\otimes 4}-\sigma_z^{\otimes 4}$ \cite{Amselem2009,Toth2005}.
We obtained Tr$\{W \rho_s^{exp} \}= -1.30 \pm 0.02$, while the values for the ideal Smolin state and the state $\rho(\psi^i)$ were both -2.
The negative witness value indicates that $\rho_s^{exp}$ has no separability in general.
Taking account of the result that $\rho_s^{exp}$ has almost no distillable entanglement for the two-two bipartite cuts, $\rho_s^{exp}$ should have distillable entanglement in one-three bipartite cuts and/or tripartite cuts.

\begin{tiny}
\begin{figure}[t!]
 \includegraphics[width=0.484\columnwidth, clip]{smolin_ad03.eps}
\includegraphics[width=0.49\columnwidth, clip]{rhoad.eps}

     \label{ }
   \caption{Density matrices of the qubits A and D before (a) and after (b) the activation experiment. (a) Reduced-density matrix $ \rho_{sAD}^{exp}$ = Tr$_{BC} \rho_s^{exp}$. (b) The density matrix $\rho_{AD}$, triggered by the two BSMs at B-B' and C-C'.  }
\end{figure}
\end{tiny}
In the following we describe the results of our activation experiment.
Figure 4 (a) shows the reduced density matrix $ \rho_{sAD}^{exp}$ = Tr$_{BC} \rho_s^{exp}$, i.e., the density matrix before the activation.
We confirmed that $ \rho_{sAD}^{exp}$ gives no distillable entanglement to A and D: $ LN_{A|D}(\rho_{sAD}^{exp}) = 0$.
Figure 4 (b) shows the density matrix after the activation $\rho_{AD}$, the reconstructed two-qubit density matrix in modes A and D triggered by the two BSMs at B-B' and C-C'.
In the process of the state reconstruction we subtracted the accidental coincidences caused by higher-order emission of SPDC (see Supplementary Information).
The fidelity $F_{AD} = \bra{\psi^+}\rho_{AD}\ket{\psi^+}_{AD} $ to the ideally activated state $\ket{\psi^+}_{AD}$ was 85$\pm 5$\%, which is larger than the classical limit of 50\%.
The obtained LN was $LN_{A|D}(\rho_{AD}) = 0.83\pm 0.08$, indicating that we have gained a certain amount of entanglement via our activation process, whereas A and D initially share no distillable entanglement.

We quantified the increase of the distillable entanglement via our activation experiment.
We evaluated the distillable entanglement before and after the activation by means of its lower and upper bounds as follows.
The upper bound of $D_{A|D}(\rho_s^{exp})$, the distillable entanglement before the activation, was given by
\begin{align}
D_{A|D}(\rho_s^{exp}) \leq LN_{AB|CD}(\rho_s^{exp} ) = 0.076.
\end{align}
The observed LN value after the activation process, $LN_{A|D}(\rho_{AD} ) = 0.83$, is larger than this value.
However, since these are just the upper bounds of the distillable entanglement, we should examine the lower bound of $\rho_{AD}$ to confirm the increase of the distillable entanglement.
To quantify the lower bound of $D_{A|D}(\rho)$, we used
\begin{align}
D_H(\rho) \leq D_{A|D}(\rho ),
\end{align}
where $D_H(\rho)$ is the distillable entanglement via a certain distillation protocol, the so-called hashing method, known as the best method for Bell diagonal states of rank 2 \cite{Bennett1996}.
$D_H(\rho)$ is given by,
\begin{align}
D_H(\rho ) = 1+F\log_2(F)+(1-F)\log_2\frac{(1-F)}{3},
\end{align}
where $F$ is the maximum state-fidelity over the four Bell states $F$ = max$(\bra{\phi^i}\rho\ket{\phi^i})$.
It is known that $D_H(\rho) > 0$ for $F > 0.8107$ \cite{Bennett1996}.
The fidelity for $\rho_{AD}$ does satisfy this criterion and the value of $D_H(\rho_{AD})$ is calculated to be 0.15.
The combination of Eq. (6) and (7) show a clear increase of the distillable entanglement via our activation experiment:  $D_{A|D}(\rho_s^{exp}) \leq 0.076 < 0.15 \leq D_{A|D}(\rho_{AD})$.

In conclusion, we have experimentally demonstrated the activation of bound entanglement, unleashing the entanglement bound in the Smolin state by means of LOCC with the help of the auxiliary entanglement of the two-qubit Bell state.
We reconstructed the density matrices of the states before and after the activation protocol by full state tomography.
We observed the increase of distillable entanglement via the activation process, examining two inequalities that bind the values of the distillable entanglement.
The gain of distillable entanglement clearly demonstrates the activation protocol in which the undistillable, bound entanglement in $\rho_s$ is essential.
Our result will be fundamental for novel multipartite quantum-communication schemes, for example, quantum key secret-sharing, communication complexity reduction, and remote information concentration, in which general classes of entanglement, including bound entanglement, are important.

This work was supported by a Grant-in-Aid for Creative Scientific Research (17GS1204) from the Japan Society for the Promotion of Science.


\section{Introduction}

Since its inception by Robin Forman \cite{F-98}, discrete Morse
theory has been a powerful and versatile tool used not only in
diverse fields of mathematics, but also in applications to other
areas \cite{Real-15} as well as a computational tool \cite{C-G-N2016}. Its
adaptability stems in part from the fact that it is a discrete
version of the beautiful and successful ``smooth'' Morse theory
\cite{M-63}. While smooth Morse theory touches many branches of
math, one such area that has its origins in critical point theory
is that of Lusternik--Schnirelmann category.  The (smooth)
Lusternik--Schnirelmann category or L--S category of a smooth
manifold $X$, denoted $\cat(X)$, was first introduced in
\cite{L-S}, where the authors proved what is now known as the
Lusternik--Schnirelmann Theorem (see also \cite{CLOT} for a
detailed survey of the topic). A version of this result can be
stated as follows:

\begin{theorem} If $M$ is a compact smooth manifold and $f\colon M \to \RR$ is a smooth map, then
$$\cat(M)+1\leq \sharp(\Crit(f))$$

where $\Crit{(f)}$ is the set of all critical points of the
function $f$.

\end{theorem}

There are many ``smooth'' versions of this theorem in various
contexts (see for example \cite{Palais-66}). The aim of this paper
is to view Forman's discrete Morse theory from a different
perspective in order to prove a discrete version of the L--S theorem
compatible with the recently defined simplicial L--S category
developed by three of the authors \cite{F-M-V}. This simplicial
version of L--S category is suitable for simplicial complexes. Other
attempts have been made to develop such a ``discrete'' L--S
category.  In \cite{AS}, one of the authors developed a discrete
version of L--S category and proved an analogous L--S theorem for
discrete Morse functions. Our version of the L--S Theorem, Theorem
\ref{sLS theorem}, relates a new generalized notion of critical
object of a discrete Morse function to the simplicial L--S category
of \cite{F-M-V}.

In this paper we use the notion of simplicial L--S category defined
in {\cite{F-M-V}}. This simplicial approach of L--S category uses
strong collapses in the sense of Barmak and Minian \cite{B-M-12}
as a framework for developing categorical sets.  As opposed to
standard collapses, strong collapses are natural to consider
in the simplicial setting since they correspond to simplicial maps
(see Figure~\ref{strong_collapse}).

\begin{figure}[htbp]
\begin{center}
\includegraphics[width=.5\linewidth]{strong_collapse}
\caption{An elementary strong collapse from $K$ to $K-\{v\}$.}
\label{strong_collapse}
\end{center}
\end{figure}

This is especially contrasted with (standard) elementary collapses, which in
general do not correspond to simplicial maps (see
Figure~\ref{standard_collapse}).

\begin{figure}[htbp]
\begin{center}
\includegraphics[width=.5\linewidth]{standard_collapse}
\caption{An elementary (standard) collapse from $K$ to $K-\{\sigma , \tau\}$.}
\label{standard_collapse}
\end{center}
\end{figure}

Furthermore, elementary strong collapses correspond to the
deletion of the open star of a dominated vertex. Notice that in
general, the deletion of a vertex is not a simplicial map.

\begin{figure}[htbp]
\begin{center}
\includegraphics[width=.6\linewidth]{vertex_deletion}
\caption{Deletion of a vertex that does not correspond to a simplicial map.}
\label{vertex_deletion}
\end{center}
\end{figure}

This paper is organized as follows.  Section \ref{Fundamentals of
simplicial complexes} contains the necessary background and basics
of simplicial complexes and collapsibility. Section \ref{Strong
discrete Morse theory} is devoted to both reviewing discrete Morse
theory and introducing a generalized notion of critical object in
this context. Here we develop a collapsing theorem for discrete
Morse functions which is analogous to the classical result of
Forman (Theorem 3.3 of \cite{F-98}).

Section \ref{simplicial Lusternik--Schnirelmann category} is the
heart of the paper. In this section, we recall the definitions and
basic properties of the simplicial L--S category and prove the
simplicial L--S theorem in Theorem \ref{sLS theorem}. The rest of
the section is devoted to examples and immediate applications.

\section{Fundamentals of simplicial complexes}\label{Fundamentals of simplicial complexes}

In this section we review some of the basics of simplicial
complexes (see \cite{Munkres} for a more detailed exposition). Let
us start with the definition of simplicial complex. The more usual
way to introduce this notion is to do it geometrically. First, let
us introduce the basic building blocks, that is, the notion of a
simplex. Given $n+1$ points $v_0,\dots,v_n$ in general position in
an Euclidean space, the $n$-simplex $\sigma$ generated by them,
$\sigma=(v_0,\dots,v_n)$, is defined as their convex hull. A
simplex $\sigma$ contains lower dimensional simplices $\tau$,
denoted by $\tau\leq \sigma$, called faces just by considering the
corresponding simplices generated by any subset of its vertices. A
simplicial complex is a collection of simplices satisfying two
conditions:

\begin{itemize}

\item Every face of simplex in a complex is also a simplex of the complex.

\item If two simplices in a complex intersect, then the intersection is also a simplex of the complex.

\end{itemize}

Alternatively, it is possible to define a simplicial complex
abstractly. It will avoid confusion when
parts of the simplicial structure are removed.

Given a finite set $[n]:=\{0,1,2,3,\ldots, n\}$, an \emph{abstract simplicial complex $K$ on $[n]$} is a collection of subsets of $[n]$ such that:

\begin{itemize}
\item If $\sigma \in K$ and $\tau \subseteq \sigma$, then $\tau \in K$.

\item $\{i\} \in K$ for every $i\in [n]$.

\end{itemize}

The set $[n]$ is called the \emph{vertex set} of $K$ and the elements $\{i\}$ are called \emph{vertices} or \emph{$0$-simplices.}   We also may write $V(K)$ to denote the vertex set of $K$.

An element $\sigma \in K$ of cardinality $i+1$ is called an \emph{$i$-dimensional simplex} or \emph{$i$-simplex} of $K$.   The \emph{dimension} of $K$, denoted $\dim(K)$, is the maximum of the dimensions of all its simplices.

\begin{example}  Let $n=5$ and $V(K):=\{0,1,2,3,4,5\}$.\par  \noindent Define $K:=\{\{0\},\{1\}, \{2\}, \{3\}, \{4\}, \{5\},  \{1,2\}, \{1,4\}, \{2,3\},
 \{1,3\}, \{3,4\}, \{3,5\},$\par \hspace*{1.55cm}$\{4,5\}, \{3,4,5\}, \emptyset\}$.\par
 This abstract simplicial complex may be regarded geometrically as follows:

\begin{center}
\includegraphics[width=.3\linewidth]{simplicial_complex}
\end{center}

\end{example}

Further definitions are in order.

\begin{definition}  A \emph{subcomplex} $L$ of $K$, denoted $L\subseteq K$, is a subset $L$ of $K$ such that $L$ is also a simplicial complex.
\end{definition}

We use $\sigma^{(i)}$ to denote a simplex of dimension $i$, and we write $\tau < \sigma^{(i)}$ to denote any subsimplex of $\sigma$ of dimension strictly less than $i$. If $\sigma, \tau\in K$ with $\tau < \sigma$, then $\tau$
is a \emph{face} of $\sigma$ and $\sigma$ is a \emph{coface} of $\tau$.  A simplex of $K$ that is not properly contained in any other simplex of $K$ is called a \emph{facet} of $K$.\\

At this point we recall a key concept in simple-homotopy theory: the notion of simplicial collapse \cite{C-73}.

\begin{definition}  Let $K$ be a simplicial complex and suppose that there is a pair of simplices $\sigma^{(p)}<\tau^{(p+1)}$ in $K$ such that $\sigma$ is a face of $\tau$ and $\sigma$ has no other cofaces.
Such a pair $\{\sigma, \tau\}$ is called a \emph{free pair.}. Then
$K-\{\sigma,\tau\}$ is a simplicial complex called an
\emph{elementary collapse} of $K$ (see
Figure~\ref{standard_collapse}).  The action of collapsing is
denoted $K \searrow K-\{\sigma, \tau\}$.

More generally, $K$ is said to \emph{collapse} onto $L$ if $L$ can
be obtained from $K$ through a finite series of elementary
collapses, denoted $K \searrow L$. In the case where $L=\{v\}$ is
a single vertex, we say that $K$ is  \emph{collapsible}.
\end{definition}

Now we introduce some basic subcomplexes related to a vertex. They play analogous role in the simplicial setting as the closed ball, sphere, and open ball play in the continuous approach.

\begin{definition} Let $K$ be a simplicial complex and let $v\in K$ be a vertex.

The \emph{star of $v$ in $K$}, denoted $\st(v)$, is the subcomplex of simplices $\sigma \in K$ such that $\sigma\cup \{v\}\in K$.

The \emph{link of $v$ in $K$}, denoted $\lk(v)$, is the subcomplex of $\st(v)$ of simplices which do not contain $v$.

The \emph{open star of $v$ in $K$} is $\st^o(v):=\st(v)-\lk(v)$. Note that $st^o(v)$ is not a simplicial subcomplex.

Finally, given a
simplicial complex $K$ and a vertex $v\notin K$, the \emph{cone} $vK$ is
the simplicial complex whose simplices are $\{v_0,\dots,v_n\}$ and $\{v,v_0,\dots,v_n\}$
where $\{v_0,\dots,v_n\}$ is any simplex of $K$.
\end{definition}

It is important to point out that simplicial collapses are not
simplicial maps in general. This suggests that it is natural to
consider a special kind of collapse which is a simplicial map. The
notion of strong collapse, introduced by Barmak and Minian in
\cite{BARMAK2011,B-M-12}, satisfies this requirement.

\begin{definition} Let $K$ be a simplicial complex and suppose there exists a pair of vertices $v,v^\prime\in K$ such that every
maximal simplex containing $v$ also contains $v^\prime$. Then we
say that $v^\prime$ \emph{dominates} $v$ and $v$ is
\emph{dominated by} $v^\prime$.

If $v$ is dominated by $v^\prime$ then the inclusion $i: K - \{v\}
\to K$ is a strong equivalence. Its homotopical inverse is the
retraction $r \colon K \to K - \{v\}$ which is the identity on $K
- \{v\}$ and such that $r(v) = v^\prime$. This retraction is
called an \emph{elementary strong collapse} from $K$ to $K -
\{v\}$, denoted by $K \searrow\searrow K - \{v\}$.

A \emph{strong collapse} is a finite sequence of elementary strong
collapses. The inverse of a strong collapse is called a
\emph{strong expansion} and two complexes $K$ and $L$ \emph{have
the same strong homotopy type} if there is a sequence of strong
collapses and strong expansions that transform $K$ into $L$.
\end{definition}

See Figure~\ref{strong_collapse} to illustrate the above notions.

Equivalently, $v$ is a dominated vertex if and only if its link is
a cone \cite{BARMAK2011}.

\section{Strong discrete Morse theory}\label{Strong discrete Morse theory}

We are now ready to introduce a key object of our study, that is,
discrete Morse functions in the R. Forman's sense \cite{F-98}.
More precisely, we are interested in a generalized notion
of critical object suitable for codifying the strong homotopy type
of a complex.

\begin{definition}  Let $K$ be a simplicial complex. A \emph{discrete Morse function} $f\colon K \to \RR$ is a function satisfying for any $p$-simplex
$\sigma\in K$:
\begin{enumerate}
\item[(M1)] $\sharp\left(\{\tau^{(p+1)}>\sigma : f(\tau)\leq f(\sigma)\}\right) \leq 1$.
\item[(M2)] $\sharp\left(\{\upsilon^{(p-1)}<\sigma : f(\upsilon)\geq f(\sigma)\}\right) \leq 1$.
\end{enumerate}
\end{definition}

Roughly speaking, we can say that it is a weakly increasing
function which satisfies the property that if $f(\sigma)=f(\tau)$,
then one of both simplices is a maximal coface of the other one.

\begin{definition}
A \emph{critical simplex} of $f$ is a simplex $\sigma$ satisfying:

\begin{enumerate}
\item[(C1)] $\sharp\left(\{\tau^{(p+1)}>\sigma : f(\tau)\leq f(\sigma)\}\right) =0$.
\item[(C2)] $\sharp\left(\{\upsilon^{(p-1)}<\sigma : f(\upsilon)\geq f(\sigma)\}\right)=0$.
\end{enumerate}
\end{definition}

If $\sigma$ is a critical simplex, the number $f(\sigma)\in \RR$ is called a \emph{critical value}. Any simplex that is not critical is called a \emph{regular simplex} while any output value of the discrete Morse function
which is not a critical value is a \emph{regular value}. The set of critical simplices of $f$ is denoted by $\Crit(f)$.\\

Given any real number $c$, the \emph{level subcomplex} of $f$ at
level $c$, $K(c)$, is the subcomplex of $K$ consisting of all
simplices $\tau$ with $f(\tau)\leq c$, as well as all of their
faces, that is,

$$ K(c)=\bigcup_{f(\tau)\leq c}\bigcup_ {\sigma \leq \tau} \sigma .$$

Any discrete Morse function induces a gradient vector field.

\begin{definition}\label{GVF}  Let $f$ be a discrete Morse function on $K$.  The \emph{induced gradient vector field} $V_f$ or $V$ when the context is clear, is defined by the following set of pairs of simplices:
$$V_f:=\{(\sigma^{(p)}, \tau^{(p+1)}) : \sigma<\tau, f(\sigma)\geq f(\tau)\}.
$$
\end{definition}

Note that critical simplices are easily
identified in terms of the gradient field as precisely
those simplices not contained in any pair in the gradient field.

\begin{definition}\label{min outside} Let $f\colon K\to \RR$ be a discrete Morse function and $V_f$ its induced gradient vector field.  For each vertex/edge pair
$(v,uv)\in V_f$, write $\mathrm{St}(v,u):=\st^o(v)\cap \st(u)$.
Define $m_v:=\min\{f(\tau): f(\tau)>f(uv), \tau \in
(K-\mathrm{St}(v,u))\bigcup (\Crit(f)\cap \mathrm{St}(v,u) )\}$.
\end{definition}

\begin{example}\label{cutoff illustration_a}
We illustrate Definition \ref{min outside}.  Let $K$ be the simplicial complex with
discrete Morse function $f$ given in Figure~\ref{DMT_example1a}.

\begin{figure}[htbp]
\begin{center}
\includegraphics[width=.95\linewidth]{DMT_example1a}
\caption{A 2-dimensional simplicial complex with a discrete Morse
function.} \label{DMT_example1a}
\end{center}
\end{figure}

Observe that taking $v=f^{-1}(10)$ and $u=f^{-1}(0)$, we have
$(v,uv)\in V_f$ and
$$\mathrm{St}(v,u)=f^{-1}(\{9,10,11,12,13,14\})\,.$$ Since $\mathrm{St}(v,u)$ in Example~\ref{cutoff illustration_a} does not contain any
critical simplex, then determining $m_v$ consists on finding the
smallest value greater than $f(uv)$ outside of $\mathrm{St}(v,u)$.
In this case, $m_v=15$.
\end{example}

\begin{example}\label{cutoff illustration_b}
Now $f$ will be slightly modified to create a new discrete Morse
function $g$ on $K$ (Figure~\ref{DMT_example1b}). It is clear that $(v,uv)\in V_g$, but in this case $m_v=11$.

\begin{figure}[htbp]
\begin{center}
\includegraphics[width=.95\linewidth]{DMT_example1b}
\caption{The same 2-dimensional simplicial complex of
Figure~\ref{DMT_example1a} with another discrete Morse function.}
\label{DMT_example1b}
\end{center}
\end{figure}

\end{example}

Notice that the values that take both functions $f$ and $g$ on
some simplices coincide. It is interesting to point out that both
examples have the same induced gradient vector field and
consequently, contain the same critical simplices in the usual
Forman sense. This justifies the need to take into account
additional information in this new approach and thus a more general concept of critical object.

\begin{definition}\label{strong definition} Continuing with the notation used in Definition \ref{min outside}, define $l_v$ as the largest regular value in $f(\mathrm{St}(v,u))$ such that
\begin{itemize}
  \item $f(uv)\leq l_v\leq m_v$
  \item every maximal regular simplex of $K(l_v)\cap \mathrm{St}(v,u)$ contains the vertex $u$.
\end{itemize}

\end{definition}

\begin{definition}
Under the notation of Definition \ref{min outside}, the
\emph{strong collapse of $v$ under $f$},\linebreak denoted
$S^f_v$, is given by $S^{f}_v:=\{ (\sigma, \tau)\in V_f :
f(uv)\leq f(\tau)\leq l_v\}$ and the interval
$I(S^f_v)=[f(uv),l_v]$ is called the \emph{strong interval} of
$S_v$.

The elements of the set $C(V_f):=V_f-\bigcup S^f_v$ are the
\emph{critical pairs} of $f$ while each element in $\bigcup
S_v$ is a \emph{regular pair} of $f$.  If $(\sigma, \tau)$ is a
critical pair, the value $f(\tau)$ is a \emph{critical value} of
$f$.

A \emph{critical object} is either a critical simplex (in the
standard Forman sense) or a critical pair. The set of all critical
objects of $f$ is denoted by $\scrit(f)$. In order to avoid
confusion, we call the images of all critical objects (either in
the Forman sense or from a critical pair) \emph{strong critical
values}.
\end{definition}

\begin{remark}

It is worthwhile to mention that critical pairs are
detecting when a standard collapse has been made. Notice that it may happen either due to combinatorial
reasons (e.g. the non-existence of dominated vertices in the
corresponding subcomplex) or to a bad choice of the values of the
discrete Morse function (i.e. noise). This second option induces a bad ordering in
the way the standard collapses are made inside a potential strong one. The
key idea is that every elementary strong collapse should ideally
be made as a uninterrupted sequence of standard collapses. So
every time it is not made in this way, a new critical object
appears.
\end{remark}

\begin{example}
Define $f$ and $g$ as in Examples \ref{cutoff illustration_a} and
\ref{cutoff illustration_b}. We see that $c^f_v=14$ while
$l_v^g=10$.  Hence (by abuse of notation), $(16,15)$ is a critical
pair under $f$ while $(14,13)$, $(16,15)$ and $(18,17)$ are
critical pairs under $g$. Notice that the strong intervals are
$I(S^f_v)=[9,14]$ and $I(S^g_v)=[9,10]$, respectively.
\end{example}

\begin{example}\label{arg exam}
To further illustrate Definition \ref{strong definition}, we shall
consider two different discrete Morse functions defined on a
collapsible but non-strongly collapsible triangulation of the
$2$-disc. Notice that both of them have only a single critical
value (in the Forman sense), but very different numbers of
critical objects, due to a bad election in the ordering of normal
collapses of $g$.

Let $f$ be given in Figure~\ref{DMT_example2a}.

\begin{figure}[htbp]
\begin{center}
\includegraphics[width=.7\linewidth]{DMT_example2a}
\caption{A discrete Morse function defined on a non-strongly
collapsible triangulation of the $2$-disc.} \label{DMT_example2a}
\end{center}
\end{figure}

By abuse of notation, we refer to the simplices by their labeling
under the discrete Morse function (the fact that pairs in $V_f$
are given the same label should not cause confusion).  For each of
the pairs $(9,9), (6,6), (3,3),(2,2),(1,1)\in V_f$, we have the
corresponding values $l_9=11, l_6=8, l_3=5, l_2=2$, and $l_1=1$.
The corresponding strong collapses under the indicated vertices
are given by $$
\begin{array}{ccl}
  S^f_9 & = & \{ (9,9), (10,10), (11,11)\},\\
  S^f_6 & = & \{(6,6),(7,7), (8,8)\}, \\
  S^f_3 & = & \{(3,3), (4,4), (5,5)\}, \\
  S^f_2 & = & \{(2,2)\} \mbox{ and }\\
  S^f_1 & = & \{(1,1)\}\,.
\end{array}$$
Thus we obtain the following strong intervals: $$\begin{array}{c}
                                                   I(S^f_9)=[9,11], I(S^f_6)=[6,8], I(S^f_3)=[3,5], I(S^f_2)=[2,2]=\{ 2 \} \\
                                                   \mbox{ and } I(S^f_1)=[1,1]=\{ 1 \}\,.
                                                 \end{array}$$
Hence there is a single critical pair, namely, $(13,13)$, so that $\scrit(f)=\{0, (13,13)\}$.

It is interesting to point out that this discrete Morse function
can be considered as optimal in the sense that it minimizes the
number of critical objects, as we will see in Theorem \ref{sLS
theorem}.

Now let $g$ be the discrete Morse function given in
Figure~\ref{DMT_example2b}.

\begin{figure}[htbp]
\begin{center}
\includegraphics[width=.7\linewidth]{DMT_example2b}
\caption{A different discrete Morse function defined on a
non-strongly collapsible triangulation of the $2$-disc.}
\label{DMT_example2b}
\end{center}
\end{figure}

Now we consider the pairs $(9,9),(8,8),(7,7), (5,5), (3,3)\in V_g$
and obtain corresponding values $ l_9=10, l_8=8, l_7=7,l_5=6$ and
$l_3=3.$ The corresponding strong collapses are then given by
$$\begin{array}{ccl}
    S^g_9 & = & \{(9,9)\}, \\
    S^g_8 & = & \{(8,8)\}, \\
    S^g_7 & = & \{(7,7)\}, \\
    S^g_5 & = & \{(5,5),(6,6)\}\mbox{ and } \\
    S^g_3 & = & \{(3,3)\}\,.
  \end{array}$$
It follows that the strong intervals are: $$\begin{array}{c}
    I(S^g_9)=[9,9]=\{ 9 \}, I(S^g_8)=[8,8]=\{ 8 \}, I(S^g_7)=[7,7]=\{ 7 \}, I(S^g_5)=[5,6]\\
    \mbox{ and } I(S^g_3)=[3,3]=\{3 \}\,.
  \end{array}$$
We conclude that $\scrit(g)=\{ 0, (15,15), (13,13), (14,14), (10,10),(12,12), (11,11)\}$ for a total of $7$ critical
objects.

\end{example}

The following Theorem is the analogue of Forman's classical result
Theorem 3.3 of \cite{F-98} and also analogous to the classical
result in the smooth setting establishing homotopy equivalence of
sublevel sets by assuming non-existence of critical values.

\begin{theorem}\label{analagous collapse} Let $f$ be a discrete Morse function on $K$ and suppose that $f$ has no strong critical values on $[a,b]$. Then $K(b)\searrow\searrow K(a)$.
In particular, if $I(S^f_v)=[f(uv),l_v]$ is a strong interval for
some vertex $v\in K$, then $K(l_v)\searrow\searrow K(f(uv))$.
\end{theorem}

\begin{proof} Let us consider the interval $[a,b]$ such that it does not contain any strong critical value.
If additionally this interval does not contain any strong interval, then we conclude that $K(l_1)=K(l_2)$.
Hence, assume that $[a, b]$ contains strong intervals. We may
then partition $[a,b]$ into subintervals such that each one
of them contains exactly one strong interval. Let us suppose
$I(S_v)=[f(uv), l_v]\subseteq [a, b]$ is the unique strong
interval in $[a, b]$. Again, since $f$ does not take values
in $(a, f(uv))$  or $(l_v,b)$, it follows that
$K(a)=K(f(uv))$ and $K(l_v)=K(b)$. Thus it suffices to show
that $K(l_v)\searrow\searrow K(f(uv))$. By definition of $l_v$,
$u$ is contained in every maximal regular simplex of $K(l_v) \cap
\mathrm{St}(v, u).$ Furthermore, since $l_v< m_v$, all the new
simplices that are attached from $K(f(uv))$ to $K(l_v)$ are
contained within $\mathrm{St}(v, u)$. Hence
$K(l_v)-K(f(uv)-\epsilon)$ is an open cone with apex $u$ and thus,
$K(l_v)\searrow\searrow K(f(uv))$.

\end{proof}

\section{Simplicial Lusternik--Schnirelmann category}\label{simplicial Lusternik--Schnirelmann category}

In this section, we recall the fundamental definitions and results found in \cite{F-M-V}.

\begin{definition} Let $K,L$ be simplicial complexes.  We say that two simplicial maps\linebreak $\phi, \psi \colon K\to L$ are \emph{contiguous},
denoted $\phi\sim_c \psi$, if for any simplex $\sigma \in K$, we
have that $\phi(\sigma)\cup \psi(\sigma)$ is a simplex of $L$.
Because this relation is reflexive and symmetric but not
transitive, we say that $\phi$ and $\psi$ are in the same
\emph{contiguity class}, denoted $\phi\sim \psi$, if there is a
sequence $\phi=\phi_0\sim_c \phi_1 \sim_c\ldots \sim_c \phi_n
=\psi$ of contiguous simplicial maps $\phi_i\colon K\to L, 0\leq i
\leq n$. A map $\phi \colon K \to L$ is a \emph{strong
equivalence} if there exists $\psi\colon  L \to K$ such that $\psi
\phi \sim \id_K$ and $\phi \psi \sim \id_L$. In this case, we say
that $K$ and $L$ are \emph{strongly equivalent}, denoted by $K
\sim L$.
\end{definition}

There is a nice link between strong equivalences and strong
collapses.

\begin{theorem}\cite[Cor. 2.12]{B-M-12} Two complexes $K$ and $L$ have the same strong homotopy type if and only if $K \sim L$.
\end{theorem}

\begin{definition}\cite{F-M-V}\label{scat definition} Let K be a simplicial complex. We say that the subcomplex $U \subseteq K$ is
\emph{categorical} in $K$ if there exists a vertex $v \in K$
such that the inclusion $i\colon  U \to K$ and the constant map
$c_v \colon U \to K$ are in the same contiguity class.  The
\emph{simplicial L--S category}, denoted \emph{$\scat (K)$}, of
the simplicial complex $K$, is the least integer $m \geq 0$ such
that $K$ can be covered by $m + 1$ categorical subcomplexes.
\end{definition}

One of the basic results of classic Lusternik-Schnirelmann theory
states that the L--S category is homotopy invariant. Next result shows
that simplicial L--S category satisfies the analogous property in
the discrete setting.

\begin{theorem}\label{FMV}\cite[Theorem 3.4]{F-M-V} Let $K \sim L$ be two strongly equivalent complexes. Then $\scat(K) = \scat(L)$.
\end{theorem}

We refer the interested reader to the papers \cite{F-M-V,F-M-M-V} for a
detailed study of this topic.

\subsection{Simplicial Lusternik--Schnirelmann theorem}

Our main result is the following simplicial version of the
Lusternik-Schnirelmann Theorem:

\begin{theorem}\label{sLS theorem} Let $f\colon K\to \RR$ be a discrete Morse function.  Then
$$\scat(K)+1\leq \sharp(\scrit(f))\,.$$
\end{theorem}

\begin{proof}
For any natural number $n$, define $c_n :=\min\{a\in \RR :
\scat(K(a))\geq n-1\}$. We claim that $c_n$ is a strong critical
value of $f$.  If $c_n$ is a regular value, then it is either
contained in a strong interval or it is not.  If $c_n$ is
contained in a strong interval $I(c_v)$, then by Theorem
\ref{analagous collapse}, we have $\scat(K(c_v))=\scat(K(c_n))$,
contradicting the minimality of $c_n$.  Otherwise, $c_n$ is
outside a strong interval. Then, by Theorem \ref{analagous
collapse} there exists $\epsilon >0$ such that
$K(c_n)\searrow\searrow K(c_n-\epsilon)$. By Theorem \ref{FMV},
$\scat(K(c_n))=\scat(K(c_n-\epsilon))$. But $c_n>c_n-\epsilon$ and
$c_n$ was the minimum value such that $\scat(K(c_n))=n-1$, which
is a contradiction.  Thus each $c_n$ is a strong critical value of
$f$.

\smallskip

We now prove by induction on $n$ that $K(c_n)$ must contain at
least $n$ critical objects.  By the well-ordering principle, the
set $\im(f)$ has a minimum, say $f(v)=0$ for some $0$-simplex
$v\in K$.  For $n=1$, $c_1=0$ so that $K(c_1)$ contains $1$
critical object.  For the inductive hypothesis, suppose that
$K(c_n)$ contains at least $n$ strong critical objects.  Since all
the strong critical values of $f$ are distinct, $c_n<c_{n+1}$ so
that there is at least one new critical object in
$f^{-1}(c_{n+1})$. Thus $K(c_{n+1})$ contains at least $n+1$
critical objects.  Hence if $c_1<c_2<\ldots < c_{\scat(K)+1}$ are
the critical objects, then $K(c_{\scat(K)+1})\subseteq K$ contains
at least $\scat(K)+1$ critical objects.  Thus $\scat(K)+1\leq
\sharp(\scrit(f))$.
\end{proof}

\begin{example}\label{LS equality}
We give an example where the upper bound of Theorem \ref{sLS
theorem} is attained. Let $A$ be the several times considered
non-strongly collapsible triangulation of the $2$-disc with the
discrete Morse function $f$ given in Example \ref{arg exam}.  This
satisfies $\sharp(\scrit(f))=2$. Since $A$ has no dominating
vertex, $\scat(A)>0$, whence $\scat(A)+1=\sharp(\scrit(f))=2$.\\

Notice that just adding one simplex to $A$, the above equality
turns into a strict inequality and thus, the number of critical
objects may increase while the simplicial category keeps the same.
To see this, let us consider $A'$ as the clique complex of $A$,
that is, we add to $A$ the triangle generated by its three
bounding vertices. This is a simplicial $2$-sphere, so by means of
Morse inequalities, it follows that every discrete Morse function
$f$ defined on $A'$ has at least two critical simplices: one
critical vertex (global minimum) and a critical triangle (global
maximum). In addition, since $A'$ is a simplicial $2$-sphere, then
it does not contain any dominated vertex. Moreover, after removing
the critical triangle (and hence obtaining $A$) no new dominated
vertices appear, so at least one critical pair arises in order to
collapse $A$ to a subcomplex containing dominated vertices. Hence,
we conclude that that any discrete Morse function $f\colon A' \to
\RR$ must have at least $3$ critical objects, which could be
considered optimal in the sense that it minimizes the number of
critical objects. Furthermore, it is easy to cover $A'$ with $2$
categorical sets so that $\scat(A')=1$. Hence $\scat(A')+1=2<3\leq
\sharp(\scrit(f))$.
\end{example}

\bibliographystyle{amcjoucc}