\section{Introduction}
\label{irl::intro}
Approximately 350,000 Americans suffer from serious spinal cord injuries (SCI), resulting in loss of
some voluntary motion control. Recently, epidural and transcutaneous spinal stimulation have proven
to be promising methods for regaining motor function. To find the optimal stimulation signal, it is
necessary to quantitatively measure the effects of different stimulations on a patient. Since motor
function is our concern, we mainly study the effects of stimulations on patient motion, represented
by a sequence of poses captured by motion sensors. One typical experiment setting is shown in Figure
\ref{irl:game}, where a patient moves to follow a physician's instructions, and a sensor records the
patient's center-of-pressure (COP) continuously. This study will assist our design of stimulating
signals, as well as advancing the understanding of patient motion with spinal cord injuries.

\begin{figure}
  \centering
  \subfloat[A patient sitting on a sensing device\label{irl::game1}]{
    \includegraphics[width=0.2\textwidth]{game.eps}
  }
  \qquad
  \subfloat[Instructions on movement directions\label{irl::game2}]{
    \includegraphics[width=0.2\textwidth]{instructions.eps}
  }
  ~
  \subfloat[The patient's COP trajectories during the movement\label{irl::game3}]{
    \includegraphics[width=0.2\textwidth]{gametrajectory.eps}
  }
  \caption{Rehabilitative game and observed trajectories: in Figure \ref{irl::game1}, the patient
  sits on a sensing device, and then moves to follow the instructed directions in Figure
  \ref{irl::game2}. Figure  \ref{irl::game3} shows the patient's center-of-pressure (COP) during
  the movements.}
  \label{irl:game}
\end{figure}

We assume the stimulating signals will alter the patient's initial preferences over poses,
determined by body weight distribution, spinal cord injuries, gravity, etc., and an accurate
estimation of the preference changes will reveal the effect of spinal stimulations on spinal cord
injuries, as other factors are assumed to be invariant to the stimulations. To estimate the
patient's preferences over different poses, the most straightforward approach is counting the pose
visiting frequencies from the motion, assuming that the preferred poses are more likely to be
visited.  However, the patient may visit an undesired pose to follow the instructions or to change
into a subsequent preferred poses, making preference estimation inaccurate without regarding the
context.

In this work, we formulate the patient's motion as a Markov Decision Process, where each state
represents a pose, and its reward value encodes all the immediate factors motivating the patient to
visit this state, including the pose preferences and the physician's instructions. With this
formulation, we adopt inverse reinforcement learning (IRL) algorithms to estimate the reward value
of each state from the observed motion of the patient.

Existing solutions of the IRL problem mainly work on small-scale problems, by collecting a set of
observations for reward estimation and using the estimated reward afterwards.  For example, the
methods in \cite{irl::irl1,irl::irl2, irl::subgradient} estimate the agent's policy from a set of
observations, and estimate a reward function that leads to the policy.  The method in
\cite{irl::maxentropy} collects a set of trajectories of the agent, and estimates a reward function
that maximizes the likelihood of the trajectories. However, the state space of human motion is huge
for non-trivial analysis, and these methods cannot handle it well due to the reinforcement learning
problem in each iteration of reward estimation. Several methods \cite{irl::guidedirl,irl::relative}
solve the problem by approximating the reinforcement learning step, at the expense of a
theoretically sub-optimal solution.

The problem can be simplified under the condition that the transition model and the action set
remain unchanged for the subject, thus each reward function leads to a unique optimal value
function. Based on this assumption, we propose a function approximation method that learns the
reward function and the optimal value function, but without the computationally expensive
reinforcement learning steps, thus it can be scaled to a large state space. We find that this
framework can also extend many existing methods to high-dimensional state spaces.

The paper is organized as follows. We review existing work on inverse reinforcement learning in
Section \ref{irl::related}, and formulate the function approximation inverse reinforcement learning
method in large state spaces in \ref{irl::largeirl}. A simulated experiment and a clinical
experiment are shown in Section \ref{irl::experiments}, with conclusions in Section
\ref{irl::conclusions}.

\section{Related Works}
\label{irl::related}
The idea of inverse optimal control is proposed by Kalman \cite{irl::kalman}, white the inverse
reinforcement learning problem is firstly formulated in \cite{irl::irl1}, where the agent observes
the states resulting from an assumingly optimal policy, and tries to learn a reward function that
makes the policy better than all alternatives. Since the goal can be achieved by multiple reward
functions, this paper tries to find one that maximizes the difference between the observed policy
and the second best policy. This idea is extended by \cite{irl::maxmargin}, in the name of
max-margin learning for inverse optimal control. Another extension is proposed in \cite{irl::irl2},
where the purpose is not to recover the real reward function, but to find a reward function that
leads to a policy equivalent to the observed one, measured by the amount of rewards collected by
following that policy.

Since a motion policy may be difficult to estimate from observations, a behavior-based method is
proposed in \cite{irl::maxentropy}, which models the distribution of behaviors as a maximum-entropy
model on the amount of reward collected from each behavior. This model has many applications and
extensions. For example, \cite{irl::sequence} considers a sequence of changing reward functions
instead of a single reward function. \cite{irl::gaussianirl} and \cite{irl::guidedirl} consider
complex reward functions, instead of linear one, and use Gaussian process and neural networks,
respectively, to model the reward function. \cite{irl::pomdp} considers complex environments,
instead of a well-observed Markov Decision Process, and combines partially observed Markov Decision
Process with reward learning. \cite{irl::localirl} models the behaviors based on the local
optimality of a behavior, instead of the summation of rewards.  \cite{irl::deepirl} uses a
multi-layer neural network to represent nonlinear reward functions.

Another method is proposed in \cite{irl::bayirl}, which models the probability of a behavior as the
product of each state-action's probability, and learns the reward function via maximum a posteriori
estimation. However, due to the complex relation between the reward function and the behavior
distribution, the author uses computationally expensive Monte-Carlo methods to sample the
distribution. This work is extended by \cite{irl::subgradient}, which uses sub-gradient methods to
simplify the problem.  Another  extensions is shown in \cite{irl::bayioc}, which tries to find a
reward function that matches the observed behavior. For motions involving multiple tasks and varying
reward functions, methods are developed in \cite{irl::multirl1} and \cite{irl::multirl2}, which try
to learn multiple reward functions.

Most of these methods need to solve a reinforcement learning problem in each step of reward
learning, thus practical large-scale application is computationally infeasible. Several methods are
applicable to large-scale applications. The method in \cite{irl::irl1} uses a linear approximation
of the value function, but it requires a set of manually defined basis functions. The methods in
\cite{irl::guidedirl,irl::relative} update the reward function parameter by minimizing the relative
entropy between the observed trajectories and a set of sampled trajectories based on the reward
function, but they require a set of manually segmented trajectories of human motion, where the
choice of trajectory length will affect the result. Besides, these methods solve large-scale
problems by approximating the Bellman Optimality Equation, thus the learned reward function and Q
function are only approximately optimal. We propose an approximation method that guarantees the
optimality of the learned functions as well as the scalability to large state space problems.

\section{Function Approximation Inverse Reinforcement Learning}
\label{irl::largeirl}
\subsection{Markov Decision Process}
A Markov Decision Process is described with the following variables:
\begin{itemize}
  \item $S=\{s\}$, a set of states
  \item $A=\{a\}$, a set of actions
  \item $P_{ss'}^a$, a state transition function that defines the probability that state $s$ becomes
    $s'$ after action $a$.
  \item $R=\{r(s)\}$, a reward function that defines the immediate reward of state $s$.
  \item $\gamma$, a discount factor that ensures the convergence of the MDP over an infinite
    horizon.
\end{itemize}

A motion can be represented as a sequence of state-action pairs:
\[\zeta=\{(s_i,a_i)|i=0,\cdots,N_\zeta\},\]
where $N_\zeta$ denotes the length of the motion, varying in different observations. Given the
observed sequence, inverse reinforcement learning algorithms try to recover a reward function that
explains the motion.

One key problem is how to model the action in each state, or the policy, $\pi(s)\in A$, a mapping
from states to actions. This problem can be handled by reinforcement learning algorithms, by
introducing the value function $V(s)$ and the Q-function $Q(s,a)$, described by the Bellman Equation
\cite{irl::rl}:
\begin{align}
  &V^\pi(s)=\sum_{s'|s,\pi(s)}P_{ss'}^{\pi(s)}[r(s')+\gamma*V^\pi(s')],\\
  &Q^\pi(s,a)=\sum_{s'|s,a}P_{ss'}^a[r(s')+\gamma*V^\pi(s')],
\end{align}
where $V^\pi$ and $Q^\pi$ define the value function and the Q-function under a policy $\pi$.

For an optimal policy $\pi^*$, the value function and the Q-function should be maximized on every
state. This is described by the Bellman Optimality Equation \cite{irl::rl}:
\begin{align}
  &V^*(s)=\max_{a\in A}\sum_{s'|s,a}P_{ss'}^a[r(s')+\gamma*V^*(s')],\\
  &Q^*(s,a)=\sum_{s'|s,a}P_{ss'}^a[r(s')+\gamma*\max_{a'\in A}Q^*(s',a')].
\end{align}

In typical inverse reinforcement learning algorithms, the Bellman Optimality Equation needs to be
solved once for each parameter updating of the reward function, thus it is computationally
infeasible when the state space is large. While several existing approaches solve the problem at the
expense of the optimality, we propose an approximation method to avoid the problem.

\subsection{Function Approximation Framework}
Given the set of actions and the transition probability, a reward function leads to a unique optimal
value function. To learn the reward function from the observed motion, instead of directly
learning the reward function, we use a parameterized function, named as \textit{VR function}, to
represent the summation of the reward function and the discounted optimal value function:
\begin{equation}
  f(s,\theta)=r(s)+\gamma*V^*(s),
  \label{equation:approxrewardvalue}
\end{equation}
where $\theta$ denotes the parameter of \textit{VR function}. The function value of a state is named
as \textit{VR value}.

Substituting Equation \eqref{equation:approxrewardvalue} into Bellman Optimality Equation, the
optimal Q function is given as:
\begin{equation}
  Q^*(s,a)=\sum_{s'|s,a}P_{ss'}^af(s',\theta),
  \label{equation:approxQ}
\end{equation}
the optimal value function is given as:
\begin{align}
  V^*(s)&=\max_{a\in A}Q^*(s,a)\nonumber\\
        &=\max_{a\in A}\sum_{s'|s,a}P_{ss'}^af(s',\theta),
  \label{equation:approxV}
\end{align}
and the reward function can be computed as:
\begin{align}
  r(s)&=f(s,\theta)-\gamma*V^*(s)\nonumber\\
      &=f(s,\theta)-\gamma*\max_{a\in A}\sum_{s'|s,a}P_{ss'}^af(s',\theta).
  \label{equation:approxR}
\end{align}

This approximation method is related to value function approximation method in reinforcement
learning, but the proposed method can compute the reward function without solving a set of linear
equations in stochastic environments.

Note that this formulation can be generalized to other extensions of Bellman Optimality Equation by
replacing the $max$ operator with other types of Bellman backup operators. For example,
$V^*(s)=\log_{a\in A}\exp Q^*(s,a)$ is used in the maximum-entropy method\cite{irl::maxentropy};
$V^*(s)=\frac{1}{k}\log_{a\in A}\exp k*Q^*(s,a)$ is used in Bellman Gradient Iteration
\cite{irl::BGI}.

For any \textit{VR function} $f$ and any parameter $\theta$, the optimal Q function $Q^*(s,a)$,
optimal value function $V^*(s)$, and reward function $r(s)$ constructed with Equation
\eqref{equation:approxQ}, \eqref{equation:approxV}, and \eqref{equation:approxR} always meet the
Bellman Optimality Equation. Under this condition, we try to recover a parameterized function
$f(s,\theta)$ that best explains the observed motion $\zeta$ based on a predefined motion model.

Combined with different Bellman backup operators, this formulation can extend many existing methods
to high-dimensional spaces, like the motion model based on the value function in
\cite{irl::motionvalue}, $p(a|s)=-v(s)-\log\sum_k p_{s,k}\exp(-v(k))$, the reward function in
\cite{irl::maxentropy}, $p(a|s)=\exp{Q(s,a)-V(s)}$, and the Q function in \cite{irl::bayirl}. The
main limitation is the assumption of a known transition model $P_{ss'}^a$, but it only requires a
partial model on the experienced states rather than a full environment model, and it can be learned
independently in an unsupervised way.

To demonstrate the usage of the framework, this work chooses $max$ as the Bellman backup operator
and a motion model $p(a|s)$ based on the optimal Q function $Q^*(s,a)$ \cite{irl::bayirl}:
\begin{equation}
  P(a|s)=\frac{\exp{b*Q^*(s,a)}}{\sum_{\tilde{a}\in
  A}\exp{b*Q^*(s,\tilde{a})}},
  \label{equation:motionmodel}
\end{equation}
where $b$ is a parameter controlling the degree of confidence in the agent's ability to choose
actions based on Q values. In the remaining sections, we use $Q(s,a)$ to denote the optimal Q values
for simplified notations.

\subsection{Function Approximation with Neural Network}
Assuming the approximation function is a neural network, the parameter $\theta=\{w,b\}$-weights and
biases-in Equation \eqref{equation:approxrewardvalue} can be estimated from the observed sequence
of state-action pairs $\zeta$ via maximum-likelihood estimation:
\begin{equation}
  \label{equation:maxtheta}
  \theta=\argmax_{\theta}\log{P(\zeta|\theta)},
\end{equation}
where the log-likelihood of $P(\zeta|\theta)$ is given by:
\begin{align}
  L_{nn}(\theta)&=\log{P(\zeta|\theta)}\nonumber\\
           &=\log{\prod_{(s,a)\in \zeta} P(a|\theta;s)}\nonumber\\
           &=\log{\prod_{(s,a)\in \zeta} \frac{\exp{b*Q^*(s,a)}}{\sum_{\hat{a}\in A}\exp{b*Q^*(s,\hat{a})}}
}\nonumber\\
  &=\sum_{(s,a)\in\zeta}(b*Q(s,a)-\log{\sum_{\hat{a}\in
  A}\exp{b*Q(s,\hat{a}))}},
  \label{equation:loglikelihood}
\end{align}
and the gradient of the log-likelihood is given by:
\begin{align}
  \nabla_\theta L_{nn}(\theta)&=\sum_{(s,a)\in\zeta}(b*\nabla_\theta Q(s,a)\nonumber\\
  &-b*\sum_{\hat{a}\in
  A}P((s,\hat{a})|r(\theta))\nabla_\theta Q(s,\hat{a})).
  \label{equation:loglikelihoodgradient}
\end{align}

With a differentiable approximation function,
\[\nabla_\theta Q(s,a)=\sum_{s'|s,a}P_{ss'}^a\nabla_\theta f(s',\theta),\]
and
\begin{align}
  \nabla_\theta L_{nn}(\theta)&=\sum_{(s,a)\in\zeta}(b*\sum_{s'|s,a}P_{ss'}^a\nabla_\theta f(s',\theta)\nonumber\\
  &-b*\sum_{\hat{a}\in
  A}P((s,\hat{a})|r(\theta))\sum_{s'|s,a}P_{ss'}^a\nabla_\theta f(s',\theta)),
  \label{equation:loglikelihoodgradient}
\end{align}
where $\nabla_\theta f(s',\theta)$ denotes the gradient of the neural network output with respect to neural
network parameter $\theta=\{w,b\}$.

If the \textit{VR function} $f(s,\theta)$ is linear, the objective function in Equation
\eqref{equation:loglikelihood} is concave, and a global optimum exists. However, a multi-layer
neural network works better to handle the non-linearity in approximation and the high-dimensional
state space data.

A gradient ascent method can be used to learn the parameter $\theta$:
\begin{equation}
  \label{equation:gradientascent}
  \theta=\theta+\alpha*\nabla_\theta L_{nn}(\theta),
\end{equation}
where $\alpha$ is the learning rate.

When the method converges, we can compute the optimal Q function, the optimal value function, and the
reward function based on Equation \eqref{equation:approxrewardvalue}, \eqref{equation:approxQ},
\eqref{equation:approxV}, and \eqref{equation:approxR}. The algorithm under a neural network-based
approximation function is shown in Algorithm \ref{alg:nnapprox}.

This method does not involve solving the MDP problem for each updated parameter $\theta$, and
large-scale state spaces can be easily handled by an approximation function based on a deep neural
network.
\begin{algorithm}[tb]
  \caption{Function Approximation IRL with Neural Network}
  \label{alg:nnapprox}
\begin{algorithmic}[1]
  \STATE Data: {$\zeta,S,A,P,\gamma,b,\alpha$}
  \STATE Result: {optimal value $V[S]$, optimal action value $Q[S,A]$, reward value $R[S]$}
  \STATE create variable $\theta=\{W,b\}$ for a neural network
  \STATE build $f[S,\theta]$ as the output of the neural network
  \STATE build $Q[S,A]$, $V[S]$, and $R[S]$ based on Equation \eqref{equation:approxrewardvalue},
  \eqref{equation:approxQ}, \eqref{equation:approxV}, and \eqref{equation:approxR}.
  \STATE build loglikelihood $L_{nn}[\theta]$ based on $\zeta$ and $Q[S,A]$
  \STATE compute gradient $\nabla_\theta L_{nn}[\theta]$
  \STATE initialize $\theta$
  \WHILE{not converging}
    \STATE $\theta=\theta+\alpha*\nabla_\theta L_{nn}[\theta]$
  \ENDWHILE
  \STATE evaluate {optimal value $V[S]$, optimal action value $Q[S,A]$, reward value $R[S]$}
  \STATE return $R[S]$
\end{algorithmic}
\end{algorithm}

\subsection{Function Approximation with Gaussian Process}
Assuming the \textit{VR function} $f$ is a Gaussian process (GP) parameterized by $\theta$, the
posterior distribution is similar to the distribution in \cite{irl::gaussianirl}:
\begin{align}
  P(\theta,f_u|S_u,\zeta)
  &\propto P(\zeta,f_u,\theta|S_u)\\\nonumber
  &= \int_{f_S}\underbrace{P(\zeta|f_S)}_\text{IRL}\underbrace{P(f_S|f_u,\theta,S_u)}_\text{GP
  posterior}df_S\underbrace{P(f_u,\theta|S_u)}_\text{GP prior},
\end{align}
where $S_u$ denotes a set of supporting states for sparse Gaussian approximation
\cite{irl::sparsegaussian}, $f_u$ denotes the \textit{VR values} of $S_u$, $f_S$ denotes the
\textit{VR values} of the whole set of states, and $\theta$ denotes the parameter of the Gaussian
process.

Without a closed-form integration, we use the mean function of the Gaussian posterior as the
\textit{VR value}:
\begin{align}
  P(\zeta,f_u,\theta|S_u)=P(\zeta|\bar{f_S})P(f_u,\theta|S_u),
  \label{equation:gpfairl}
\end{align}
where $\bar{f_S}$ denotes the mean function.

Given a kernel function $k(x_i,x_j,\theta)$, the log-likelihood function is given as:
\begin{align}
  &L_{gp}(\theta,f_u)\label{eq:gausslikelihood}\\&=\log P(\zeta|\bar{f_S})+\log P(f_u,\theta|S_u)\\
               &=b*\sum_{(s,a)\in\zeta}(\sum_{s'|s,a}P_{ss'}^a\bar{f(s')}
  -\log{\sum_{\hat{a}\in A}\exp{\sum_{s'|s,\hat{a}}P_{ss'}^{\hat{a}}\bar{f(s'))}}}\label{eq:irlterm}\\
  &-\frac{f_u^TK_{S_u,S_u}^{-1}f_u}{2}-\frac{\log|K_{S_u,S_u}|}{2}-\frac{n\log
  2\pi}{2}\label{eq:gaussterm}
\\  &+\log P(\theta),\label{eq:priorterm}
\end{align}
where $K$ denotes the covariance matrix computed with the kernel function,
$\bar{f(s)}=K_{s,S_u}^TK_{S_u,S_u}^{-1}f_u$ denotes the \textit{VR value} with the mean function
$\bar{f_S}$, expression \eqref{eq:irlterm} is the IRL likelihood, expression \eqref{eq:gaussterm} is
the Gaussian prior likelihood, and expression \eqref{eq:priorterm} is the kernel parameter prior.

The parameters $\theta,f_u$ can be similarly learned with gradient methods. It has similar
properties with neural net-based approach, and the full algorithm is shown in Algorithm
\ref{alg:gpapprox}.
\begin{algorithm}[tb]
  \caption{Function Approximation IRL with Gaussian Process}
  \label{alg:gpapprox}
\begin{algorithmic}[1]
  \STATE Data: {$\zeta,S,A,P,\gamma,b,\alpha$}
  \STATE Result: {optimal value $V[S]$, optimal action value $Q[S,A]$, reward value $R[S]$}
  \STATE create variable $\theta$ for a kernel function and $f_u$ for supporting points
  \STATE compute $\bar{f(s,\theta,f_u)}=K_{s,S_u}^TK_{S_u,S_u}^{-1}f_u$
  \STATE build $Q[S,A]$, $V[S]$, and $R[S]$ based on Equation \eqref{equation:approxrewardvalue},
  \eqref{equation:approxQ}, \eqref{equation:approxV}, and \eqref{equation:approxR}.
  \STATE build loglikelihood $L_{gp}[\theta,f_u]$ based on Equation \eqref{eq:gausslikelihood}.
  \STATE compute gradient $\nabla_{\theta,f_u} L_{gp}[\theta,f_u]$
  \STATE initialize $\theta,f_u$
  \WHILE{not converging}
    \STATE $[\theta,f_u]=[\theta,f_u]+\alpha*\nabla_{\theta,f_u} L_{gp}[\theta,f_u]$
  \ENDWHILE
  \STATE evaluate {optimal value $V[S]$, optimal action value $Q[S,A]$, reward value $R[S]$}
  \STATE return $R[S]$
\end{algorithmic}
\end{algorithm}

\section{Experiments}
\label{irl::experiments}
We use a simulated environment to compare the proposed methods with existing methods and demonstrate
the accuracy and scalability of the proposed solution, then we show how the function approximation
framework can extend existing methods to large state spaces.  In the end, we apply the proposed
method to a clinical application.
\subsection{Simulated Environment}
\begin{figure}
  \centering
 \includegraphics[width=0.2\textwidth]{objectworld.eps}
  \caption{An example of a reward table for one objectworld mdp on a $10\times 10$ grid: it depends
  on randomly placed objects.}
  \label{fig:objectworld}
\end{figure}

The simulated environment is an objectworld mdp \cite{irl::gaussianirl}. It is a $N*N$ grid, but
with a set of objects randomly placed on the grid. Each object has an inner color and an outer
color, selected from a set of possible colors, $C$. The reward of a state is positive if it is
within 3 cells of outer color $C1$ and 2 cells of outer color $C2$, negative if it is within 3 cells
of outer color $C1$, and zero otherwise. Other colors are irrelevant to the ground truth reward. One
example of the reward values is shown in Figure \ref{fig:objectworld}. In this work, we place two
random objects on a $5*5$ grid, and the feature of a state describes its discrete distance to each
inter color and outer color in $C$.

We evaluate the proposed method in three aspects. First, we compare its accuracy in reward learning
with other methods. We generate different sets of trajectory samples, and implement the
maximum-entropy method in \cite{irl::maxentropy}, deep inverse reinforcement learning method in
\cite{irl::deepirl}, and Bellman Gradient Iteration approaches \cite{irl::BGI}. The \textit{VR
function} based on a neural network has five-layers, where the number of nodes in the first four
layers equals to the feature dimensions, and the last layer outputs a single value as the summation
of the reward and the optimal value. The \textit{VR function} based on a Gaussian process uses an
automatic relevance detection (ARD) kernel \cite{irl::gaussianml} and an uninformed prior, and the
supporting points are randomly picked. The accuracy is defined as the correlation coefficient
between the ground truth reward value and the learned reward value.

The result is shown in Figure \ref{fig:objectaccuracy}. The accuracy is not monotonously increasing
as the number of sample grows. The reason is that a function approximator based on a large neural
network will overfit the observed trajectory, which may not reflect the true reward function
perfectly. During reward learning, we observe that as the loglikelihood increases, the accuracy of
the updated reward function reaches the maximum after a certain number of iterations, and then
decreases to a stable value. A possible solution to this problem is early-stopping during reward
learning. For a function approximator with Gaussian process, the supporting set is important,
although an universal solution is unavailable.
\begin{figure}
  \centering
  \includegraphics[width=0.4\textwidth]{accuracy.eps}
  \caption{Accuracy comparison with different numbers of observations: "maxent" denotes maximum
  entropy method; "deep maxent" denotes the deep inverse reinforcement learning approach, "pnorm
  irl" and "gsoft irl" denote Bellman Gradient Iteration method; "fairl with nn" denotes the
  function approximation inverse reinforcement learning with a neural network; "fairl with gp"
  denotes the function approximation inverse reinforcement learning with a Gaussian process.}
  \label{fig:objectaccuracy}
\end{figure}

Second, we evaluate the scalability of the proposed method. Since all these methods involve gradient
method, we choose different numbers of states, ranging from 25 to 9025, and compute the time for
one iteration of gradient ascent under each state size with each method. "Maxent" and "BGI" are
implemented with a mix of Python and C programming language; "DeepMaxent" is implemented with
Theano, and "FAIRL" is implemented with Tensorflow. They all have C programming language in the
backend and Python in the forend.

\begin{table}
  \caption{The computation time (second) of one iteration of gradient method under different number
  of states with different methods: "Maxent" denotes maximum entropy method, "DeepMaxent" denotes
  the deep inverse reinforcement learning approach, "BGI" denotes Bellman Gradient Iteration method,
  and "FAIRLNN" and "FAIRLGP" denote the function approximation inverse reinforcement learning.}
  \label{tab:time}
  \begin{tabular}{|c|c|c|c|c|c|}
    \hline
    States (\#)&Maxent&DeepMaxent&BGI&FAIRLNN&FAIRLGP\\
    \hline
      25&0.017&0.012&0.0313&0.197&0.331\\
    \hline
      225& 1.831&0.178& 2.031& 0.397&0.721\\
    \hline
      625& 24.151&0.95& 20.963& 0.724&1.317\\
     \hline
      1225& 133.839&3.158& 102.460& 0.921&2.163\\
     \hline
      2025& 474.907&8.119& 352.007& 0.776&2.332\\
      \hline
      3025& 1319.365&20.253& 1061.147& 0.762&3.723\\
      \hline
      4225& 3030.723&59.279& 2630.309& 2.468&4.459\\
      \hline
      5625& 6197.718&101.434& 5228.343& 2.831&6.495\\
      \hline
      7225& 12234.417&229.752& 10147.628& 2.217&9.316\\
      \hline
      9025& 20941.9&10466.784& 16345.874& 3.347&12.372\\
    \hline
  \end{tabular}
\end{table}

The result is shown in Table \ref{tab:time}. Even though the computation time may be affected by
different implementations, it still shows that the  proposed method is significantly better than
the alternatives in scalability, and in practice, it can be further improved by paralleling the
computation of the reward function, the value function, and the Q function from the function
approximator. Besides, the Gaussian process-based method requires more time than the neural net,
because of the matrix inverse operations.

Third, we demonstrate how the proposed framework extends existing methods to large-scale state
spaces.  We increase the objectworld to a $80*80$ grid, with 10 objects in 5 colors, and generate
a large sample set with size ranging from 16000 to 128000 at an interval of 16000. Then we show the
accuracy and computation time of inverse reinforcement learning with different combinations of
Bellman backup operators and motion models. The combinations include LogSumExp as Bellman backup
operator with a motion model based on the reward value \cite{irl::maxentropy} and three Bellman
backup operators ($max$, $pnorm$, $gsoft$) with a motion model based on the Q values. We do not use
even larger state spaces because the generation of trajectories from the ground truth reward function
requires a computation-intensive and memory-intensive reinforcement learning step in larger state
spaces. A three-layer neural network is adopted for function approximation, implemented with
Tensorflow on NVIDIA GTX 1080. The training is done with batch sizes 400, learning rate 0.001, and
20 training epochs are ran. The accuracy is shown in Figure \ref{fig:extendaccuracy}.  The
computation time for one training epoch is shown in Figure \ref{fig:extendtime}.
\begin{figure}
  \centering
  \includegraphics[width=0.4\textwidth]{largeaccuracy.eps}
 \caption{Reward learning accuracy of existing methods in large state spaces: "LogSumExp", "Max",
  "PNorm", and "GSoft" are the Bellman backup operators; "Reward" and "QValues" are the types of
  motion models; different combinations of extended methods are plotted. The accuracy is measured as
  the correlation between the ground truth and the recovered reward.}
  \label{fig:extendaccuracy}
\end{figure}
\begin{figure}
  \centering
  \includegraphics[width=0.4\textwidth]{largetime.eps}
 \caption{Computation time for one training epoch of existing methods in large state spaces:
  "LogSumExp", "Max", "PNorm", and "GSoft" are the Bellman backup operators; "Reward" and "QValues"
  are the types of motion models; different combinations of extended methods are plotted.}
  \label{fig:extendtime}
\end{figure}

The results show that the proposed method achieves accuracy and efficiency simultaneously. In
practice, multi-start strategy may be adopted to avoid local optimum.
\subsection{Clinical Experiment}
In the clinic, a patient with spinal cord injuries sits on a box, with a force sensor, capturing the
center-of-pressure (COP) of the patient during movement. Each experiment is composed of two
sessions, one without transcutaneous stimulation and one with stimulation. The electrodes
configuration and stimulation signal pattern are manually selected by the clinician.

In each session, the physician gives eight (or four) directions for the patient to follow, including
left, forward left, forward, forward right, right, right backward, backward, backward left, and the
patient moves continuously to follow the instruction. The physician observes the patient's behaviors
and decides the moment to change the instruction.

Six experiments are done, each with two sessions. The COP trajectories in Figure \ref{fig:patient1}
denote the case with four directional instructions; Figure \ref{fig:patient2}, \ref{fig:patient3},
\ref{fig:patient4}, \ref{fig:patient5}, and \ref{fig:patient6} denote the sessions with eight
directional instructions.
\begin{figure}
  \centering
  \includegraphics[width=0.4\textwidth]{motion5.eps}
  \caption{Patient 1 under four directional instructions: "unstimulated motion" means that the
  patient moves without transcutaneous stimulations, while "stimulated motion" represents the motion
  under stimulations.}
  \label{fig:patient1}
\end{figure}

\begin{figure}
  \centering
  \includegraphics[width=0.4\textwidth]{motion0.eps}
  \caption{Patient 2 under eight directional instructions: "unstimulated motion" means that the
  patient moves without transcutaneous stimulations, while "stimulated motion" represents the motion
  under stimulations.}
  \label{fig:patient2}
\end{figure}
\begin{figure}
  \centering
  \includegraphics[width=0.4\textwidth]{motion1.eps}
  \caption{Patient 3 under eight directional instructions: "unstimulated motion" means that the
  patient moves without transcutaneous stimulations, while "stimulated motion" represents the motion
  under stimulations.}
  \label{fig:patient3}
\end{figure}
\begin{figure}
  \centering
  \includegraphics[width=0.4\textwidth]{motion2.eps}
  \caption{Patient 4 under eight directional instructions: "unstimulated motion" means that the
  patient moves without transcutaneous stimulations, while "stimulated motion" represents the motion
  under stimulations.}
  \label{fig:patient4}
\end{figure}
\begin{figure}
  \centering
  \includegraphics[width=0.4\textwidth]{motion3.eps}
  \caption{Patient 5 under eight directional instructions: "unstimulated motion" means that the
  patient moves without transcutaneous stimulations, while "stimulated motion" represents the motion
  under stimulations.}
  \label{fig:patient5}
\end{figure}
\begin{figure}
  \centering
  \includegraphics[width=0.4\textwidth]{motion4.eps}
  \caption{Patient 6 under eight directional instructions: "unstimulated motion" means that the
  patient moves without transcutaneous stimulations, while "stimulated motion" represents the motion
  under stimulations.}
  \label{fig:patient6}
\end{figure}

The COP sensory data from each session is discretized on a $100\times 100$ grid, which is fine
enough to capture the patient's small movements. The problem is formulated into a MDP, where each
state captures the patient's discretized location and velocity, and the set of actions changes the
velocity into eight possible directions. The velocity is represented with a two-dimensional vector
showing eight possible velocity directions. Thus the problem has 80000 states and 8 actions, and the
transition model assumes that each action will lead to one state with probability one.
\begin{table*}
  \centering
  \caption{Evaluation of the learned rewards: "forward" etc. denote the instructed direction; "1u"
  denote the patient id "1", with "u" denoting unstimulated session and "s" denoting stimulated
  sessions. The table shows the correlation coefficient between the ideal reward and the recovered
  reward.}
  \label{tab:feature1}
 \begin{tabular}{|c|c|c|c|c|c|c|c|c|c|}
      \hline
     &forward&backward&left&right&top left&top right&bottom left&bottom right&origin\\\hline
1u&-0.352172&-0.981877&-0.511908&-0.399777&&&&&-0.0365778\\\hline
1s&-0.36437&-0.999993&-0.14757&-0.321745&&&&&0.154132\\\hline
2u&-0.459214&-0.154868&0.134229&0.181629&0.123853&0.677538&-0.398259&0.264739&-0.206476\\\hline
2s&-0.115516&-0.127179&0.569024&0.164638&0.360013&0.341521&0.0817681&0.134049&-0.00986036\\\hline
3u&0.533031&0.0364088&0.128325&-0.729293&0.397182&0.155565&-0.48818&-0.293617&-0.176923\\\hline
3s&-0.340902&-0.091139&0.344993&0.0557266&0.162783&0.740827&-0.0897398&-0.00674047&-0.414462\\\hline
4u&0.099563&-0.0965766&0.145509&-0.912844&0.250434&-0.299531&0.577489&0.134106&-0.151334\\\hline
4s&-0.258762&-0.019275&-0.263354&0.549305&0.0910128&0.755755&-0.225137&0.289126&-0.216737\\\hline
5u&0.287442&0.0859648&-0.368503&0.504589&-0.297166&0.401829&0.0583192&-0.23662&-0.0762139\\\hline
5s&-0.350374&-0.0969275&0.538291&-0.617767&-0.00442265&0.0923481&0.115864&-0.576655&-0.0108339\\\hline
6u&0.205348&0.302459&0.550447&0.0549231&-0.348898&0.420478&0.378317&0.56191&0.145699\\\hline
6s&0.105335&-0.155296&0.0193898&-0.283895&-0.0577008&0.220243&-0.31611&-0.296682&-0.0753326\\\hline
  \end{tabular}
\end{table*}

To learn the reward function from the observed trajectories based on the formulated MDP, we use the
coordinate and velocity direction of each grid as the feature, and learn the reward function
parameter from each set of data. The function approximator is a neural network with three hidden
layers and $[100,50,25]$ nodes.

We only test the proposed method with a neural-net function approximator, because it will take
prohibitive amount of time to learn the reward function with other methods, and the GP approach
relies on the set of supporting points. Assuming it takes only 100 iterations to converge, the
proposed method takes about one minute while others run for two to four weeks, and in practice, it
may take more iterations to converge.

To compare the reward function with and without stimulations, we adopt the same initial parameter
during reward function learning, and run both learning process with 10000 iterations with learning
rate 0.00001.

Given the learned reward function, we score the patient's recovery with the correlation coefficient
between the recovered rewards and the ideal rewards under the clinicians' instructions of the states
visited by the patient. The ideal reward for each state is the cosine similarity between the state's
velocity vector and the instructed direction.

The result is shown in Table \ref{tab:feature1}. It shows that the patient's ability to follow the
instructions is affected by the stimulations, but whether it is improved or not varies among
different directions. The clinical interpretations will be done by physicians.

\section{Conclusions}
\label{irl::conclusions}
This work deals with the problem of inverse reinforcement learning in large state spaces, and solves
the problem with a function approximation method that avoids solving reinforcement learning problems
during reward learning. The simulated experiment shows that the proposed method is more accurate and
scalable than existing methods, and can extends existing methods to high-dimensional spaces. A
clinical application of the proposed method is presented.

In future work, we will remove the requirement of a-priori known transition function by combining an
 environment model learning process into the function approximation framework.


\section{Introduction}
There are two frequently appeared structured eigenvalue problems in
(relativistic) electronic structure
methods, which can be written into a unified way as
\begin{eqnarray}
\mathbf{M}_s\mathbf{z}&=&\mathbf{z}\,\omega,\quad s=\pm1,\nonumber\\
\mathbf{M}_s&=&\left[\begin{array}{cc}
\mathbf{A} & \mathbf{B} \\
-\mathbf{B}^* & s\mathbf{A}^* \\
\end{array}\right],\quad
\mathbf{z}=\left[\begin{array}{cc}
\mathbf{x} \\
\mathbf{y} \\
\end{array}\right],
\nonumber\\
\mathbf{A}^\dagger&=&\mathbf{A},\quad \mathbf{B}^T = -s \mathbf{B},\label{Mdef}
\end{eqnarray}
where $\mathbf{M}_s\in\mathbb{C}^{2n\times2n}$, $\mathbf{z}\in \mathbb{C}^{2n}$,
$\mathbf{A}\in\mathbb{C}^{n\times n}$ is Hermitian, and
$\mathbf{B}\in\mathbb{C}^{n\times n}$ is antisymmetric for $s=+1$ or
symmetric for $s=-1$.

The $s=+1$ case appears in matrix representations
of Hermitian operators, such as the Fock operator
of closed-shell systems, in a Kramers paired basis\cite{dyall2007introduction}.
Another example is the equation-of-motion method\cite{rowe1968equations}
for ionization and electron attachment
from a closed-shell reference, where the excitation operator
$O_n^\dagger$ is expanded in a paired basis $\{a_p^\dagger,a_p\}$,
viz., $O_n^\dagger=\sum_{p}(a_p^\dagger X_p-a_q Y_p)$.
The Hermitian matrix $\mathbf{M}_+$ is usually referred as quaternion matrix\cite{rosch1983time,bunse1989quaternion,saue1999quaternion},
since it can be rewritten as
\begin{eqnarray}
\mathbf{M}_+=\mathbf{I}_2\otimes\mathbf{A}_{\mathrm{R}}+i\sigma_z\otimes \mathbf{A}_{\mathrm{I}}
+i\sigma_y\otimes \mathbf{B}_{\mathrm{R}}+i\sigma_x\otimes \mathbf{B}_{\mathrm{I}}.
\end{eqnarray}
where $\{\mathbf{I}_2,i\sigma_z,i\sigma_y,i\sigma_x\}$ is isomorphic to
the set of quaternion units $\{1,\breve{i},\breve{j},\breve{k}\}$, where
$\mathbf{A}_{\mathrm{R}}$ (or $\mathbf{A}_{\mathrm{I}}$) represents the real (or imaginary) part of $\mathbf{A}$.
The corresponding eigenvalue problem is well-studied,
and several efficient algorithms have been presented\cite{rosch1983time,dongarra1984eigenvalue,bunse1989quaternion,saue1999quaternion,shiozaki2017efficient},
based on the generalization of established algorithms for complex matrices to quaternion
algebra or the use of unitary symplectic transformations.

The $s=-1$ case appears in the linear response problem\cite{olsen1985linear,olsen1988solution,sasagane1993higher,christiansen1998response,
casida1995response,gao2004time,bast2009relativistic,egidi2016direct,liu2018relativistic,komorovsky2019four}
for excitation energies of Hartree-Fock (HF), density functional theory (DFT), multi-configurational self-consistent field (MCSCF), or the Bethe-Salpeter equation (BSE)\cite{salpeter1951relativistic}.
Compared with the $s=1$ case, the linear response eigenvalue problem is more challenging since
$\mathbf{M}_-$ is non-Hermitian. In practice, we are mostly interested in the real eigenvalues,
which correspond to physical excitation energies. Unfortunately, the
condition for the existence of all \emph{real} eigenvalues is only partially understood.
In the nonrelativistic\cite{stratmann1998efficient} and some relativistic cases\cite{liu2018relativistic},
where $\mathbf{M}_-$ becomes real, the eigenvalue problem \eqref{Mdef} is equivalent to
the reduced problem
\begin{eqnarray}
(\mathbf{A}-\mathbf{B})(\mathbf{A}+\mathbf{B})(\mathbf{x}+\mathbf{y})=(\mathbf{x}+\mathbf{y})\omega^2,\label{AmBApB}
\end{eqnarray}
or
\begin{eqnarray}
(\mathbf{A}+\mathbf{B})(\mathbf{A}-\mathbf{B})(\mathbf{x}-\mathbf{y})=(\mathbf{x}-\mathbf{y})\omega^2.\label{ApBAmB}
\end{eqnarray}
Thus, the eigenvalues of the original problem are all real if and only if
the eigenvalues of the reduced matrix $(\mathbf{A}-\mathbf{B})(\mathbf{A}+\mathbf{B})$ (or its transpose $(\mathbf{A}+\mathbf{B})(\mathbf{A}-\mathbf{B})$) are all nonnegative,
i.e., $\omega^2\ge 0$. Besides, the use of Eq. \eqref{AmBApB} or \eqref{ApBAmB} also
reduces the cost for diagonalization compared with that for Eq. \eqref{Mdef}.
If $\mathbf{M}_-$ is complex as in the relativistic case in general,
such reduction is not possible. Assuming the positive definiteness of the so-called
electronic Hessian,
\begin{eqnarray}
\left[\begin{array}{cc}
\mathbf{A} & \mathbf{B} \\
\mathbf{B}^* & \mathbf{A}^* \\
\end{array}\right]\succ0,
\end{eqnarray}
one can show that all eigenvalues of $\mathbf{M}_-$ are real\cite{shao2015properties,shao2016structure,benner2018some}.
However, this condition is only a sufficient condition.
In the real case, this implies $\mathbf{A}-\mathbf{B}\succ0$ and $\mathbf{A}+\mathbf{B}\succ0$.
Another sufficient condition is $\mathbf{B}=0$, in which case
$\mathbf{M}_-$ is block-diagonal and all its eigenvalues are real,
even though there can be negative eigenvalues in $\mathbf{A}$.
The situation, where the electronic Hessian is not positive definite but
$\mathbf{M}_-$ still have all real eigenvalues, is also practically meaningful.
It happens in using an excited state as reference,
a trick that has been commonly used in time-dependent DFT to treat
excited states of systems with a spatially degenerate ground state\cite{seth2005calculation}.
Besides, it also happens in scanning the potential energy curves,
where a curve crossing is encountered between the ground and an excited
state with a different symmetry\cite{li2011spin}.
In such cases, the negative eigenvalues correspond to de-excitations to a lower
energy state.

Due to the similarity in mathematical structures for the $s=1$ and
$s=-1$ cases, in this work we examine these two problems from a unified perspective.
We first identify the Lie group structures for their eigenvectors (see Sec. \ref{sec:liegroup}).
Then, by using the same reduction algorithm for the $s=1$ case (see Sec. \ref{sec:reduction}),
we provide a condition as a generalization of the real case based
on the reduced problems (Eqs. \eqref{AmBApB} and \eqref{ApBAmB})
to characterize the different scenarios, where the eigenvalues of the
complex linear response problem are real, purely imaginary, or complex (see Sec. \ref{sec:condition}).
Some typical examples are provided in Sec. \ref{sec:examples} to illustrate the complexity of
the eigenvalue problem \eqref{Mdef} in the $s=-1$ case.

\section{Lie group structures of the eigensystems}\label{sec:liegroup}
The matrices in Eq. \eqref{Mdef} with $s=1$ and $s=-1$ are closely
related with the skew-Hamiltonian matrix $\mathbf{W}$ and Hamiltonian matrix $\mathbf{H}$
in real field\cite{kressner2005}, respectively,
\begin{eqnarray}
\mathbf{W}&=&\left[\begin{array}{cc}
\mathbf{W}_{11} & \mathbf{W}_{12} \\
\mathbf{W}_{21} & \mathbf{W}_{11}^T \\
\end{array}\right],\; \mathbf{W}_{12}=-\mathbf{W}_{12}^T,\; \mathbf{W}_{21}=-\mathbf{W}_{21}^T,\label{SkewH}\\
\mathbf{H}&=&\left[\begin{array}{cc}
\mathbf{H}_{11} & \mathbf{H}_{12} \\
\mathbf{H}_{21} & -\mathbf{H}_{11}^T \\
\end{array}\right],\; \mathbf{H}_{12}=\mathbf{H}_{12}^T,\; \mathbf{H}_{21}=\mathbf{H}_{21}^T.\label{Ham}
\end{eqnarray}
The identification of the Hamiltonian structure for the linear
response problem was presented in Ref. \onlinecite{list2014identifying}
for the real matrix case. It can be shown that the eigenvalues of $\mathbf{W}$ appear
in pairs $\{\omega,\omega\}$, while the eigenvalues of $\mathbf{H}$ appear in pairs $\{\omega,-\omega\}$\cite{kressner2005}.
The same results also hold for complex matrices. Besides,
the additional relations with complex conjugation in Eq. \eqref{Mdef} compared
with Eqs. \eqref{SkewH} and \eqref{Ham}, viz.,
$\mathbf{W}_{21}=-\mathbf{W}_{12}^*=-\mathbf{B}^*$,
$\mathbf{W}_{11}^T=\mathbf{W}_{11}^*=\mathbf{A}^*$,
$\mathbf{H}_{21}=-\mathbf{H}_{12}^*=-\mathbf{B}^*$,
and
$\mathbf{H}_{11}^T=\mathbf{H}_{11}^*=\mathbf{A}^*$,
lead to further structures on eigenvalues and eigenvectors,
\begin{eqnarray}
\mathbf{Z}_s=
\left[\begin{array}{cc}
\mathbf{X} & -s\mathbf{Y}^* \\
\mathbf{Y} &   \mathbf{X}^* \\
\end{array}\right],\quad
\boldsymbol{\omega}_s=
\left[\begin{array}{cc}
\boldsymbol{\omega} &  \mathbf{0} \\
\mathbf{0} & s\boldsymbol{\omega}^* \\
\end{array}\right].\label{eigenvectors}
\end{eqnarray}
Thus, the symmetry relationships among eigenvalues of Eq. \eqref{Mdef} can be deducted.
For $s=1$, the eigenvalues are real doubly degenerate $\{\omega,\omega=\omega^*\}$,
which is a reflection of the time reversal symmetry.
For $s=-1$, the quadruple of eigenvalues $\{\omega, -\omega, \omega^*, -\omega^*\}$ appears.
If $\omega=\omega^*$ (or $-\omega^*$) is real (purely imaginary),
then the quadruple reduces to the pair $\{\omega,-\omega\}$.
Note that the pair structure \eqref{eigenvectors} always holds for $s=1$,
since $(\mathbf{X},\mathbf{Y})$ and $(-\mathbf{Y}^*,\mathbf{X}^*)$ are orthogonal. However, for $s=-1$,
the situation becomes more complicated in the
degenerate case $\omega=-\omega^*$, where the pair structure
of eigenvectors does not necessarily hold. The following discussion in this section only
works for the $s=-1$ case where all the eigenvectors have the pair structure \eqref{eigenvectors},
while the algorithm presented in Sec. \ref{sec:condition} does not have this assumption.

While most of the previous studies focused on the paired structure \eqref{eigenvectors}
for a given matrix $\mathbf{M}_s$, we note that if the set of matrices with
the same form as the eigenvectors $\mathbf{Z}_s$ \eqref{eigenvectors} is considered,
along with the commonly applied normalization conditions,
\begin{eqnarray}
\mathbf{Z}^\dagger_s \mathbf{N}_s\mathbf{Z}_s =\mathbf{N}_s,\quad
\mathbf{N}_s=\left[\begin{array}{cc}
\mathbf{I}_n & \mathbf{0} \\
\mathbf{0} & s\mathbf{I}_n
\end{array}\right],\label{normal}
\end{eqnarray}
these matrices actually form matrix groups, viz.,
\begin{eqnarray}
\mathcal{G}_s =\{\mathbf{Z}_s:\,
\mathbf{Z}_s=\left[\begin{array}{cc}
\mathbf{X} & -s\mathbf{Y}^* \\
\mathbf{Y} &   \mathbf{X}^* \\
\end{array}\right],\,\mathbf{Z}^\dagger_s \mathbf{N}_s\mathbf{Z}_s =\mathbf{N}_s\}.\label{Gs}
\end{eqnarray}
This can be simply verified by following the definition of groups as follows:

(1) This set is closed under multiplication of two matrices, since
\begin{eqnarray}
\mathbf{Z}_{s,1}\mathbf{Z}_{s,2}
&=&
\left[\begin{array}{cc}
\mathbf{X}_1 & -s\mathbf{Y}^*_1 \\
\mathbf{Y}_1 &   \mathbf{X}^*_1 \\
\end{array}\right]
\left[\begin{array}{cc}
\mathbf{X}_2 & -s\mathbf{Y}^*_2 \\
\mathbf{Y}_2 &   \mathbf{X}^*_2 \\
\end{array}\right]\nonumber\\
&=&
\left[\begin{array}{cc}
\mathbf{X}_1\mathbf{X}_2-s\mathbf{Y}^*_1\mathbf{Y}_2 & -s(\mathbf{X}_1\mathbf{Y}^*_2+\mathbf{Y}^*_1\mathbf{X}^*_2) \\
\mathbf{X}^*_1\mathbf{Y}_2+\mathbf{Y}_1\mathbf{X}_2 &  \mathbf{X}^*_1\mathbf{X}^*_2-s\mathbf{Y}_1\mathbf{Y}^*_2 \\
\end{array}\right]\nonumber\\
&\triangleq&
\left[\begin{array}{cc}
\mathcal{X} & -s\mathcal{Y}^* \\
\mathcal{Y} &   \mathcal{X}^* \\
\end{array}\right],
\end{eqnarray}
and $(\mathbf{Z}_{s,1}\mathbf{Z}_{s,2})^\dagger\mathbf{N}_s(\mathbf{Z}_{s,1}\mathbf{Z}_{s,2})=\mathbf{N}_s$,
that is $\mathbf{Z}_{s,1}\mathbf{Z}_{s,2} \in \mathcal{G}_s$.

(2) The identity element is just $\mathbf{I}_{2n}$.

(3) The inverse of any element $\mathbf{Z}_s$ exists, since $\mathbf{Z}_s$ satisfies
the normalization condition \eqref{normal},
\begin{eqnarray}
\mathbf{Z}_s^{-1}
=\mathbf{N}_s\mathbf{Z}_s^\dagger\mathbf{N}_s
=\left[\begin{array}{cc}
\mathbf{X}^\dagger & s\mathbf{Y}^\dagger \\
-\mathbf{Y}^T & \mathbf{X}^T
\end{array}\right]\in \mathcal{G}_s.\label{zinv}
\end{eqnarray}

In fact, the groups \eqref{Gs} are just the
unitary symplectic Lie groups, viz., $\mathrm{USp}(2n)=\mathrm{U}(2n)\cap \mathrm{Sp}(2n,\mathbb{C})$ for $s=+1$,
and $\mathrm{USp}(n,n)=\mathrm{U}(n,n)\cap \mathrm{Sp}(2n,\mathbb{C})$ for $s=-1$.
Here, $\mathrm{U}(p,q)$ represents the unitary group with signature $(p,q)$, i.e.,
\begin{eqnarray}
\mathbf{U}(p,q)=\{\mathbf{M}:\;\mathbf{M}^\dagger \mathbf{I}_{p,q}\mathbf{M} = \mathbf{I}_{p,q}\},\;\;
\mathbf{I}_{p,q}=\left[
\begin{array}{cc}
\mathbf{I}_{p} & \mathbf{0} \\
\mathbf{0} & -\mathbf{I}_{q} \\
\end{array}\right],\label{Upq}
\end{eqnarray}
and $\mathrm{Sp}(2n,\mathbb{C})$ represents the complex symplectic
group,
\begin{eqnarray}
\mathrm{Sp}(2n,\mathbb{C})=\{\mathbf{M}:\;\mathbf{M}^T \mathbf{J}\mathbf{M} = \mathbf{J}\},\;\;
\mathbf{J}=\left[\begin{array}{cc}
\mathbf{0} & \mathbf{I}_{n} \\
-\mathbf{I}_{n} & \mathbf{0}\\
\end{array}\right].\label{Sp2n}
\end{eqnarray}
Such equivalence can be established by realizing that
a combination of  the conditions in Eqs. \eqref{Upq} and \eqref{Sp2n} leads to
a condition
\begin{eqnarray}
\mathbf{J}_s\mathbf{M}^*=\mathbf{M}\mathbf{J}_s,\quad
\mathbf{J}_s=\left[\begin{array}{cc}
\mathbf{0} & \mathbf{I}_{n} \\
-s\mathbf{I}_{n} & \mathbf{0}\\
\end{array}\right],\label{JMsMJ}
\end{eqnarray}
which implies the block structures in Eq. \eqref{Gs}
for the $2n$-by-$2n$ matrix $\mathbf{M}$. Note in passing that
the case $s=1$ for Eq. \eqref{JMsMJ} reveals the
underlying commutation between an operator
and the time reversal operator.

The identification of Lie group structures for $\mathcal{G}_s$
has important consequences. In particular, it simplifies the
design of structure-preserving algorithms. For instance,
for the case $s=1$, the Lie algebra $\mathfrak{usp}(2n)$
corresponding to the Lie group $\mathrm{USp}(2n)$ is
\begin{eqnarray}
\mathfrak{usp}(2n)=\{\mathbf{K}:\;
\mathbf{K}^\dagger =-\mathbf{K},\;
\mathbf{J}\mathbf{K}^*=\mathbf{K}\mathbf{J}\},\label{usp}
\end{eqnarray}
where the matrix $\mathbf{K}$ can be written more explicitly as
\begin{eqnarray}
\mathbf{K}=\left[\begin{array}{cc}
\mathbf{K}_{11} & -\mathbf{K}_{21}^* \\
\mathbf{K}_{21} & \mathbf{K}_{11}^* \\
\end{array}\right],\;
\mathbf{K}_{11}^\dagger = -\mathbf{K}_{11},\;
\mathbf{K}_{21}^T=\mathbf{K}_{21}.
\end{eqnarray}
This applies to the construction of time reversal adapted
basis operators\cite{aucar1995operator}.
Furthermore, the exponential map $\exp(\mathbf{K})$ can be used to
transform one Kramers paired basis into another Kramers
paired basis, which was previously used in the Kramers-restricted MCSCF\cite{fleig1997spinor}. More generally,
such Lie group \eqref{Gs} and Lie algebra structures for
the case $s=1$ also implies the possibility to apply the numerical
techniques for the optimization on manifolds\cite{absil2009optimization}
to relativistic spinor optimizations while preserving
the Kramers pair structure. Due to the unified framework presented here,
one can expect that the similar techniques can also be applied to the $s=-1$ case.

\section{Reduction algorithm for the $s=1$ case}\label{sec:reduction}
In this section, we will briefly recapitulate the reduction algorithm for the $s=1$ case,
by adapting the techniques developed for the real skew Hamiltonian matrices\cite{paige1981schur,van1984symplectic}
to the complex matrix $\mathbf{M}_+$ \eqref{Mdef}. Such techniques also form basis for developing diagonalization
algorithms for the relativistic Fock matrix\cite{dongarra1984eigenvalue,shiozaki2017efficient}.
The essential idea is to realize that the unitary symplectic transformation
\begin{eqnarray}
\mathbf{G}=\left[\begin{array}{cc}
\mathbf{U} & -\mathbf{V}^* \\
\mathbf{V} & \mathbf{U}^* \\
\end{array}\right]\in\mathrm{USp}(2n),
\end{eqnarray}
when acting on a skew Hamiltonian (complex) matrix, such as
$\mathbf{W}$ \eqref{SkewH} via $\tilde{\mathbf{W}}=\mathbf{G}\mathbf{W}\mathbf{G}^\dagger$, will preserve
the skew-Hamiltonian structure, viz.,
\begin{eqnarray}
\tilde{\mathbf{W}}_{22}=\tilde{\mathbf{W}}_{11}^T,\;\;
\tilde{\mathbf{W}}_{12}^T=-\tilde{\mathbf{W}}_{12},\;\;
\tilde{\mathbf{W}}_{21}^T=-\tilde{\mathbf{W}}_{21}.
\end{eqnarray}
Then, there is a constructive way\cite{paige1981schur} to eliminate the lower-left block of the matrix $\mathbf{W}$ \eqref{SkewH}
by a series of unitary symplectic Householder and Givens transformations,
such that the transformed matrix $\tilde{\mathbf{W}}$ is in the following Paige/Van Loan (PVL) form\cite{kressner2005},
\begin{eqnarray}
\tilde{\mathbf{W}}=\mathbf{G}\mathbf{W}\mathbf{G}^{\dagger}=
\left[\begin{array}{cc}
\tilde{\mathbf{W}}_{11} & \tilde{\mathbf{W}}_{12} \\
\mathbf{0} & \tilde{\mathbf{W}}_{11}^T \\
\end{array}\right].\label{PVLform}
\end{eqnarray}
The crucial point for being able to transform $\mathbf{W}$
into the form \eqref{PVLform} is $\mathbf{W}_{21}=-\mathbf{W}_{21}^T$, and such
property can be preserved during the transformations,
see Ref. \onlinecite{paige1981schur} for details of the transformations.
The computational scaling of such transformation is cubic in the dimension
of the matrix.

Applying this result to $\mathbf{M}_+$ and realizing that the transformed matrix $\tilde{\mathbf{M}}_+$
is still Hermitian, one can conclude that $(\tilde{\mathbf{M}}_+)_{12}=0$ and
$\tilde{\mathbf{M}}_{11}^T=\tilde{\mathbf{M}}_{11}^*$, viz.,
$\tilde{\mathbf{M}}_+$ becomes block-diagonal\cite{shiozaki2017efficient}.
Then, the eigenvalues can be obtained by diagonalizing the Hermitian matrix $\tilde{\mathbf{M}}_{11}$ by
a unitary matrix $\bar{\mathbf{U}}$, which reduces the computational cost compared with the original problem.
The final solution to the original problem can be obtained as
\begin{eqnarray}
\mathbf{Z}_+=\mathbf{G}\left[\begin{array}{cc}
\bar{\mathbf{U}} & \mathbf{0} \\
\mathbf{0} & \bar{\mathbf{U}}^* \\
\end{array}\right]=
\left[\begin{array}{cc}
\mathbf{U}\bar{\mathbf{U}} & -\mathbf{V}^*\bar{\mathbf{U}} \\
\mathbf{V}\bar{\mathbf{U}}^* & \mathbf{U}^*\bar{\mathbf{U}}^* \\
\end{array}\right]\in \mathcal{G}_{+},
\end{eqnarray}
which still preserves the structure \eqref{Gs} due to
the closeness of groups.

\section{Reduction algorithm for the $s=-1$ case}\label{sec:condition}
Due to the non-Hermicity of $\mathbf{M}_-$, a straightforward generalization
of the above reduction algorithm to $\mathbf{M}_-$ using the transformation
in $\mathrm{USp}(n,n)$ does not seem to be possible. Because the validity of the
form \eqref{eigenvectors} depends on the properties of the eigensystem.
In some cases, $\mathbf{M}_-$ cannot be diagonalizable, e.g.,
$\mathbf{M}_-=\left[\begin{array}{cc}
1 & 1 \\
-1 & -1 \\
\end{array}\right]$.
To avoid such difficulty, following the observation for
the Hamiltonian matrix by van Loan\cite{van1984symplectic},
one can find the square matrix $\mathbf{M}_-^2$ is a skew
Hamiltonian matrix,
\begin{eqnarray}
\mathbf{M}^2_-&=&
\left[\begin{array}{cc}
\mathbf{A}^2-\mathbf{B}\B^* & \mathbf{A}\mathbf{B}-\mathbf{B}\mathbf{A}^* \\
-\mathbf{B}^*\mathbf{A}+\mathbf{A}^*\mathbf{B}^* & (\mathbf{A}^*)^2-\mathbf{B}^*\mathbf{B}\\
\end{array}\right]\nonumber\\
&\triangleq&
\left[\begin{array}{cc}
\mathcal{A} & \mathcal{B} \\
\mathcal{B}^* & \mathcal{A}^* \\
\end{array}\right],\label{M2}
\end{eqnarray}
where
\begin{eqnarray}
\mathcal{A} &=& \mathbf{A}^2-\mathbf{B}\B^*,\nonumber\\
\mathcal{B} &=& \mathbf{A}\mathbf{B}-\mathbf{B}\mathbf{A}^*,\nonumber\\
\mathcal{A}^\dagger &=& (\mathbf{A}^\dagger)^2-\mathbf{B}^T\mathbf{B}^\dagger
=\mathbf{A}^2-\mathbf{B}\B^* =\mathcal{A},\nonumber\\
\mathcal{B}^T &=& \mathbf{B}^T\mathbf{A}^T-\mathbf{A}^\dagger\mathbf{B}^T
=\mathbf{B}\mathbf{A}^*-\mathbf{A}\mathbf{B} = -\mathcal{B}.
\end{eqnarray}
Thus, it can be brought into the PVL form \eqref{PVLform} using unitary symplectic transformations
as for $\mathbf{M}_+$. However, it deserves to be pointed out that unlike for the Hermitian
matrix $\mathbf{M}_+$, such transformations will not
preserve the relation between the lower-left and upper-right blocks of $\mathbf{M}_-^2$, such that
while $(\tilde{\mathbf{M}}^2_-)_{21}$ is made zero in the transformed matrix, $(\tilde{\mathbf{M}}^2_-)_{12}$ is nonzero.
But the upper-left block $(\tilde{\mathbf{M}}^2_-)_{11}$ can still be used
to compute $\omega^2$ with reduced cost for diagonalization.

This basically gives a criterion for the different scenarios of eigenvalues of the linear response problem.
However, to make better connection to the conditions \eqref{AmBApB} and \eqref{ApBAmB} for the real case.
We use the following transformation\cite{shao2016structure},
\begin{eqnarray}
\mathbf{Q} = \frac{1}{\sqrt{2}}\left[
\begin{array}{cc}
\mathbf{I}_n & \mathbf{I}_n \\
i\mathbf{I}_n & -i\mathbf{I}_n \\
\end{array}\right]
\end{eqnarray}
which transforms $\mathbf{M}_-$ into a purely imaginary matrix
\begin{eqnarray}
\mathbf{Q}\mathbf{M}_-\mathbf{Q}^\dagger=i\left[
\begin{array}{cc}
(\mathbf{A}+\mathbf{B})_{\mathrm{I}} & -(\mathbf{A}-\mathbf{B})_{\mathrm{R}} \\
(\mathbf{A}+\mathbf{B})_{\mathrm{R}} & (\mathbf{A}-\mathbf{B})_{\mathrm{I}} \\
\end{array}\right].
\end{eqnarray}
Then, the square matrix is transformed into a real skew-Hamiltonian matrix,
\begin{widetext}
\begin{eqnarray}
\mathbf{\Pi} \triangleq \mathbf{Q}\mathbf{M}_-^2\mathbf{Q}^\dagger =
\left[\begin{array}{cc}
(\mathbf{A}-\mathbf{B})_{\mathrm{R}}(\mathbf{A}+\mathbf{B})_{\mathrm{R}}-(\mathbf{A}+\mathbf{B})_{\mathrm{I}}(\mathbf{A}+\mathbf{B})_{\mathrm{I}} &
(\mathbf{A}-\mathbf{B})_{\mathrm{R}}(\mathbf{A}-\mathbf{B})_{\mathrm{I}}+(\mathbf{A}+\mathbf{B})_{\mathrm{I}}(\mathbf{A}-\mathbf{B})_{\mathrm{R}} \\
-(\mathbf{A}+\mathbf{B})_{\mathrm{R}}(\mathbf{A}+\mathbf{B})_{\mathrm{I}}-(\mathbf{A}-\mathbf{B})_{\mathrm{I}}(\mathbf{A}+\mathbf{B})_{\mathrm{R}} &
(\mathbf{A}+\mathbf{B})_{\mathrm{R}}(\mathbf{A}-\mathbf{B})_{\mathrm{R}}-(\mathbf{A}-\mathbf{B})_{\mathrm{I}}(\mathbf{A}-\mathbf{B})_{\mathrm{I}} \\
\end{array}\right].\label{pi}
\end{eqnarray}
\end{widetext}
It is clear that if $\mathbf{M}_-$ is real, then $\mathbf{\Pi}$ becomes diagonal,
with the diagonal blocks being simply $(\mathbf{A}-\mathbf{B})(\mathbf{A}+\mathbf{B})$ and $(\mathbf{A}+\mathbf{B})(\mathbf{A}-\mathbf{B})$, and
the transformed eigenvalue problem becomes equivalent to Eqs. \eqref{AmBApB} and \eqref{ApBAmB}, since
\begin{eqnarray}
\mathbf{Q}\mathbf{z}=\mathbf{Q}
\left[\begin{array}{cc}
\mathbf{x} \\
\mathbf{y} \\
\end{array}\right]
=\frac{1}{\sqrt{2}}
\left[\begin{array}{cc}
\mathbf{x}+\mathbf{y} \\
i(\mathbf{x}-\mathbf{y}) \\
\end{array}\right].
\end{eqnarray}
By applying unitary symplectic transformations to reduce $\tilde{\mathbf{\Pi}}=\mathbf{G}\mathbf{\Pi}\mathbf{G}^\dagger$
into the PVL form \eqref{PVLform}, the real non-Hermitian matrix
$\tilde{\mathbf{\Pi}}_{11}$ can be used to compute eigenvalues of $\mathbf{M}_-^2$.
Suppose the eigenvalues of $\tilde{\mathbf{\Pi}}_{11}$ are denoted by $\lambda$, then
we have the following three cases:

(1) $\lambda=\lambda_{\mathrm{R}}\ge 0$: the pair of real eigenvalues of $\mathbf{M}_-$ is $\{\sqrt{\lambda_{\mathrm{R}}},-\sqrt{\lambda_{\mathrm{R}}}\}$.

(2) $\lambda=\lambda_{\mathrm{R}}<0$: the pair of purely imaginary eigenvalues of $\mathbf{M}_-$ is $\{i\sqrt{-\lambda_{\mathrm{R}}},
-i\sqrt{-\lambda_{\mathrm{R}}}\}$.

(3) $\lambda=\lambda_{\mathrm{R}}+i\lambda_{\mathrm{I}}$ is complex: $\lambda^*=\lambda_{\mathrm{R}}-i\lambda_{\mathrm{I}}$ will also be an eigenvalue of $\tilde{\mathbf{\Pi}}_{11}$, and the quadruple of complex eigenvalues of
$\mathbf{M}_-$ is $\{\omega, -\omega, \omega^*, -\omega^*\}$ with $\omega$ given by
\begin{eqnarray}
\omega = \frac{\zeta}{\sqrt{2}}+i\frac{\lambda_{\mathrm{I}}}{\sqrt{2}\zeta},\quad
\zeta=\sqrt{\lambda_{\mathrm{R}}+\sqrt{\lambda_{\mathrm{R}}^2+\lambda_{\mathrm{I}}^2}}.
\end{eqnarray}

Thus, the goal to characterize the eigenvalues of the complex linear response problem in a way similar
way as the real case is accomplished based on the eigenvalues of the reduced matrix $\tilde{\mathbf{\Pi}}_{11}$.
In the next section, we will examine some concrete examples.

\section{Illustrative examples}\label{sec:examples}
We first illustrate the simplification due to using the square matrix $\mathbf{M}_-^2$ by considering
a 2-by-2 example,
\begin{eqnarray}
\mathbf{M}_-=\left[
\begin{array}{cc}
x & 3i \\
3i & -x \\
\end{array}
\right],\;\;
\mathbf{M}_-^2=\left[
\begin{array}{cc}
x^2-9 & 0 \\
0 & x^2-9 \\
\end{array}
\right],\label{example1}
\end{eqnarray}
where $x\in\mathbb{R}$ is a parameter to mimic the effect of changing physical parameters
in the linear response problem, such as the change of bond length of diatomic molecules in
scanning potential energy curves. It is seen that the matrix $\mathbf{M}_-^2$ is already diagonal,
which gives two identical eigenvalues $\lambda=x^2-9$. Consequently, the original problem
have two eigenvalues $\omega=\pm\sqrt{x^2-9}$. The eigenvalues as a function of $x$ are shown in
Fig. \ref{fig:case1}. The graphs can be classified into three regions:

(1) $x\le-3$: $\lambda\ge 0$ and a pair of real eigenvalues $\omega=\pm\sqrt{x^2-9}$
appears, although the electronic Hessian is not positive definite.

(2) $-3<x<3$: $\lambda<0$ and a pair of purely imaginary eigenvalue $\omega=\pm i\sqrt{9-x^2}$ appear.

(3) $x\ge3$: $\lambda\ge 0$ and the electronic Hessian is positive definite.

\begin{figure}\centering
\begin{tabular}{c}
\resizebox{0.4\textwidth}{!}{\includegraphics{figures/case1-w}} \\
(a) $\omega$ as a function of $x$ \\
\resizebox{0.4\textwidth}{!}{\includegraphics{figures/case1-w2}} \\
(b) $\lambda$ as a function of $x$ \\
\end{tabular}
\caption{A 2-by-2 example for $\mathbf{M}_-$ \eqref{example1}:
(a) eigenvalues of $\mathbf{M}_-$ (denoted by $\omega$) as a function of $x$; (b) eigenvalues of $\mathbf{M}_-^2$ (denoted by $\lambda$) as a function of $x$. The blue solid (dashed) lines represent the real (imaginary) parts of eigenvalues.
The two red vertical lines represent critical values of $x=\pm3$.}\label{fig:case1}
\end{figure}

Next, we examine a more complex example, which covers all the scenarios for eigenvalues of $\mathbf{M}_-$.
The matrices $\mathbf{A}$ and $\mathbf{B}$ are chosen as
\begin{eqnarray}
\mathbf{A}=\left[
\begin{array}{cc}
x & 3+i \\
3-i & 5 \\
\end{array}
\right],
\quad
\mathbf{B}=
\left[
\begin{array}{cc}
4 & 4 \\
4 & 4 \\
\end{array}
\right].\label{example2}
\end{eqnarray}
The eigenvalues of $\mathbf{M}_-$ can be found analytically as
\begin{eqnarray}
\omega&=&\pm\sqrt{\frac{-19+x^2\pm\sqrt{\Delta}}{2}},\nonumber\\
\Delta&=&25 + 272 x - 74 x^2 + x^4.\label{example2:omega}
\end{eqnarray}
Following the procedure described in the previous section, one can find the corresponding
skew-Hamiltonian $\mathbf{\Pi}$ \eqref{pi} as
\begin{eqnarray}
\mathbf{\Pi}=
\left[\begin{array}{cccc}
-22+x^2 & -37+7x & 0 & -3+x \\
3-x & 3 & 3-x & 0 \\
0 & -13-x & -22+x^2 & 3-x \\
13+x & 0 & -37+7x & 3 \\
\end{array}\right]
\end{eqnarray}
Applying the following Givens rotation $\mathbf{G}$ with an appropriate angle $\theta$ to eliminate $\Pi_{41}$,
\begin{eqnarray}
\mathbf{G} =
\left[\begin{array}{cccc}
1 & 0 & 0 & 0 \\
0 & \cos\theta & 0 & -\sin\theta \\
0 & 0 & 1 & 0 \\
0 & \sin\theta & 0 & \cos\theta \\
\end{array}\right],
\end{eqnarray}
we can find the upper-left block of $\tilde{\mathbf{\Pi}}=\mathbf{G}\mathbf{\Pi}\mathbf{G}^\dagger$ as
\begin{eqnarray}
\tilde{\mathbf{\Pi}}_{11}=
\left[\begin{array}{cc}
-22+x^2 & \frac{\sqrt{2}(-3+x)(-25+3x)}{\sqrt{89+10x+x^2}} \\
-\sqrt{2(89+10x+x^2)} & 3 \\
\end{array}\right].
\end{eqnarray}
It can be verified that its two eigenvalues are given by
$\lambda_\pm=\frac{-19+x^2\pm\sqrt{\Delta}}{2}$, which is
consistent with Eq. \eqref{example2:omega}.
As shown in Fig. \ref{fig:case2}, the eigenvalues $\omega$ and $\lambda$ as a function of $x$ are much more complicated
in this example. The conditions $\Delta=0$ and $\lambda_{\pm}=0$ determine four real critical values of $x$ in total, viz.,
\begin{eqnarray}
&x_1\approx -10.04,\quad x_2\approx -0.09,\nonumber\\
&x_3=\frac{14}{9}\approx 1.56,\quad x_4=6.\label{critical2x}
\end{eqnarray}
Consequently, the graphs can be classified into five regions:

(1) $x\le x_1$: $\lambda_+\ge\lambda_->0$ and $\mathbf{M}_-$ has two pairs of real eigenvalues.

(2) $x_1<x< x_2$: $\lambda_+=\lambda_-^*$ become complex, such that $\mathbf{M}_-$ has a quadruple of eigenvalues.

(3) $x_2\le x< x_3$: $0>\lambda_+\ge\lambda_-$ and $\mathbf{M}_-$ has two pairs of purely imaginary eigenvalues.

(4) $x_3\le x<x_4$: $\lambda_+\ge0>\lambda_-$ and $\mathbf{M}_-$ has a pair of real eigenvalues and a pair of purely imaginary eigenvalues.

(5) $x\ge x_4$: $\lambda_+>\lambda_-\ge0$ and $\mathbf{M}_-$ has two pairs of real eigenvalues.

This example covers all the three different scenarios of eigenvalues of $\mathbf{M}_-$ discussed in the previous section.
All of them can be easily characterized by eigenvalues of a simpler matrix $\tilde{\mathbf{\Pi}}_{11}$ with halved dimension,
which is a natural generalization of $(\mathbf{A}-\mathbf{B})(\mathbf{A}+\mathbf{B})$ or $(\mathbf{A}+\mathbf{B})(\mathbf{A}-\mathbf{B})$ in the real case.
Finally, we mention that for larger matrices, the eigenvalues cannot be computed analytically, but it is straightforward to
implement the reduction procedure numerically. The behaviors of eigenvalues can be understood in the same way following
the examples presented here.

\begin{figure}\centering
\begin{tabular}{c}
\resizebox{0.4\textwidth}{!}{\includegraphics{figures/case2-w}} \\
(a) $\omega$ as a function of $x$ \\
\resizebox{0.4\textwidth}{!}{\includegraphics{figures/case2-w2}} \\
(b) $\lambda$ as a function of $x$ \\
\end{tabular}
\caption{A 4-by-4 example for $\mathbf{M}_-$ \eqref{example2}:
(a) eigenvalues of $\mathbf{M}_-$ (denoted by $\omega$) as a function of $x$; (b) eigenvalues of $\mathbf{M}_-^2$ (denoted by $\lambda$) as a function of $x$.
The blue solid (dashed) lines represent the real (imaginary) parts of eigenvalues.
The four red vertical lines represent critical values of $x$ given in Eq. \eqref{critical2x}.
}\label{fig:case2}
\end{figure}

\section{Conclusion}
In this work, we provided a unified view for the two commonly appeared structured eigenvalue problems
in (relativistic) electronic structure methods - the quaternion matrix eigenvalue problem and
the linear response eigenvalue problem for excitation energies. Using the same
reduction algorithm, we derived a generalized condition
to characterize the different scenarios for eigenvalues of the
complex linear response problem. Such understandings may allow to design
more efficient and robust diagonalization algorithms in future.

\section*{Acknowledgements}
This work was supported by the National Natural Science Foundation of China (Grants
No. 21973003) and the Beijing Normal University Startup Package.

\bibliographystyle{apsrev4-1}