\section{\label{sec:introduction}Introduction}
Optomechanical systems are widely used in experiments of precise measurements~\cite{Aspelmeyer:2013lha}.
In particular, optomechanical systems consisting of massive oscillators are suitable for testing macroscopic quantum mechanics~\cite{Chen:2013sgs}, investing Newtonian interaction of quantum objects~\cite{Miao:2019pxw, Kafri:2014zsa}, measuring gravitational force of milligram masses~\cite{Schmole:2016mde}, and gravitational wave detection~\cite{LIGOScientific:2016aoc}.
An optomechanical system consists of mechanical oscillators coupled with optical fields.
The displacement of or the force acting on the oscillator can be measured precisely by using optical interferometry.
Massive oscillators not only make the optomechanical systems resistant to noise sources~\cite{LIGOScientific:2016aoc}, but also allow for the exploration of the unique physics~\cite{Chen:2013sgs,Miao:2019pxw,Kafri:2014zsa,Schmole:2016mde}.
Ultimately, macroscopic quantum optomechanical systems are expected to elucidate the quantum nature of gravity~\cite{Marletto:2017kzi, Bose:2017nin, Belenchia:2018szb}.

Suspended pendulums are often used as mechanical oscillators in optomechanical systems over a milligram scale~\cite{Pontin:2016nem, Matsumoto:2013sua, Matsumoto:2014fda, Komori:2019zlg, Sakata:2010zz, Matsumoto:2018via, Corbitt:2007spn, Neben:2012edt, LIGOScientific:2009mif, LIGOScientific:2020luc}, while membranes and cantilevers are used in many experiments of smaller mass scales~\cite{Chan:2011ivv, Teufel:2011smx, Peterson:2016ayo}.
Suspended pendulums are advantageous in that they can be isolated from the environment.
In other words, pendulums are resistant to seismic noise and thermal noise.
Furthermore, a pendulum can be regarded as a free mass in the broad frequency range over the resonant frequency.
In general, pendulums have a wide range of sensitivity for this reason.
Recently, detecting schemes for ultralight dark matter using optomechanical oscillators were proposed~\cite{Graham:2015ifn, Pierce:2018xmy, Carney:2019cio}, and the wide searchable range was crucial for the dark matter search because the mass of dark matter was scarcely known.

\begin{figure}
    \centering
    \includegraphics[width=0.7\columnwidth,clip]{rotation.pdf}
    \caption{Schematic illustration of a suspended linear cavity. The rotational degrees of freedom we focus on are defined along the vertical axis.
    The input mirror is much heavier than the end mirror for our configuration.}
    \label{fig:rotation}
\end{figure}
However, the issue of suspended pendulums as mechanical oscillators is their instability in the rotational degree of freedom, the so-called Sidles-Sigg instability~\cite{Sidles:2006vzf}.
In a linear cavity, the yaw rotational motion of a suspended mirror that is indicated in Fig.~\ref{fig:rotation} changes the position of the beam spot of the laser beam.
Due to the change in the position of the beam spot, the radiation pressure in the cavity can behave as an anti-restoring force and destabilize the cavity.

In the case that the mechanical restoring torque is dominant in the rotational degree of freedom, the Sidles-Sigg instability does not matter because the radiation pressure torque would be too weak to make a pendulum unstable.
Thus, several experiments used multiple wires to suspend a mirror to stiffen the pendulum in the rotational degree of freedom~\cite{Corbitt:2007spn, Neben:2012edt, Kelley:2015axa}.
However, the increase of wires induces a stronger coupling to a thermal bath.
As a result, the thermal noise due to the suspension gets larger.

One other way to deal with Sidles-Sigg instability is introducing feedback control in the rotational motion~\cite{Barsotti:2010zz}.
If an oscillator has actuators applying torque on it, the unstable rotational motion can be suppressed.
Therefore, modern gravitational wave detectors use active feedback controls for the angular motion~\cite{Liu:2018dfs}.
However, active feedback control systems were not always available, especially for macroscopic quantum experiments and quantum measurements.
This was because milligram or gram scale oscillators were often too small to attach actuators to~\cite{Sakata:2010zz}.
In such cases, a more complicated feedback system was required to actuate a tiny oscillator remotely~\cite{Enomoto:2016lee, Nagano:2016wcg}.

As a different approach, some experiments used triangular cavities to avoid the Sidles-Sigg instability~\cite{Matsumoto:2013sua, Matsumoto:2014fda, Komori:2019zlg}.
When the number of mirrors consisting in the cavity is odd, the radiation pressure behaves as a positive restoring torque.
Thus, the Sidles-Sigg instability can be avoided.
However, additional mirrors are noise sources~\cite{Komori:2019zlg}.
For the best sensitivity, a cavity with only two mirrors (a linear cavity) is favorable.
Therefore, a stable configuration of a linear cavity is desired.

In this paper, we propose and experimentally validate a stable trapping configuration for a suspended mirror in a linear cavity.
We utilize radiation pressure inside the cavity as the restoring torque.
Thus, our system can trap a rotational motion of the suspended mirror without additional active feedback controls.
Therefore, our trapping scheme is applicable to tiny systems that cannot have actuators attached to them, and it is free from the feedback-control noises.
In our system, we operate the cavity in the negative-$g$ regime, and the input mirror is much heavier than the end test mass.
The combined characteristics of the negative-$g$ regime and unbalanced masses produce a stable rotational trapping.
Furthermore, we validate the configuration experimentally with an 8 mg mirror.
In addition, we discuss the feasibility to observe the quantum radiation pressure fluctuation of a milligram scale optomechanical system for testing macroscopic quantum mechanics by using our trapping configuration.

\section{\label{sec:theory}Theoretical description}
We analyze the rotational motion of suspended mirrors in a linear cavity.
The following calculation shows the suspended mirrors are trapped with the positive radiation pressure torque under the condition that the cavity is in the negative-$g$ regime and one mirror is much heavier than the other one.
As shown in Fig.~\ref{fig:rotation}, we define the angles of two mirrors as $\alpha_i$ and the torques exerted to them as $T_i$ ($i = 1,~2)$.
The equation of motion of the rotational modes of the two mirrors is given by
\begin{align}
    (\bm{K}_\mathrm{opt} + \bm{K}_\mathrm{mech} - \bm{I}\omega^2)
    \begin{pmatrix}
    \alpha_1 \\ \alpha_2
    \end{pmatrix}
    =
    \begin{pmatrix}
    T_1 \\ T_2
    \end{pmatrix}
    ,
    \label{eq:motion}
\end{align}
where
\begin{align}
    \bm{K}_\mathrm{mech} =
    \begin{pmatrix}
    K_1 & 0 \\
    0   & K_2
    \end{pmatrix}
    ,~~
    \bm{I} =
    \begin{pmatrix}
    I_1 & 0 \\
    0   & I_2
    \end{pmatrix}
\end{align}
are the matrices of the mechanical restoring torques and the moment of inertia of each mirror.
The optical torsional stiffness matrix is represented as~\cite{Sidles:2006vzf}
\begin{align}
    \bm{K}_\mathrm{opt} = \frac{2P}{c(R_1 + R_2 - L)}
    \begin{pmatrix}
    R_1(L - R_2) & R_1R_2 \\
    R_1R_2       & R_2(L - R_1)
    \end{pmatrix},
    \label{eq:stiffness}
\end{align}
where $P$, $c$, $R_1$, and $L$ are the intracavity power, the speed of light, the radii of curvature, and the cavity length, respectively.
The optical torque is generated by the change of the beam spot on the mirror due to the movement of the cavity axis as the mirror rotates.
Equation~(\ref{eq:motion}) can be rewritten as
\begin{align}
    \begin{pmatrix}
    K_1 - \beta g_2 - I_1\omega^2 & \beta \\
    \beta                         & K_2 - \beta g_1 -I_2\omega^2
    \end{pmatrix}
    \begin{pmatrix}
    \alpha_1 \\ \alpha_2
    \end{pmatrix}
    =
    \begin{pmatrix}
    T_1 \\ T_2
    \end{pmatrix},
    \label{eq:rewritemotion}
\end{align}
by defining $\beta = 2PL/[c(1 - g_1g_2)]$, $g_i = 1 - L/R_i$.
$g_i$ is determined by the geometry of the cavity and generally called the $g$ factor.

Hereafter, we consider a case where the mirror 2 is much heavier than the mirror 1, and the mechanical restoring torque of the mirror 1 is much smaller than that of the mirror 2.
This assumption is practical for actual experiments.
This optomechanical system can be a sensitive force sensor by using the lighter mirror as a test mass.
At the same time, we can control the cavity length to maintain the resonance by attaching actuators on the larger (heavier) mirror; $I_1 \ll I_2$, and $K_1 \ll K_2$.
In this case, the diagonalization of Eq.~(\ref{eq:rewritemotion}) indicates the resonant frequency of the differential mode is
\begin{align}
    \omega_\mathrm{diff} \simeq \sqrt{\frac{K_1 - \beta g_2}{I_1}}.
    \label{eq:resfreqdiff}
\end{align}
The diagonalization of Eq.~(\ref{eq:rewritemotion}) derives two decoupled modes.
Here, the eigen mode where the two mirrors rotate in the same direction is named the differential mode.
On the other hand, the eigen mode in which the mirrors rotate in the opposite directions is named the common mode.

When the lighter mirror is flat ($g_1 = 1$), $g_2$ should be positive to satisfy the optical cavity condition of $0 < g_1g_2 <1$.
In this case, the resonant frequency rapidly goes to zero with the increase of the intracavity laser power, which causes an angular instability.
On the other hand, we can avoid the instability and even can stiffen the differential mode in the negative-$g$ regime.
Equation~(\ref{eq:rewritemotion}) also indicates the resonant frequency of the common mode decreases as
\begin{align}
    \omega_\mathrm{com} \simeq \sqrt{\frac{K_2 + \beta(1 - g^2)/g}{I_2}}.
\end{align}
Here, we assume the two curvatures of the mirrors are identical ($R_1 = R_2 = R$, $g_1 = g_2 = g$) for simplicity.
The mechanical resonant frequency of the common mode can be high enough by using a heavy enough mirror for the mirror 2.
Therefore, the decrease of the resonant frequency due to the radiation pressure torque can be ignored.
In other words, the radiation pressure torque will not make the common mode unstable when one mirror is much heavier than the other.
The tolerable intracavity power can be increased by increasing only the mass of one mirror.
Thus, this configuration allows trapping of a small mirror.

Figure~\ref{fig:theoretical}. shows the dependence of the resonant frequency on the intracavity power.
\begin{figure}
    \centering
    \includegraphics[width=\columnwidth,clip]{theoretical.pdf}
    \caption{Dependence of the resonant frequencies of differential and common modes.
    The negative resonant frequency implies that the mode is unstable.
    For comparison, both the negative-$g$ regime ($g_1 = g_2 = -0.1$) and the positive-$g$ regime ($g_1 = g_2 = +0.1$) cases are plotted; other parameters are described in the main text.
    In the range between 0.72~W and 34~kW of the intracavity power, only the negative-$g$ cavity is stable.}
    \label{fig:theoretical}
\end{figure}
As for this plot, we use similar parameters of our experimental setup as follows: mirror masses of $m_1 = 10~\mathrm{mg}$ and $m_2 = 10~\mathrm{g}$, mirror radii of $r_1 = 1.5~\mathrm{mm}$ and $r_2 = 10~\mathrm{mm}$, (momentum of intertia of $I_1 = 5.6\times 10^{-12}~\mathrm{kg~m^2}$ and $I_2 = 2.5\times 10^{-7\textbf{}}~\mathrm{kg~m^2}$,) mechanical resonant frequency of $\omega_1/(2\pi) = 0.5~\mathrm{Hz}$ and $\omega_2/(2\pi) = 5~\mathrm{Hz}$, and the cavity length of $L = 1.1R = 11~\mathrm{cm}~(g = -0.1)$.
For comparison, we also plot the resonant frequencies of the differential and common modes for the case that the $g$ factors of the cavity mirrors are positive ($g_1 =g_2 = +0.1$).
The cavity in the negative-$g$ regime is tolerant over the intracavity power of 10~kW, while the cavity in the positive-$g$ regime is unstable just over 0.72~W.
\begin{figure*}
    \centering
    \includegraphics[width=2\columnwidth,clip]{setupphoto.pdf}
    \caption{(a)Schematic drawing of the experimental setup.
                The test mass is an 8~mg mirror, which is suspended as a double pendulum.
                The test mass and the input mirror consist of the main cavity on the platform in vacuum chamber.
                The laser source is an Nd:YAG laser, and the wavelength is 1064~nm.
                The cavity length is feedback controlled for the continuous resonance.
                The laser intensity fluctuation is also suppressed by the feedback control with an acousto-optic modulator.
                The beam spot on the test mass is monitored with a quadrant photo detector for the transfer function measurement.
             (b)Photographs of the main cavity on the platform.
                The platform is suspended with springs as a double pendulum to isolate the main cavity from seismic vibration.
                The test mass is focused on the lower left side. The input mirror and the coil magnet actuator on it is focused on the lower right side.}
    \label{fig:setup}
\end{figure*}

\section{\label{sec:method}Experimental demonstration}
We experimentally demonstrate that our trapping configuration works properly.
As mentioned in the previous section, our configuration utilizes the radiation pressure of the laser light inside a linear cavity.
Thus, the restoring torque due to the radiation pressure also increases as the intracavity power increases.
We design our experiment to observe this increase of the restoring torque.

The experimental setup is shown in Fig.~\ref{fig:setup}.
The instability due to the radiation pressure of the laser light inside the cavity will be an issue when the radiation pressure torque is dominant.
To realize the predominance of the radiation pressure, we build a linear cavity with a tiny mirror of 8~mg (0.5~mm thick with a diameter 3~mm) as the test mass.
This tiny mirror is suspended with a thin carbon fiber (6~$\mu$m in diameter and 2~cm long) to lower the mechanical restoring torque.
The $Q$ value of a single pendulum with this carbon fiber is measured to be $Q \sim 8\times 10^4$ at the mechanical resonant frequency, 3~Hz.
The test mass is suspended as a double pendulum via the intermediate mass for isolation from the seismic motion.
The input mirror is much heavier.
Its mass is 60~g.
Two coil-magnet actuators are attached to the input mirror to apply a force and a torque.
The radii of the curvature of the mirrors are 10~cm, and the cavity length is $11.0~\pm 0.3$~cm.
Thus, the cavity is in the negative-$g$ regime of $g = -0.1$.
The finesse of the cavity is measured to be $(3.0 \pm 0.3)\times 10^3$.
The cavity is built on a platform board; the platform is also suspended as a double pendulum in a vacuum chamber.
The pressure is kept about 1~Pa to suppress acoustic disturbances.

The cavity length is feedback controlled to resonate continuously.
We use a side locking to keep the cavity at a detuned point near half of the resonant peak.
The displacement of the mirror from the control point is sensed by the reflected-light power that is monitored by the photo detector.
The error signal is obtained by comparing the output signal from the photo detector with a constant voltage.
The error signal is filtered and sent to the coil-magnet actuator attached to the input mirror.
The transmitted light from the cavity is monitored by a quadrant photo detector, as described in Fig.~\ref{fig:setup}.
The half of the injected laser beam is also monitored for the laser intensity control.
By sending the feedback signal to the acousto-optic modulator (AOM), we suppress the laser intensity fluctuation.

To show the radiation pressure works as a positive restoring torque, we evaluate the resonant frequency that is described in Eq.~(\ref{eq:resfreqdiff}).
The resonant frequency is determined by the transfer function of the rotational motion of the mirrors.
We apply torque to the input mirror by injecting differential sine-wave signals into the coil-magnet actuators on the input mirror~\ref{fig:setup}.
Then, the test mass is also swung via the radiation pressure inside the cavity.
The rotation of the test mass results in the changes of the beam spot on the test mass because the cavity axis changes.
We observe the transmitted light from the cavity by a quadrant photo detector to detect the change in the beam spot.
A convex lens is placed halfway between the test mass and the quadrant photo detector to measure only the change in the beam spot without being affected by the change in the cavity axis.
The distance between the lens and the test mass (the quadrant photo detector) is twice as long as the focal length of the lens.
The transfer function from the injected signal to quadrant photo detector signal includes the transfer function of the suspended mirrors~\cite{Nagano:2016wcg}.
Thus, the transfer function has the characteristic form of the resonant peak.
We fit the transfer function to estimate the resonant frequency.
Note that the cavity length is feedback controlled to keep the resonance of the cavity during this transfer function measurement.

\section{\label{sec:result}Result}
The measured transfer functions from the excitation to the quadrant photo detector signal are plotted in Fig.~\ref{fig:transfunc}.
We measure them with five different intracavity powers.
\begin{figure}
    \centering
    \includegraphics[width=\columnwidth,clip]{YawtransPaper.pdf}
    \caption{The measured transfer functions of the five measurements of different intracavity powers. The points are the measured data, and the lines represent the fittings. The peaks and the phase flips indicate the resonant points. By the fitting, we estimate the resonant frequencies.}
    \label{fig:transfunc}
\end{figure}
The peaks and the phase flips due to the resonance of the test mass are observed in each measurement.
The intracavity power in each measurement is estimated by the transmitted light power.
We fit the gain of the transfer functions to determine the resonant frequency.
The fitted parameters are resonant frequency, damping ratio of the resonance, and the overall gain factor.
The fitted curves are also plotted in Fig.~\ref{fig:transfunc}.
The uncertainty in the intracavity power is dominated by the fluctuation in the power of the transmitted light.
The fluctuation in the transmitted light power is at the frequency of the excitation signal.
Thus, the fluctuation would be due to the misalignment when the mirror is swung.

\begin{figure}
    \centering
    \includegraphics[width=\columnwidth,clip]{ResFreq_m.pdf}
    \caption{The resonant frequencies of the differential mode in the rotational degree of freedom. The shaded region represents the theoretically predicted values with the width corresponding to the uncertainties in mirror reflectivities and the cavity length.}
    \label{fig:resfreq}
\end{figure}
We show the dependence of the resonant frequency on the intracavity power in Fig.~\ref{fig:resfreq}.
The uncertainty of the resonant frequency is estimated from the fit of the transfer function measurement.
We also show the predicted region from the theoretical calculation using Eq.~(\ref{eq:resfreqdiff}) with the parameters of the optics.
The width of the region corresponds to the uncertainty of the design reflectivities of the mirrors and the uncertainty of the cavity length.
The measured dependency is consistent with the theoretical prediction.

We note that the longitudinal optical spring effect does not affect our results since the beam spot on the test mass mirror is precisely adjusted to the center.
When the beam spot is off the center of the mirror, the longitudinal optical spring also acts as a restoring torque.
However, the deviation of the beam spot from the center of the test mass must be smaller than 2~$\mu$m with our experimental parameters.
Otherwise, the constant radiation pressure rotates the test mass mirror and breaks the cavity locking.
When the deviation is smaller than 2~$\mu$m, the restoring torque from the longitudinal optical spring is smaller than several $10^{-11}$~Nm/rad.
This is smaller than the original restoring torque of the pendulum's suspension wire.
Furthermore, the trapping potential by the longitudinal optical spring is suppressed by the feedback controlling of the cavity length, and thus, its effect is further minimized.
Therefore, we conclude that the effect of the coupling of the longitudinal optical spring is negligible.

\section{\label{sec:discussion}Discussion}
One of the essential applications of our trapping scheme is the observation of the quantum radiation pressure fluctuation for testing macroscopic quantum mechanics; the quantum radiation pressure fluctuation is a radiation pressure fluctuation due to the quantum fluctuation of the photon number of light.
As mentioned in the Introduction, one attracting target of milligram scale optomechanical systems is testing macroscopic quantum mechanics.
A proposed experimental test of macroscopic quantum mechanics requires to prepare a conditional state~\cite{Chen:2013sgs}.
To set an optomechanical system in a conditional state, the quantum radiation pressure fluctuation needs to dominate force noises~\cite{Michimura:2020yvn}.
For the observation of quantum radiation pressure fluctuation, the combination of a light mirror and a high power laser beam is preferable.
The high power laser beam enhances the quantum radiation pressure fluctuation itself, and a light mirror enhances the displacement of the mirror that is sensed by the interference.
Therefore, our trapping system is suitable because it overcomes the Sidles-Sigg instability, which limited the optimal sensitivities in the previous milligram and gram scale experiments~\cite{Corbitt:2007spn,Neben:2012edt,Kelley:2015axa,Sakata:2010zz,Matsumoto:2013sua,Matsumoto:2014fda,Komori:2019zlg}

We estimate the sensitivity of our experimental setup to discuss the feasibility of observing the quantum radiation pressure fluctuation.
The calculation reveals that the quantum radiation pressure fluctuation will be dominant with the intracavity power of over 14~W (see the Appendix~\ref{sec:appendix} for more details).
Since our trapping scheme overcomes the limitation of the Sidles-Sigg instability, the cavity can accumulate 30~kW or more power inside, according to our measurement result.
Therefore, we conclude that our system can realize the observation of the quantum radiation pressure fluctuation, though technical classical noises should be well suppressed.

\section{\label{sec:conclusion}Conclusion}
We propose a configuration to trap the rotational motions of the suspended mirrors in a linear cavity.
By operating a linear cavity in the negative-$g$ regime and using unbalanced-mass mirrors, the two rotational modes of the cavity mirrors are stable with the radiation pressure inside the cavity.
Furthermore, we demonstrate an experimental validation of the trapping by building a linear cavity with an 8~mg mirror.
We observe the rotational restoring torque on the mirror increases as the intracavity power increases.
The behavior is consistent with theoretical prediction.
Therefore, we confirm that the 8~mg mirror obtains the positive restoring torque originating from the radiation pressure of the inside laser beam.

We also discuss the possibility of observing quantum radiation pressure fluctuation by using our trapping scheme.
The calculation shows that the quantum radiation pressure fluctuation can be observed with the realistic parameters that are identical or similar to our realized experiment.
Considering the successful result of the trapping together, we confirm that the cavity can accumulate enough large laser power to enhance the quantum radiation pressure fluctuation to observe it.
Thus, this work is a crucial step towards testing macroscopic quantum mechanics, while the configuration is also applicable to a broad range of optomechanical systems.

\begin{acknowledgments}
We thank Yutaro Enomoto and Ooi Ching Pin for fruitful discussions.
This work was supported by a Grant-in-Aid for Challenging Research Exploratory Grant No. 18K18763 from the Japan Society for the Promotion of Science (JSPS), by JST CREST Grant No. JPMJCR1873, and by MEXT Quantum LEAP Flagship Program (MEXT Q-LEAP) Grant No. JPMXS0118070351.
T.K. is supported by KAKENHI Grant No. 19J21861 from the JSPS.
\end{acknowledgments}