\section{Introduction}
\label{sec:intro}
Determining the stability of planetary systems is a foundational question in dynamics with multiple applications to Solar System and extrasolar planetary system problems. The two planet case is the only one that is essentially solved \citep{marchal82, gladman93, Petit2018, Hadden2018}. For higher multiplicity, there is a qualitative understanding that chaos and instability result from resonance overlap \citep{lecar01}, and important partial semi-analytic results exist \citep{petit20,yalinewich20, tamayo21}. Numerical integrations, however, are needed to address most questions of astrophysical interest.

The evolution of systems of multiple planets is affected by both secular and resonant processes \citep{murray99}. Under secular evolution, planets exchange angular momentum at fixed energy, leading to changes in orbital eccentricity and mutual inclination while semi-major axes remain constant. The Angular Momentum Deficit (AMD) is conserved in this limit, and if the AMD satisfies ${\rm AMD} < {\rm AMD}_{\rm crit}$, where ${\rm AMD}_{\rm crit}$ is the minimum required for orbit crossing, then the system is ``AMD-stable" \citep{laskar17}. Orbits within an AMD-unstable system, conversely, can potentially evolve until planetary close-encounters occur. Qualitatively different behavior occurs when mean-motion resonances (MMRs) play a significant role. Under resonant dynamics the semi-major axes of the orbits evolve, and non-zero eccentricity can be generated from initially circular and co-planar configurations. Instability arises in such a system due to the overlap of mean-motion \citep{obertas17} and/or three-body resonances \citep{quillen11}. In model systems of equal mass planets that are initially spaced evenly in $\log$(semi-major axis), the average instability time is an approximately exponential function of the planetary spacing \citep{chambers96,smith09,obertas17,lissauer21}, with a dispersion that is set by the chaotic dynamics \citep{hussain20}.
In general systems, where the masses or separations are non-uniform, good analytic estimates of the instability time have only recently been worked out \citep{petit20}.
The average instability time is a smooth enough function of the system parameters that it can also be estimated reliably using Machine Learning methods \citep{tamayo16,tamayo20,cranmer21}.

The formation and evolution of the Solar System and extrasolar systems is understood to involve a complex mix of secular and resonant dynamics \citep{davies14}, with further complications arising when tides or interactions with gas or planetesimal discs are present \citep{morbidelli18}. In the current Solar System, the dynamics of the planets is controlled by secular effects \citep{laskar90}, and the system is quasi-stable, with a low-probability destabilization of Mercury's orbit remaining as an allowed possibility \citep{laskar94,batygin08,abbot21}. In extrasolar planetary systems, a subset of the currently observed configurations are hypothesized to result from an earlier phase of instability. For systems of giant planets, scattering from near-circular initial conditions -- unstable due to resonant effects -- may explain the observed eccentricity distribution \citep{rasio96,weidenschilling96,ida97,chatterjee08,juric08}. Secular chaos \citep{wu11,teyssandier19}, in multi-planet systems with non-zero initial AMD, is one of several ways to form hot Jupiters via a combination of secular dynamical processes and tidal dissipation \citep{naoz11,petrovich15,dawson18}. At lower masses, the close-in planetary systems that dominate the {\em Kepler}-discovered population \citep{thompson18} may also derive from unstable progenitors, with both resonant and secular dynamics being potentially important \citep{pu15,volk20}. Population analyses suggest that in addition to N-body dynamics, dissipation likely also played a role in forming observed systems \citep{yee21}.

Planetary systems dominated by secular dynamics typically have large separation that exclude the presence of low-order MMRs, and are sometimes studied using numerical schemes that explicitly filter out resonant effects \citep{laskar89}. Physically, however, there could be a class of planetary system where both secular and weaker resonant processes control the stability \citep{wu11,petit17}, and our goal in this paper is to explore the interplay between these processes. We employ a setup that is a variant of the one used by \citet{chambers96} and many subsequent authors: three equal mass planets uniformly spaced in $\log$(semi-major axis). We focus on massive planets, and initialize our systems with moderately large eccentricities. This is evidently a highly idealized system, and we note at the outset that it means that weak resonances in the system always occur in chains between both adjacent pairs of planets rather than between just one pair.

The layout of the paper is as follows. In section \ref{sec:methods} we describe the setup and execution of the integrations done in this work. Our results and interpretations are introduced in section \ref{sec:results} and the implications of these results are discussed in section \ref{sec:discussion}.

\section{Methods}
\label{sec:methods}

In brief, we numerically integrated systems of three massive (Jupiter mass, around a Solar mass star) planets for $10^8$ orbits of the inner planet. The initial conditions were coplanar, with eccentricity set to $e=0.33$ for each planet. In nature, such high eccentricity systems will likely also have significant mutual inclinations, but this coplanar setup allows us to isolate the partitioning of AMD into one variable for ease of analysis. The planets were uniformly spaced in $\log$(semi-major axis), with simulations populating the range of semi-major axis ratios $3.33 \leq \alpha^{-1} \leq 5$, where $\alpha$ is defined as
\begin{equation}
    \alpha \equiv \frac{\mathrm{a}_{\text{inner}}}{\mathrm{a}_{\text{outer}}} \label{eqn:alphadef}
\end{equation}
for a pair of adjacent planets. Integrations were performed using \texttt{REBOUND} \citep{rebound}.

\subsection{Integration Scheme}
\label{subsec:intscheme}

We used \texttt{REBOUND}, an open source N-body integration package developed by \cite{rebound}. \texttt{REBOUND} is equipped with a suite of integrators that differ in their methods and suitability. After comparing and benchmarking \texttt{IAS15} and \texttt{WHFast} on this problem, we adopted the \texttt{WHFast} integrator \citep{wh, reboundwhfast}. To boost performance, we de-toggled the built-in "safe mode" and set the integrator to use an eleventh-order symplectic corrector.

\subsection{Termination Conditions}
\label{subsec:terminationconds}

Over the course of the integration, pairs of planets may experience collisions, close encounters, and orbit crossings. We terminate runs upon any of these events. For collisions, we set a condition to terminate if the number of planets present in the simulation changes from its initial value.
We define a close encounter as when the separation between two adjacent planets, $k$ and $k+1$, becomes less than their mutual Hill radius, defined by
\begin{equation}
\mathrm{R}_{\text{Hill, mutual}} \equiv \left( \frac{ \mathrm{m}_k + \mathrm{m}_{k+1} }{3 \mathrm{M}_\star } \right)^{1/3} \frac{ \mathrm{a}_k + \mathrm{a}_{k+1} }{2}.
\end{equation}
Here, m$_i$ are the planet masses and M$_\star$ is the mass of the star. Lastly, an orbit crossing occurs when the apocentre of planet $k$ becomes exterior to the pericentre of planet $k+1$. Quantitatively, this is when
\begin{equation}
\mathrm{a}_{k+1} \left( 1 - \mathrm{e}_{k+1} \right) < \mathrm{a}_k \left( 1 + \mathrm{e}_k \right).
\end{equation}
Any of these events signify instability for our initially well-separated planets.

\subsection{Initial conditions}
\label{subsec:initconds}

We focus on a model system of three massive planets on initially eccentric, coplanar orbits around a Solar mass star. Each planet is given the same mass of m =  1 M$_\text{J}$ and initial eccentricity of e = 0.33. We set the innermost planet at a$_1 = 1$ AU, then initialise the semi-major axes of the outer planets according to the separation factor $\alpha$. We initially set $\varpi = \Omega = 0$ for each planet, but randomize their initial true anomalies.  This initial alignment in $\varpi$ does not have a significant impact on our results. We found similar results in a suite of test simulations with initially misaligned $\varpi$ which we will discuss in Section~\ref{subsec:initcondcomments}.

We split the $\alpha$ range into 333 systems, uniformly spaced in log(semi-major axis). It is important to note that defining the separation as in (\ref{eqn:alphadef}) will force any mean-motion resonances to occur in a chain, meaning each pair of adjacent planets will be in the same resonance. These $\alpha$ values are widely spaced enough that we expect secular effects to dominate, and no low-order MMRs to be present, but this range does contain several high-order MMRs. The simulations are purely gravitational, and scale-free apart from the physical collision termination condition. For practical purposes, the results can be re-scaled to different initial orbital radii with an appropriate re-scaling of time units. We ran the integrations for $10^8$ years, equivalent to $10^8$ orbits of the innermost planet, with a timestep of $10^{-3}$ years.

Under secular exchange of AMD, large peak eccentricities are most readily reached if the inner planets are less massive than the outer planets. By considering equal mass planets, exchange of AMD leads to smaller eccentricity excursions than the unequal mass case that has often been used in simulations of hot Jupiter formation via secular chaos.

\section{Results}
\label{sec:results}

\subsection{Eccentricity Evolution}
\label{subsec:edistributions}
We simulated a range of systems that varied in separation, targeting separation ranges that should largely exhibit secular dynamical evolution. To visualize the ensemble behavior, we rely on a probability distribution-like metric that measures how often each system at each separation was at a particular eccentricity during its integration. We consider 100 eccentricity bins, uniformly spaced across the full range of bound eccentricities $0 \leq$ e $\leq 1$. Then, using the eccentricity time-series for each planet in each system, we evaluate the probability that the planet's eccentricity at any time falls within each bin, then normalize to the total number of data points in the time-series. We denote this distribution as $\mathscr{P}($e$, \alpha)$. Quantitatively, for each system (i.e. each $\alpha$), we compute
\begin{equation}
    \mathscr{P}(e_{\text{bin}}) = \frac{\text{number of counts that fall in bin}}{\text{total number of counts}}. \label{eqn:P(e)bindef}
\end{equation}
We display these distributions in a two-dimensional representation, with the separation variable $\alpha$ as the other axis, in Figures \ref{fig:planet1P}, \ref{fig:planet2P}, and \ref{fig:planet3P}. We also show the corresponding period ratio between the inner and outer planet in each pair, P$_1$ / P$_2$, as a secondary axis. These plots are color-coded so that brighter and warmer colors represent higher values of $\mathscr{P}($e$,\alpha)$. In systems where the integration terminates due to collision, close approach, or orbit crossing, we evaluate $\mathscr{P}($e$,\alpha)$ from the truncated time series data.

Deterministic secular evolution will lead to periodic, bounded oscillations in eccentricity, such that $\mathscr{P}($e$,\alpha)$ for each planet is a smooth function of $\alpha$. Secular chaos, or departures from secular evolution due to the effects of MMRs, shows up as local (in $\alpha$) deviations from the smooth trend.

\begin{figure}
    \centering
    \includegraphics[width = \columnwidth]{figures/pythonfigs/P1e_plot.png}
    \caption{
    Distribution of $\mathscr{P}($e$,\alpha$) for the innermost planet. We also give the ratio of inner and outer planet periods, P$_1$/P$_2$, as a secondary axis. Here, warmer colors indicate a larger value of $\mathscr{P}($e$,\alpha$). The range of eccentricity achieved is widest at lower $\alpha$ and narrows with increasing $\alpha$. There are notable disruptions at $\alpha \approx$ 0.202, 0.215, 0.235, 0.250, 0.261, 0.273, and 0.287. }
    \label{fig:planet1P}
\end{figure}

\begin{figure}
    \centering
    \includegraphics[width = \columnwidth]{figures/pythonfigs/P2e_plot.png}

    \caption{Distribution of $\mathscr{P}($e$,\alpha$) for the middle planet. We also give the ratio of inner and outer planet periods, P$_1$/P$_2$, as a secondary axis. Here, warmer colors indicate a larger value of $\mathscr{P}($e$,\alpha$). The range of eccentricity achieved is most narrow at lower $\alpha$, and widens slightly with increasing $\alpha$. As with that of the innermost planet, there are notable disruptions at $\alpha \approx$ 0.202, 0.215, 0.235, 0.250, 0.261, 0.273, and 0.287.}
    \label{fig:planet2P}
\end{figure}

\begin{figure}
    \centering
    \includegraphics[width = \columnwidth]{figures/pythonfigs/P3e_plot.png}

    \caption{Distribution of $\mathscr{P}($e$,\alpha$) for the outermost planet. We also give the ratio of inner and outer planet periods, P$_1$/P$_2$, as a secondary axis. Here, warmer colors indicate a larger value of $\mathscr{P}$(e$,\alpha$). The range of eccentricity achieved is very narrow at low $\alpha$ which widens very slightly with increasing $\alpha$. As with the other two planets, there are notable disruptions at $\alpha \approx$ 0.202, 0.215, 0.235, 0.250, 0.261, 0.273, and 0.287.}
    \label{fig:planet3P}
\end{figure}

\subsubsection{Planet Eccentricity Distributions}
\label{subsubsec:differences}

We first examine the $\mathscr{P}($e$,\alpha)$ distribution for the innermost planet, shown in Figure~\ref{fig:planet1P}. The smooth part of the distribution is wider near lower $\alpha$, then narrows as the separation factor increases. This width represents the maximum and minimum eccentricities achieved by these systems. At $\alpha = 0.2$, the distribution ranges between about $0.07 \leq$ e $\leq 0.34$, which narrows to a range $0.25 \leq$ e $\leq 0.4$ at $\alpha = 0.3$. The innermost planet explores a wider range of eccentricity when the system is more widely separated than when it is more compact. In particular, this planet's minimum eccentricity reaches a lower value when the system is more widely separated, while the maximum eccentricity is less affected.

The innermost planet spends more time at the extremes of its eccentricity distribution. This is evident as the brighter regions and countours along each boundary of the distribution shown in Figure~\ref{fig:planet1P}. The value of $\mathscr{P}($e$,\alpha$) is higher within these contours than within the region between them. This same behavior is also seen in the eccentricity distributions of the Solar System's giant planets, and is a natural realization of quasiperiodic evolution \citep[see e.g.][]{laskar08, mogavero17}.

The contour near the system's initial eccentricity is wider and has higher $\mathscr{P}($e$,\alpha$) than the other. These converge toward one another as the separation factor increases. Consistent with how the maximum and minimum eccentricities behave, the contour at higher e remains near-stationary in e while the contour at lower e moves significantly toward increasing e.

The corresponding plots for the middle and outermost planets are shown in Figure~\ref{fig:planet2P} and Figure~\ref{fig:planet3P}. Focusing again on the smooth part of these distributions, the width of the distributions is narrower for these planets, and they are generally centrally peaked. The width of the middle planet's eccentricity distribution broadens toward large $\alpha$ (i.e. more compact configurations). Eccentricity excursions for the outermost planet are only of modest amplitude.

At any given separation, the outermost planet explores a narrower range of eccentricities than the inner two planets. This makes sense in the secular picture. The planets are all equal mass, so the semi-major axis is the leading variable in each planet's angular momentum. As AMD is partitioned between planets, the inner planets are more susceptible to be excited to larger eccentricities, as observed.

\subsubsection{Instabilities in Planet Eccentricity Distributions}
\label{subsubsec:shared}

Departures from smooth secular evolution occur at multiple islands in $\alpha$ that are common across all three planets' eccentricity distributions. At separations of $\alpha \approx$ 0.202, 0.215, 0.235, 0.250, 0.261, 0.273, and 0.287, we see disruptions in the smooth distribution trends where each planet reaches eccentricities beyond the limits of the surrounding trend. These regions occur at the same values of separation and have the same width in $\alpha$ in each distribution, so they are a system-wide phenomenon. The extent of the excursions in eccentricities is not consistent between planets. Whatever causes this disruption affects all planets in the system but not to the same degree.

Inspection of individual time series for the simulated systems shows that the evolution can be broadly divided into three classes. Examples are shown in Figure~\ref{fig:e(t)examples}. The top panel of Figure~\ref{fig:e(t)examples} shows a system at $\alpha = 0.2102$, outside any disruption region. This system has stable oscillations in eccentricity, composed of short-period oscillations enveloped by longer-period modulations. The figure is truncated at $10^7$ years but this steady signal continues until the end of integration. The middle panel of Figure \ref{fig:e(t)examples} shows a system at $\alpha = 0.2150$, which is within a disruption region. This system also exhibits steady short-period oscillations but the longer-period modulation is disrupted and has no obvious simple periodicity. The eccentricity amplitude reaches moderately extreme values. This figure has again been truncated at $10^7$ years. Lastly, the bottom panel of Figure~\ref{fig:e(t)examples} shows a system at $\alpha = 0.2722$, within another disruption region. In this system, both the short-term and long-term oscillations have been disrupted and the system ended up having a planetary encounter just before 2.5~Myr. The planet exhibits what appears to be strongly chaotic evolution where the eccentricity attains extreme values as high as near e$=0.8$ and as low as near-zero.

\begin{comment}
\begin{figure*}
    \centering
    \begin{subfigure}{0.75\textwidth}
        \includegraphics[width=0.9\linewidth]{figures/sys34.png}
        \subcaption{}
        \label{subfig:sys34e(t)}
    \end{subfigure}
\medskip
     \begin{subfigure}{0.75\textwidth}
        \includegraphics[width=0.9\linewidth]{figures/sys50.png}
        \subcaption{}
        \label{subfig:sys50e(t)}
    \end{subfigure}
\medskip
    \begin{subfigure}{0.75\textwidth}
        \includegraphics[width=0.9\linewidth]{figures/sys240.png}
        \subcaption{}
        \label{subfig:sys240e(t)}
    \end{subfigure}
    \caption[]{Examples of the e(t) signal in several systems. \ref{subfig:sys34e(t)} shows a 'stable' signal from a system at $\alpha = 0.2102$ which lies outside any disruption region. Notice how the signal exhibits both short-term and long-term periodicity. \ref{subfig:sys50e(t)} shows a signal from a system at $\alpha = 0.215$ which lies within a disruption region. Notice that the longer-term modulations have been disrupted. \ref{subfig:sys240e(t)} shows a signal from a system $\alpha = 0.272$ which lies within a different disruption region. Notice that all periodicity has been disrupted and the system ended with an encounter after 2 Myr}
    \label{fig:e(t)examples}
\end{figure*}
\end{comment}

\begin{figure*}
    \centering
    \includegraphics[width=\linewidth]{figures/etimeseries_stacked.png}
    \caption{Examples of the e(t) signal in several systems. The top panel shows a 'stable' signal from a system at $\alpha = 0.2102$ which lies outside any disruption region. Notice how the signal exhibits both short-term and long-term periodicity. The middle panel shows a signal from a system at $\alpha = 0.215$ which lies within a disruption region. Notice that the longer-term modulations have been disrupted. The bottom panel shows a signal from a system $\alpha = 0.272$ which lies within a different disruption region. Notice that all periodicity has been disrupted and the system ended with an encounter after 2 Myr}
    \label{fig:e(t)examples}
\end{figure*}

\subsection{Resonant Interactions}
\label{subsec:resonanceinteractions}

The departures from regular secular evolution shown in Figures~\ref{fig:planet1P} through \ref{fig:planet3P} are presumptively due to a combination of secular chaos and the action of weak mean-motion resonances. Qualitatively, we observe
two degrees of disruption: one that disrupts only the long-term modulations in eccentricity, and one that disrupts all periodicity of the signal and leads to an encounter. These could be the result of two different processes, or of the same process but to two differing degrees. Not every system within a disruption regions ends with an encounter on the timescale that was tested, but every disrupted system at least exhibits the disruption of long-period modulations, which supports the latter inference.

We computed the ratio of mean-motions for each pair of planets at each separation, which are the same between each pair in each system. The solid curve in Figure~\ref{fig:mmrlocations} shows these mean-motion ratios as a function of separation, with the positions of nearby MMRs overlayed. Also indicated are the ranges of the observed disruption regions, defined based on the distributions shown in Figures~\ref{fig:planet1P}-\ref{fig:planet3P} and checking what range of systems showed disrupted evolutions around each region.

\begin{figure}
    \centering

    \includegraphics[width=\columnwidth]{figures/pythonfigs/MMRplot.png}
    \caption{
    Locations of intersections with mean-motion resonances within the range of separations tested. The solid orange curve gives the mean-motion ratio of the planetary pairs as a function of separation. The horizontal dashed lines represent the positions of nearby mean-motion resonances. The horizontal blue shared regions are the locations of the observed disruption regions in Figures \ref{fig:planet1P} - \ref{fig:planet3P}. All but one disruption region intersects exactly with a mean-motion resonance.}
    \label{fig:mmrlocations}
\end{figure}

Nearly all disruption regions align with a mean-motion resonance. From left to right (in Figure~\ref{fig:mmrlocations}), the first region intersects the 11:1 resonance, the second region intersects the 10:1 resonance, the third region intersects the 9:1 resonance, the fourth region intersects the 8:1 resonance, the fifth region intersects the 15:2 resonance, the sixth region intersects the 7:1 resonance, and the seventh region intersects the 13:2 resonance. Each intersected MMR is a chain resonance, meaning that each planet in the system is in resonance with its adjacent neighbor. Evidently, these MMR chains contribute to the observed disruption. These are all high-order resonances and are relatively weak compared to low-order resonances commonly seen in planetary systems. However, the fact that both adjacent pairs of planets share the resonance may significantly strengthen their effect.

Secular resonances occur when there is a commensurability in the $\varpi$ precession rates of a pair of planets. The $\varpi$ time series for each planet in each system exhibits near-linear circulation from -$\pi$ to $\pi$. We computed the precession rate as the slope of the first linear portion of each system's time series. In Figure~\ref{fig:precessions}, we plot the ratio of these precession rates for each adjacent pair of planets as a function of separation. The positions of nearby secular resonances and the observed disruption regions are indicated.

\begin{figure}
    \centering

    \includegraphics[width=\columnwidth]{figures/pythonfigs/pomegaprecessionplot.png}
    \caption{
    Locations of secular resonances within the range of separations tested. The solid blue curve represents the ratio of $\varpi$ precession rates for the innermost pair of planets, while the solid orange curve is the same for the outermost pair of planets. The horizontal dashed lines represent the positions of nearby secular resonances. The arrows point to the positions of the observed disruption regions. Notice that the solid curves lie along secular resonances across most of the separation range. At most disruption regions, the curves are deflected off the resonance line.}
    \label{fig:precessions}
\end{figure}

Both pairs of planets lie along a secular resonance for nearly the entire range of separations. The innermost pair are in a first order 2:3 resonance and the outermost pair are in a high-order 1:10 resonance. When a disruption region is entered, and correspondingly an MMR is intersected, the pairs appear to be knocked off from their secular resonance. This is first evident at the separation corresponding to the 11:1 MMR region and continues up to the 7:1 MMR region, but does not seem to occur for the 15:2 or 13:2 MMR. The degree to which the planets are knocked from their secular resonance mostly increases with the increasing strength (i.e. lower order) of the intersected MMR, with the 7:1 MMR region exhibiting the strongest effect. After this region, the outermost pair strays from the 1:10 secular resonance and instead continues to higher $\alpha$ along the negative 2:3 resonance.

Comparing Figures~\ref{fig:planet1P}-\ref{fig:planet3P} and \ref{fig:e(t)examples} shows that systems within the observed disruption regions have partially to fully disturbed eccentricity time series, and those outside disruption regions exhibit stable evolution. Figures \ref{fig:mmrlocations} and \ref{fig:precessions} show that systems within the disruption regions are in the chaotic space surrounding an MMR chain while those outside disruption regions are only in a secular resonance. All together, this suggests that the MMRs cause disrupted eccentricity evolution whereas those systems only in secular resonance are stable. The degree of instability is correlated to the order of the MMR. At all intersections, at least the long-term modulations in e are disrupted and the magnitude of e is driven to relatively extreme values. The higher order MMR chains are unable to fully dominate the stability enforced by the secular resonance, so these are only able to disrupt the longer-period secular modulations. As the MMRs become stronger, they become more dominant. The lower order MMRs generally affect a wider width in $\alpha$, more fully push the planets out of secular resonance, disrupt even short-term periodicity, and sometimes enable planetary encounters.

\begin{figure}
    \centering

    \includegraphics[width=\columnwidth]{figures/pythonfigs/encountersplot.png}
    \caption{Distribution of encounter times and types as function of the system's separation factor.}
    \label{fig:instability_times}
\end{figure}

A subset of the systems experience encounters over the duration of the simulations. The distribution of encounter times is shown in Figure~\ref{fig:instability_times}. There are two major groupings around $\alpha = 0.25$ and $\alpha = 0.273$ that coincide with the 8:1 and 7:1 MMR chains, which are the two strongest MMRs intersected. 15 of the 17 encounters were an orbit crossing, with 13 of these being between the outermost pair of planets.

\begin{figure}
    \centering

    \includegraphics[width=\columnwidth]{figures/pythonfigs/AMDplot.png}
    \caption{Values of the angular momentum deficit within our systems as a function of separation. The blue curve represents the initial AMD in each system and the orange curve represents the maximum AMD achieved by that system during the integration. The dotted line represents the Hot Jupiter stability condition given in equation (\ref{eqn:HJamdcondition}). The vertical shaded regions represent the positions of the observed MMR regions.}
    \label{fig:amd}
\end{figure}

The angular momentum deficit can be used as an indicator of stability \citep[e.g.][]{laskar17, petit17, wu11, tamayo21}. System AMD is conserved in the secular limit, so the initial value of AMD can forecast the future stability of a purely secular system. Secular chaos can drive massive planets to become hot Jupiters through eccentricity excitation followed by orbital decay and then eventual tidal capture by the star. This defines one limit of AMD stability: the amount of AMD that, if transferred in full to the innermost planet, would lead it to become a hot Jupiter. For a coplanar system, assuming that the orbit becomes tidally circularized by the star at $\mathrm{a}_1 \left( 1 - \mathrm{e}_1 \right) \leq 0.05~AU$, this condition is \citep{wu11}
\begin{equation}
\mathrm{AMD} \geq \Lambda_1 \left( 1 - \sqrt{ \frac{ 0.1~AU }{\mathrm{a}_1} } \right), \label{eqn:HJamdcondition}
\end{equation}
where $\Lambda_k = \frac{\mathrm{m}_k \mathrm{M}_\star}{\mathrm{m}_k + \mathrm{M}_\star} \sqrt{ G \left( \mathrm{M}_\star + \mathrm{m}_k \right) \mathrm{a}_k }$ is the circular angular momentum of the planet.
The presence of non-secular processes like nearby MMRs can be a source of additional AMD \citep{wu11}, violating AMD conservation. We find that the MMR chains present in our simulated systems do lead to the violation of AMD conservation. To show this, we plot the initial AMD in each system as well as the maximum value of AMD ever achieved by that system in Figure~\ref{fig:amd}. The horizontal dotted line is the hot Jupiter condition of equation (\ref{eqn:HJamdcondition}). The initial AMD slowly decreases with decreasing separation, which is to be expected.
The initial AMD of every system is below the condition for the onset of hot Jupiter formation, so these systems are all initially AMD-stable. In most systems, the AMD is conserved throughout the integration as the maximum AMD achieved tracks the initial AMD. However, there are several spikes in the maximum AMD curve that align with the intersected mean-motion resonance chains, indicated as the shaded regions in Figure \ref{fig:amd}. Within the intersections of the 9:1, 8:1, 7:1, and 13:2 MMR chains, the maximum AMD surpasses the condition for the onset of hot Jupiter formation. This shows that chains of high order mean-motion resonances can be a significant source of AMD in widely separated systems and lead to instability in an initially AMD-stable system, albeit only for a small region of parameter space.

\subsection{Time Series Analysis}

The defining feature of chaos is a loss of direct determinism, due to the extreme sensitivity of the final state of the system on its initial conditions. To identify the presence of chaos in the simulated systems, we employ an analysis that is based upon a Hilbert transform of the time series data \citep{liu10}. The Hilbert transform takes a real time series, $q(t)$, and computes a corresponding imaginary time series, $\Tilde{q}(t)$, through the transformation
\begin{equation}
    \tilde{q} (t) =  P.V. \left[ \frac{1}{\pi} \int_{- \infty }^{\infty } \frac{ q (t^{'} ) }{ t - t^{'} } d t^{'} \right] ,
\end{equation}
where P.V. represents the Cauchy principal value. Then, the real and imaginary series form a so-called \textit{analytic} signal, $\psi(t)$, which can be written as a rotation in complex space as
\begin{equation}
\psi (t) = q (t) + i \Tilde{q} (t) = A(t) e^{i \phi (t) } .
\end{equation}
The amplitude is
\begin{equation}
    A(t) = \sqrt{ {q(t)}^2 + {\Tilde{q}(t)}^2 },
\end{equation}
and the phase variable is \citep{liu10}
\begin{equation}
    \phi (t) = \arctan \left( \frac{ \Tilde{q} (t) } { q (t) } \right) .
\end{equation}
Chaos can show up as either irregular modulations in the amplitude or phase of the signal. By transforming into Hilbert space and determining the amplitude and phase variable of the analytic signal, we separate and can independently characterize the effects of amplitude and phase modulation. We expect that the components for stable and chaotic systems will exhibit quantitatively different behavior.

\begin{figure}
    \centering
    \includegraphics[width=\columnwidth]{figures/pythonfigs/HilbertPhaseplot.png}
    \caption{The maximum value achieved by the Hilbert transform phase variable in each system. In systems which intersect a mean motion resonance chain, the phase variable reaches upwards of $\pi$/2, whereas systems outside of mean motion resonance remain bounded at lower values.}
    \label{fig:maxphases}
\end{figure}

We computed the Hilbert transformation for the inner planet's eccentricity time series for each system. The amplitude series resemble that of the eccentricity series, so their distributions reflect Figures~\ref{fig:planet1P}-\ref{fig:planet3P} and do not reveal new information. The phase variables, however, show significant differences. Each phase variable series librates in time. We plot the maximum phase achieved by each series, representing the limits of that libration, in Figure~\ref{fig:maxphases}. We see markedly different behavior between the stable systems and the systems which intersect an MMR chain. The systems which intersect the chaotic region around an MMR approach a maximum limit phase of $\pi / 2$. The stable achieve a lower limit, around $0.7$ for low $\alpha$ which decreases to a limit of $\approx \pi / 8$ as $\alpha$ increases.

The eccentricity phase variable of the stable systems is unwrapped, meaning the eccentricity evolution behaves as a multi-valued function. Slightly different input conditions can, in general, evolve to the same end states.
In contrast, we find that the MMR intersecting systems have a wrapped phase variable with a principal value of $\pi / 2$. This is an indicator of chaos, as each possible initial condition of such a system will have a near-unique end state.

\section{Summary \& Discussion}
\label{sec:discussion}

In this work, we set out to investigate the interplay between secular and resonant interactions in model systems of three eccentric, equal mass (1~M$_{\rm J}$, for a Solar mass host), widely separated planets. We simulated 333 coplanar systems in which the planets were initially separated by factors of 3.33-5 in semi-major axes. A summary of our findings is as follows:
\begin{itemize}
    \item
    Systems that were not in mean-motion resonance remained stable for the duration of the integrations ($10^8$ orbits of the innermost planet). Evolution of these systems was consistent with deterministic secular dynamics.
    \item
   Systems within the chaotic region surrounding high-order mean-motion resonant chains became unstable, evidenced primarily by a loss of periodic evolution. The stronger mean-motion resonances also led to planetary encounters, including orbit crossings and close encounters.
    \item
    The high-order mean-motion resonant chains provided a significant source of angular momentum deficit, that was sufficient in many cases to drive initially AMD-stable systems to become unstable.
\end{itemize}

\subsection{Comments on the Initial Conditions}
\label{subsec:initcondcomments}

We chose to initialise our systems with aligned periapsis longitudes, such that $\varpi_{init}$ = 0 for all planets. This choice did not have a significant impact on the results of our simulations. To test the effect of this choice, we ran additional simulations in which adjacent planets had anti-aligned periapsis longitudes. The initially anti-aligned systems evolved nearly the same as the initially aligned systems. The only major difference was that, in some cases, the limits of a planet's eccentricity time-series where shifted -- e.g. in the aligned case, the eccentricity oscillated between the initial value and a minimum with some amplitude and, in the anti-aligned case, the eccentricity oscillated between the initial value and a maximum instead with the same amplitude. Also, we specifically tested systems at the boundaries of the chaotic regions surrounding the MMRs and found that switching to the anti-aligned case did not extend these regions in the $\alpha$ direction.

The even spacing of the planets in $\log$(semi-major axis), means that we cannot generalize our results to arbitrary massive planets evolving in the nominally secular regime. Instability driven solely by secular processes is well studied, particularly in the context of hot Jupiter formation \citep[see e.g.][]{wu11,lithwick13, laskar17,teyssandier19}. The majority of these studies have explicitly avoided the presence of MMRs, while some have admitted first-order MMRs. \cite{wu11} showed that a first-order MMR aided AMD diffusion between planets and proposed that such resonances could be a significant source of AMD. \cite{petit17} expanded the framework of AMD-stability to account for the effects of first-order MMRs and found that, when applied to observed exoplanetary systems, a small percentage of otherwise AMD-stable systems were actually unstable due to the presence of mean-motion resonances. Our work supports the hypothesis that higher-order resonant interactions may also contribute to instability.

\subsection{Discussion of Results and Implications}
\label{subsec:implications}
We have found that secular and resonant processes are jointly important for determining the stability of the class of systems that we simulated. Among observed exoplanets, there are examples of systems whose dynamical stability results from the presence of both secular and mean-motion resonances. A combination of secular resonance and a first-order 2:1 MMR contribute to stability in the HD 160691 system \citep{kiselva04}, the GJ 876 system \citep{lee02}, and the HD 82943 system \citep{Hadj03}. Similarly, secular resonance in addition to a nearby high-order 11:2 MMR contribute to the stability of the HD 12661 system \citep{Gozd03}. On the other hand, there are also examples of exoplanetary systems where both secular and mean-motion resonances contribute to instability and chaotic orbits. This has been particularly observed in planets in binary star systems \citep{satyal14,sutherland19}. Considerably less work has been done on systems in which stabilizing secular interactions are overcome by MMRs. This type of interaction has been shown for small bodies within the Solar System, such as the Quadrantid meteors \citep{sambarov20}. Our work has shown that such an interaction can occur between planets in a massive, widely separated, initially eccentric system.

Determining the space of initial conditions that would evolve under dynamical processes to form the observed population of planetary systems remains an open theoretical challenge. Within the framework of the Nice model \citep{tsiganis05}, statistical constraints have been placed on the initial configuration of the outer Solar System \citep{nesvorny12,clement21}, and current theoretical tools \citep{tamayo20,cranmer21} may be adequate to enable similar studies of the population of compact low-mass planetary systems, where instability is driven primarily by resonant dynamics. More work, however, is needed to quantify the interplay of secular and resonant effects in more general classes of planetary system, and from that to infer constraints on planet formation models from observed exoplanet populations.

\section*{Data availability}

The original data underlying this article are available upon reasonable request to the authors.

\section*{Acknowledgements}

The authors would like to thank our anonymous journal reviewer and scientific editor for their positive feedback and useful suggestions which helped improve this work. The authors would also like to thank Stony Brook Research Computing and Cyberinfrastructure, and the Institute for Advanced Computational Science at Stony Brook University for access to the high-performance SeaWulf computing system, which was made possible by a \$1.4M National Science Foundation grant (\#1531492).

This work was partially supported with funding from the Stony Brook University Undergraduate Research \& Creative Activities program. Simulations in this paper made use of the REBOUND N-body code \citep{rebound}. The simulations were integrated using WHFast, a symplectic Wisdom-Holman integrator \citep{reboundwhfast,wh}.

\bibliographystyle{mnras}


\section{Introduction}
\label{sec:Introduction}
\subsection{Overview and Motivation}
In this paper, we discuss a family of iterative Gauss-Newton shooting methods for numerically solving unconstrained optimal control problems, and illustrate the effectiveness of our algorithms with various robotics examples.
We outline the connection between a number of `direct' optimal control methods and Gauss-Newton methods from the class of Differential Dynamic Programming (DDP)~\cite{mayne1966ddp} algorithms.
Additionally, we present a natural extension arising from this connection and introduce a family of hybrid Gauss-Newton Multiple Shooting methods.

In direct approaches to optimal control, infinite-dimensional optimal control problems are transcribed into finite dimensional Nonlinear Programs (NLPs). Two prominent ways of transcription by `shooting' are direct single shooting (SS) and direct multiple shooting (MS)~\cite{bock1984direct}, which differ in the choice of decision variables.
In single shooting, solely the control inputs are the decision variables. Generally speaking, the control trajectory is discretized in a piece-wise polynomial fashion (for simplicity, we focus on piece-wise constant controls in this paper, c.f. Fig.~\ref{fig:algorithms}a). A corresponding state trajectory is obtained by means of numerical forward integration of the system dynamics, starting at a given initial state. SS is often called a `sequential' approach.
In multiple shooting, the same discretization scheme is employed for the control inputs, but additionally, intermediate states are added to the decision variables. This provides several advantages~\cite{diehl2006fast}, but requires the introduction of additional matching constraints to ensure continuity of the state trajectory.
The technique of introducing these additional degrees of freedom into the original problem, combined with adding matching constraints, is called \emph{lifting}~\cite{albertsmeyerLifted}, and results in a `simultaneous' method.

The formulation of both SS and MS as standard NLPs is straightforward and any state-of-the-art NLP solvers can be used to solve them.
It is important to note that under the assumption of having a piece-wise polynomial control parameterization, the intrinsic sparsity structure of the underlying optimal control problem of both SS and MS allow them to achieve linear time complexity by performing a Riccati recursion~\cite{Rao1998application}.

Classical single shooting often does not perform well for unstable systems due to the pure open-loop forward integration of the system dynamics. In DDP, this is handled by doing a closed-loop forward integration, using a feedforward plus a time-varying state-feedback control law. The Riccati backward sweep designs time-varying feedback gains on the fly without additional computational cost.
DDP is an exact-Hessian method, requiring the computation of second derivatives of the dynamics. While this gives the algorithm quadratic convergence, this can be impractical for use in systems with complex dynamics. For that reason, Hessian-approximating variants of DDP have become quite popular in the robotics community~\cite{koenemann2015whole,tassaSynthesis,neunert:2017:ral,ponton2016risk,farshidian16efficient}.

An important Hessian-approximating variant of DDP is the \emph{iterative Linear-Quadratic Regulator} (iLQR)~\cite{todorov2005ilqg}, which is also known as \emph{Sequential Linear Quadratic Optimal Control}~\cite{slq:2005}. This method can be classified as closed-loop single shooting using a Gauss-Newton Hessian approximation and a Riccati backward sweep to solve linear-quadratic (LQ) subproblems. The Gauss-Newton Hessian approximation is based on the assumption that the objective function can be locally approximated as a sum of quadratic terms, and requires only first-order derivatives of the system dynamics. This comes at the cost of giving only linear convergence, however.
The Gauss-Newton approach can be lifted, too, which has for example been shown in~\cite{schaeferThesis}. While it initially appears to be a drawback to increase the number of decision variables, it is important to emphasize that the lifted problem can be solved at approximately the same computational cost as the original non-lifted problem, and can lead to a significant increase of convergence speed~\cite{albertsmeyerLifted}.
Therefore, the fundamental motivation for this paper is to combine the benefits of iLQR with a multiple-shooting approach.

\subsection{Contribution}
\label{sec:contribution}
In this work, we derive a lifted equivalent of iLQR, called \emph{Gauss-Newton Multiple Shooting} (GNMS), which introduces the intermediate states as additional decision variables.
Next, we extend this relationship to form an entire family of open-loop multiple-shooting algorithms, denoted GNMS($M$), and closed-loop multiple shooting algorithms, denoted as iLQR-GNMS($M$). The latter is shown to be a generalization of iLQR and can be considered multiple-shooting iLQR. We outline the relationship between these algorithms and existing methods. We give simulation examples including a complex underactuated robot and compare the performance of the full-step algorithms using data gained from hardware experiments.
Furthermore, we show the benefits of iLQR-GNMS($M$) methods for nonlinear model predictive control.

\subsection{Outline}
This paper is structured as follows.
In Section~\ref{sec:gnms}, we derive GNMS, and present the basic update routine for state and control trajectories. Using these update equations, we generalize iLQR and GNMS to a family of algorithms in Section~\ref{sec:algorithms}.
Section~\ref{sec:simulation_results} showcases several simulation results, based on data gained from hardware experiments.
A discussion and outlook concludes the paper in Section~\ref{sec:Discussion}.

\section{Gauss-Newton Multiple Shooting}
\label{sec:gnms}

\begin{figure}
\centering
\includegraphics[width=0.99\columnwidth]{pics/shorterShots.pdf}
\caption{Intuition about different shooting variants.
(a) A zero-order hold control parameterization, with a constant control input at each stage.
(b) Single shooting, where the state trajectory is obtained through a single forward integration over the whole problem horizon.
(c) In GNMS, intermediate states at every time-index $n$ are introduced as additional decision variables.
(d) In hybrid versions of GNMS, the multiple-shooting intervals span several control intervals. The intermediate states at the beginning of the multiple-shooting intervals are decision variables, and the states in between are obtained by forward integration.
}\label{fig:algorithms}
\end{figure}

In the following, a brief derivation of the unconstrained Gauss-Newton Multiple Shooting method is presented. We show the derivation using an intuitive value-function approach in the style of~\cite{todorov2005ilqg} in order to highlight the close relationship between GNMS and iLQR. However, from the beginning, we lift the optimization problem and introduce intermediate states as additional decision variables besides the controls. In that sense, having each control decision variable accompanied with a state-decision variable, GNMS is closely related to the original multiple-shooting algorithm~\cite{bock1984direct}.

Consider the following discrete-time, finite-horizon, nonlinear optimal control problem
\begin{align}
&\min_{\mathbf u_n, \mathbf x_n} \left \{\Phi(\mathbf x_{N})+\sum_{n=0}^{N-1} L_n (\mathbf x_n,\mathbf u_n, n) \right \}
\label{eq:nonlinear_cost}\\
& \textrm{s.t.} \qquad \mathbf x_{n+1} - \mathbf f_n(\mathbf x_n,\mathbf u_n, n) = 0,\hspace{4mm} \mathbf x_0=\mathbf x_{init}
\label{eq:nonlinear_dynamics}
\end{align}
with state-vector $\mathbf{x}_n \in \mathbb R^m$ and control input vector $\mathbf{u}_n \in \mathbb R^p$. Let $L_n$ be the intermediate cost at time-step $n$ and $\Phi(\mathbf{x}_N)$ the terminal cost at the time horizon~$N$.

\subsection{Forming LQ subproblems}
\label{sec:formingLQsubproblems}
The optimal control law is computed in an iterative way. In each
iteration $k=0,1,2,\ldots$, we construct a LQ optimal control problem around the state trajectory
$\mathbf X^{[k]} = \{ \mathbf x_0^{[k]}, \mathbf x_1^{[k]}, \ldots, \mathbf x_N^{[k]} \}$, with \mbox{$\mathbf x_0^{[k]}=\mathbf{x}_{init}$} and the control trajectory
$\mathbf U^{[k]} = \{ \mathbf u_0^{[k]}, \mathbf u_1^{[k]}, \ldots, \mathbf u_{N-1}^{[k]} \}$.
In the first iteration, $k=0$, the LQ problem is hence constructed around the initial guesses for $\mathbf X^{[0]}$, $\mathbf U^{[0]}$. Possible initialization strategies are summarized in Section~\ref{sec:initialization}.

At each iteration, we numerically forward integrate all multiple shooting intervals using the respective control inputs $\mathbf u_n^{[k]}$, starting at every state $\mathbf x_n^{[k]} \ \forall n=0,1,\ldots, N-1$.
Fig.~\ref{fig:algorithms}c shows a sketch of the multiple-shooting intervals in GNMS, where the resulting state at the end of each interval is denoted $\mathbf F^{[k]}(\mathbf x_n^{[k]}, \mathbf u_n^{[k]})$. Accordingly, we define the `defect' between the integrated trajectory segment and the next intermediate state $\mathbf x_{n+1}^{[k]}$ as
\begin{equation}
 \mathbf d_n^{[k]} = \mathbf F^{[k]}(\mathbf x_n^{[k]}, \mathbf u_n^{[k]}) - \mathbf x_{n+1}^{[k]} \ \text{.}
 \label{eq:defect}
\end{equation}
Defining state and control increments $\delta \mathbf x_n^{[k]}$ and $\delta \mathbf u_n^{[k]}$ for every single time-stage $n$, we can write the nonlinear system dynamics constraint~\eqref{eq:nonlinear_dynamics} in terms of the simulated interval as
\begin{equation}
\mathbf x_{n+1}^{[k]} + \delta \mathbf x_{n+1}^{[k]} - \mathbf F^{[k]}(\mathbf x_n^{[k]} + \delta \mathbf x_n^{[k]}, \mathbf u_n^{[k]} + \delta \mathbf u_n^{[k]}) = 0
\label{eq:matching_condition}
\end{equation}
which can also be considered a matching condition which ensures the continuity of the state trajectory w.r.t. state and control increments. Performing a first-order Taylor expansion of Equation~\eqref{eq:matching_condition} w.r.t. $\delta \mathbf x_n^{[k]}$ and $\delta \mathbf u_n^{[k]}$, denoting the sensitivities w.r.t state and control $\mathbf A_n$ and $\mathbf B_n$ and taking into account the defects as defined by Equation~\eqref{eq:defect}, results in the following affine system dynamics constraint
\begin{equation}
\delta \mathbf x_{n+1}^{[k]} - \mathbf A_n^{[k]} \delta\mathbf x_n^{[k]} - \mathbf B_n^{[k]} \delta\mathbf u_n^{[k]} - \mathbf d_n^{[k]} = 0  \ \textrm{.}
\label{eq:affine_constraint}
\end{equation}

Analogously performing a second-order Taylor expansion of the nonlinear cost function~\eqref{eq:nonlinear_cost} gives rise to the following LQ optimal control problem
\begin{align}
\min_{\delta \mathbf u_n, \delta \mathbf x_n} \bigg\{&q_N +\delta \mathbf x_N^\top\mathbf q_N+\tfrac{1}{2}\delta \mathbf x_N^\top\mathbf Q_N\delta \mathbf x_N  \notag \\
	&+\sum_{n=0}^{N-1} q_n+ \delta \mathbf x_n^\top\mathbf q_n + \delta \mathbf u_n^\top \mathbf r_n+ \tfrac{1}{2}\delta \mathbf x_n^\top\mathbf Q_n\delta \mathbf x_n \notag  \\
	& \quad +\tfrac{1}{2}\delta \mathbf u_n^\top\mathbf R_n\delta \mathbf u_n + \delta \mathbf u_n^\top\mathbf P_n\delta \mathbf x_n \bigg\}
	\label{eq:linearized_cost}\\
\textrm{s.t.} \quad & \delta \mathbf x_{n+1} = \mathbf A_n \delta\mathbf x_n + \mathbf B_n\delta\mathbf u_n +\mathbf d_n \label{eq:affine_system} \\
&\mathbf x_0=\mathbf x_{init} \label{eq:initial_condition}
\end{align}
where we assume $\mathbf Q_n\textrm{,} \  \mathbf Q_N \geq 0$ and $\mathbf R_n > 0$. Here, and in the following subsection, we drop the superscript indices~$[k]$ for better readability.

\subsection{Computing the Optimal Control by Riccati Recursion}
\label{sec:riccati_recursion}
Considering the LQ subproblem~\eqref{eq:linearized_cost}-\eqref{eq:initial_condition}, the optimal control and state updates can be computed using a value-function approach. Assume a quadratic value function of the form
\begin{equation}
\label{eq:cost_to_go}
V_{n}(\delta \mathbf x_{n}) = s_{n} + {\delta \mathbf x_{n}^\top \mathbf s_{n}}
 + \tfrac{1}{2} \delta \mathbf x_{n}^\top \mathbf S_{n} \delta \mathbf x_{n}
\end{equation}
with weighting matrices $\mathbf S_n \in \mathbb R^{m \times m}$, $\mathbf s_n \in \mathbb R^{m \times 1}$ and \mbox{$s_n \in \mathbb R$}. The optimal control update can be derived by minimizing the value function $V_n$ as a function of $\delta \mathbf x_n$.

As Equation~\eqref{eq:cost_to_go} is quadratic in $\delta \mathbf x_{n+1}$ at time $n+1$, it remains quadratic during back-propagation in time, given the affine system dynamics and the linear-quadratic cost in Equations~\eqref{eq:linearized_cost}-\eqref{eq:initial_condition}. Due to Bellman's Principle of Optimality, the optimal control $\delta \mathbf u_n^*$ at time $n$ can be computed from
\begin{align*}
&V^*_n(\delta \mathbf x_n) = \min_{\delta \mathbf u_n} \bigg[ q_n + \delta \mathbf x^\top (\mathbf q_n + \tfrac{1}{2} \mathbf Q_n \delta \mathbf x_n)
+ \delta \mathbf u_n^\top \mathbf P_n \delta \mathbf x_n  \notag \\
&+ \delta \mathbf u_n^\top(\mathbf r_n + \tfrac{1}{2} \mathbf R_n \delta \mathbf u_n)
+ V^*_{n+1}(\mathbf A_n \delta \mathbf x_n + \mathbf B_n \delta \mathbf u_n +\mathbf d_n) \bigg]
\end{align*}
Inserting Equation~\eqref{eq:cost_to_go} and the affine system dynamics~\eqref{eq:affine_system} and minimizing the overall expression w.r.t. $\delta \mathbf u_n$ leads to an optimal control update of the form
\begin{equation}
\delta \mathbf u_n^*= -\mathbf H_n^{-1} \mathbf h_n - \mathbf H_n^{-1} \mathbf G_n \delta \mathbf x_n
\label{eq:optimal_control_input}
\end{equation}
where we have defined
\begin{align*}
\mathbf h_n & = \mathbf r_n + \mathbf B_n^\top (\mathbf s_{n+1} + \mathbf S_{n+1} \mathbf d_n )\notag\\
\mathbf G_n & = \mathbf P_n + \mathbf B_n^\top \mathbf S_{n+1} \mathbf A_n \notag\\
\mathbf H_n & = \mathbf R_n + \mathbf B_n^\top \mathbf S_{n+1} \mathbf B_n
\end{align*}
and $\mathbf B_n^\top \mathbf S_{n+1} \mathbf B_n + \mathbf R_n > 0$.
After equating coefficients with a quadratic value function ansatz~\eqref{eq:cost_to_go} for $\delta \mathbf x_n$, we define
\mbox{$\mathbf l_n = -\mathbf H_n^{-1} \mathbf h_n$}
and
\mbox{$\mathbf L_n = - \mathbf H_n^{-1} \mathbf G_n$}
and obtain the following recursive Riccati difference equations for $\mathbf S_n$, $\mathbf s_n$ and $s_n$
\begin{align}
\mathbf S_n &= \mathbf Q_n + \mathbf A_n^\top \mathbf S_{n+1} \mathbf A_n - \mathbf L_n^\top \mathbf H_n \mathbf L_n \label{eq:riccati_vS}\\
\mathbf s_n &= \mathbf q_n + \mathbf A_n^\top (\mathbf s_{n+1} + \mathbf S_{n+1} \mathbf d_n) \notag \\& \qquad + \mathbf G_n^\top \mathbf l_n + \mathbf L_n^\top (\mathbf h_n + \mathbf H_n\mathbf l_n) \label{eq:riccati_vs}\\
s_n &= q_n + s_{n+1} + \mathbf d_n^\top \mathbf s_{n+1} + \tfrac{1}{2} \mathbf d_n^\top \mathbf S_{n+1} \mathbf d_n \notag \\
& \qquad + \mathbf l_n^\top (\mathbf h_n + \tfrac{1}{2} \mathbf H_n \mathbf l_n) \label{eq:riccati_s}
\end{align}
for $n \in {0,\ldots, N-1}$. For the final time-step~$N$ we obtain the terminal conditions \mbox{$\mathbf S_N = \mathbf Q_N$}, \mbox{$\mathbf s_N = \mathbf q_N$} and \mbox{$s_N = q_N$}, and the recursion is subsequently swept backwards. Note that Equation~\eqref{eq:riccati_s} does not contribute to the control update and can therefore be omitted in practice.

\subsection{Updating State and Control Trajectories}
Finally, using Equations~\eqref{eq:affine_system} and~\eqref{eq:optimal_control_input}, and readopting the superscript indices $[k]$ for the iteration count, we obtain equations for a forward sweep resulting in a full-step update for the control and state decision variables $\mathbf X^{[k+1]}$, $\mathbf U^{[k+1]}$
\begin{align}
\mathbf u_n^{[k+1]} &= \mathbf u_n^{[k]} + \mathbf l_n^{[k]} + \mathbf L_n^{[k]} (\mathbf x_n^{[k+1]} - \mathbf x_n^{[k]}) \label{eq:gnms_control_update}\\
\mathbf x_{n+1}^{[k+1]} &= \mathbf x_{n+1}^{[k]} + (\mathbf A_n^{[k]} + \mathbf B_n^{[k]} \mathbf L_n^{[k]})(\mathbf x_n^{[k+1]} - \mathbf x_n^{[k]}) \notag \\
& \quad + \mathbf B_n^{[k]} \mathbf l_n^{[k]} + \mathbf d_n^{[k]} \label{eq:gnms_state_update}
\end{align}
with initial condition \mbox{$\mathbf x_0^{[k+1]} = \mathbf x_{init}$}. The updated decision variables are dynamically consistent w.r.t. the LQ subproblem dynamics.
The nonlinear optimal control problem is solved iteratively, starting from Section~\ref{sec:formingLQsubproblems} and solving LQ subproblems at each iteration, until convergence.

\section{A Family of iLQR-GNMS Algorithms}
\label{sec:algorithms}
Equations~\eqref{eq:gnms_control_update} and~\eqref{eq:gnms_state_update} present the GNMS update rule where all states and controls (except for $\mathbf x_{init}$) are decision variables. For every time-step, both states and controls are updated using a linear forward sweep.
Considering Equations~\eqref{eq:gnms_control_update} and~\eqref{eq:gnms_state_update}, we can now draw connections between GNMS and other existing algorithms and extend them to a bigger family of `hybrid' variants.
\subsection{Connection to iLQR and Single Shooting}
Interestingly, full-step iLQR employs the very same control update rule as in Equation~\eqref{eq:gnms_control_update}. In fact, GNMS can be transcribed into iLQR by substituting the state update equation~\eqref{eq:gnms_state_update} with a numeric forward integration of the nonlinear system~\eqref{eq:nonlinear_dynamics} using the time-varying state-feedback control law provided by Equation~\eqref{eq:gnms_control_update}.
In this case, the forward integration naturally results in a dynamically consistent state trajectory, all defects $\mathbf d_n$ become zero and the formulation from section~\ref{sec:riccati_recursion} drops back to the well-known iLQR Riccati recursion.
Moreover, standard unconstrained single shooting can be recovered by additionally ignoring the state feedback gains and running the forward-integration purely open-loop.

\subsection{Hybrid Algorithms}
Consider a case where the overall time horizon $N$ is split into an integer number of multiple shooting intervals $M$ with length $l$ and \mbox{$1<M<N$}, while the control input discretization is kept at its original resolution.
Without loss of generality, let us assume that the MS integration intervals start at time indices $i \in \mathcal I$, with $\mathcal I = \{0, l, 2l,\ldots \}$. Fig.~\ref{fig:algorithms}d sketches an example of such a hybrid case with $l=3$.
Every interval is simulated using the nonlinear system dynamics~\eqref{eq:nonlinear_dynamics} and the initial states and controls $\mathbf x_i^{[k]}$ and $\mathbf u_i^{[k]}$. All $\mathbf x_j^{[k]}$ with $j \notin \mathcal I$ are \emph{overwritten} by the integration. For an open-loop forward integration, $\mathbf U^{[k]}$ remains as is, but for a closed-loop forward integration, we additionally overwrite all $\mathbf u_j^{[k]}$ with $j \notin \mathcal I$ using the given feedback control law. Note that in this case, the defect equation~\eqref{eq:defect} remains valid, but is zero along the multiple-shooting intervals. The only non-zero defects occur at $\mathbf d_{i+l-1}^{[k]}, i \in \mathcal I$.
In this setting, the LQ approximation, Riccati recursion and state- and control updates~\eqref{eq:gnms_control_update}-\eqref{eq:gnms_state_update} can be performed as described before.
This gives rise to two `hybrid' GNMS variants:
\begin{itemize}
\item \emph{GNMS(M)}, using solely the feedforward control and thus performing an open-loop forward integration on each of the $M$ multiple shooting intervals, which themselves are multiples of the control interval. Herewith, standard single shooting is the limit case GNMS(1).
\item \emph{iLQR-GNMS(M)}, using the full state feedback controller~\eqref{eq:gnms_control_update} and a closed-loop forward integration of each multiple-shooting interval. In other words, this is equivalent to a multiple-shooting variant of iLQR. The standard iLQR algorithm is the limit case of iLQR-GNMS(1), with only one multiple-shooting interval.
\end{itemize}
Note that both GNMS($N$) and iLQR-GNMS($N$), with the number of multiple shooting-intervals being equal to the number of stages, revert to the standard GNMS formulation as introduced in Section~\ref{sec:gnms}.
Table~\ref{tab:compTable} provides a compact overview of the algorithmic variants and compares their features.

\begin{table}
\caption{An overview of different GNMS-type methods.}
\label{tab:compTable}
\footnotesize
\tabcolsep=0.11cm
\begin{tabular}{ |m{7em} |c|c|c|c|c|  }
\hline
            & SS    & iLQR  & GNMS  & GNMS($M$) & iLQR-GNMS($M$) \\ \hline \hline
closed-loop & --     & \checkmark     & --     & --         & \checkmark\\ \hline
No. intervals   & 1     & 1     & N     & M         & M\\ \hline
overwrite states by integration & \checkmark & \checkmark  & -- & \checkmark & \checkmark\\ \hline
need stable initial policy & \checkmark & \checkmark  & -- & depends & depends\\ \hline
\end{tabular}
\vspace{1ex}

\raggedright (\checkmark = true, -- = false)
\end{table}

\subsection{Main Iteration and Implementation}
We emphasize that all algorithmic variants feature linear complexity in the time horizon, $O(N)$. All algorithms execute almost identical linear algebra operations during one major iteration and therefore have very similar computational effort.
Since the discussed family of GNMS algorithms only differs in a few features, it can be summarized in one framework, given in Algorithm~\ref{alg:general_algorithm}.
From a software-engineering perspective, the algorithmic variants are easy to implement and can all be treated at once, given a proper design of classes and interfaces. We provide an open-source C++ implementation of all discussed algorithms in~\cite{adrlCT}.

\subsection{Initialization}
\label{sec:initialization}
The GNMS variants listed in Table~\ref{tab:compTable} differ in their requirements for initialization.
For iLQR and SS, the nominal state (and control) input trajectories are first updated through a forward integration.
This implies that for unstable systems, an initialization with a stabilizing initial control policy, which keeps the first rollout in the vicinity of the expected optimum, is essential.
For iLQR, the initially provided state trajectory $\mathbf X^{[0]}$ serves as state reference trajectory for the feedback controller. For SS, it is irrelevant, except for the initial state. Common choices for SS and iLQR initial guesses are policies that stabilize the given initial state or draw the system towards the goal state, for example simple LQR or PD controllers. Generally, the increased efforts for initial guess design for SS and iLQR can be a significant disadvantage. In the worst case, a poor initial guess can lead to a local minimum with a solution far from desired behavior.

Multiple-shooting algorithms, by contrast, offer greater flexibility and simplicity at initial guess design, and are often more robust w.r.t. bad initial policies. It is a well known fact that the convergence of multiple-shooting methods can be accelerated through an `educated' initial guess, such as direct interpolation between initial and desired final state. For the hybrid algorithms iLQR-GNMS($M$) and GNMS($M$) it often depends on the system characteristics if a stabilizing control policy is required, or if the multiple-shooting intervals are short enough to prevent significant divergence during integration.
In the video attachment~\cite{videoAttachment}, we show two simulation examples where initialization with a bad state-feedback controller significantly extends the runtime of iLQR compared to GNMS, or even causes iLQR to fail.

Note that, when all possible GNMS variants are initialized  with a dynamically consistent state trajectory and corresponding control trajectory, the defects for the first iteration are zero and the feedforward control updates are identical.

\begin{algorithm}[tpb]
\caption{Generalized iLQR-GNMS($M$) Algorithm}
\label{alg:general_algorithm}
\begin{algorithmic}
\scriptsize
\STATE \textbf{Given}
\STATE - Nonlinear dynamics, cost function and initial state $\mathbf x_{init}$ as given in~\eqref{eq:nonlinear_cost}-\eqref{eq:nonlinear_dynamics}
\STATE - Initial state trajectory $\mathbf X^{[0]}$
\STATE - Initial feedforward trajectory $\mathbf U^{[0]}$
\STATE - (Initial feedback law, if applicable)
\STATE - maximum total constraint violation $d_{max}$
\STATE - minimum relative cost change $J^{rel}_{min}$
\STATE \textbf{Prepare}
\STATE - set iteration count $k=0$
\STATE - split time horizon $N$ into $M$ MS integration intervals of length $l$,
\STATE \hspace{1em}each starting at  an index $i \in \mathcal I$, $\mathcal I = \{0, l, 2l,\ldots \}$
\STATE - \textbf{Initial multiple-shooting rollouts}
\STATE \hspace{1em} - simulate $M$ intervals using the nonlinear system dynamics~\eqref{eq:nonlinear_dynamics} and initial
\STATE \hspace{2em} states and controls $\mathbf x_i^{[0]}$ and $\mathbf u_i^{[0]}$, overwrite all $\mathbf x_j^{[0]}$ and $\mathbf u_j^{[0]}$ for $j \notin \mathcal I$
\STATE \hspace{1em} - compute defects $\mathbf d_n^{[0]}$ according to Equation~\eqref{eq:defect}.
\STATE \textbf{Repeat (main iteration)}
\STATE \hspace{1em} \textbf{LQ approximation}
\STATE \hspace{1em} - Linearize the dynamics along the trajectories, obtain the affine constraint~\eqref{eq:affine_constraint}
\STATE \hspace{1em} - Quadratize cost function along the trajectories to obtain~\eqref{eq:linearized_cost}
\STATE \hspace{1em} \textbf{Riccati backward sweep}
\STATE \hspace{1em} - Backwards solve the Riccati-like difference equations~\eqref{eq:riccati_vS}-\eqref{eq:riccati_vs} with
\STATE \hspace{2em} boundary conditions $\mathbf S_N = \mathbf Q_N$ and $\mathbf s_N = \mathbf q_N$
\STATE \hspace{1em} \textbf{Linear forward sweep}
\STATE \hspace{1em} - compute state and control solution candidates $\mathbf X^{[k+1]}$ and $\mathbf U^{[k+1]}$ by
\STATE \hspace{2em} forward sweeping Equations~\eqref{eq:gnms_control_update} and~\eqref{eq:gnms_state_update}.
\STATE \hspace{1em} \textbf{Rollout multiple-shooting intervals}
\STATE \hspace{1em} \textbf{IF}(\textit{open loop shooting})
\STATE \hspace{1em} \hspace{1em} - set feedback gain in Equation~\eqref{eq:gnms_control_update} to zero.
\STATE \hspace{1em} \textbf{ENDIF}
\STATE \hspace{1em} - simulate $M$ shooting intervals using nonlin. dynamics~\eqref{eq:nonlinear_dynamics}, controller~\eqref{eq:gnms_control_update}
\STATE \hspace{1em} \hspace{1em} and initial states and controls $\mathbf x_i^{[k+1]}$ and $\mathbf u_i^{[k+1]} \in \mathcal I$,
\STATE \hspace{1em} \hspace{1em} overwrite all $\mathbf x_j^{[k+1]}$ and $\mathbf u_j^{[k+1]}$ for $j \notin \mathcal I$
\STATE \hspace{1em} - compute defects $\mathbf d_n^{[k+1]}$ according to Equation~\eqref{eq:defect}
\STATE \hspace{1em} - compute cost $J^{[k+1]}$ by evaluating Equation~\eqref{eq:nonlinear_cost}
\STATE \hspace{1em} - increment iteration count $k$
\STATE \textbf{until} $|J^{[k]}-J^{[k-1]}|/J^{[k]} < J^{rel}_{min}$ \textbf{and} $\sum |\mathbf d_n^{[k]}| < d_{max}$
\end{algorithmic}
\end{algorithm}

\section{Results and Comparison}
\label{sec:simulation_results}
\subsection{An Illustrative, One-Dimensional System}
\label{sec:simpleSystem}
\begin{figure}
\centering
\includegraphics[width=0.999\columnwidth]{pics/simpleSystem.pdf}
\caption{
Results for a one-dimensional illustrative system, main text for detailed description.
}
\label{fig:simpleSystem}
\end{figure}
As an illustrative example, we present a simple one-dimensional system, which is slightly nonlinear, unstable and constructed to help the reader build an intuition about the methods.
The system dynamics are
$\dot x = (1+x)x + u$, $x(0)=1.5$
and discretized with $\Delta t = 0.01$~s, $N=300$. The cost function is defined as quadratic cost of form~\eqref{eq:linearized_cost} with desired terminal state \mbox{$\mathbf x_N^{des}= 0$}, $\mathbf Q_N = 10$ and $\mathbf R_n = 0.01$.
Fig.~\ref{fig:simpleSystem} shows results for iLQR, GNMS and the hybrid variants with five multiple-shooting intervals, GNMS(5) and iLQR-GNMS(5). We plot the state, control and defect trajectories for the first iteration of the algorithms, along with the initial guess and the converged solution.

The state and control trajectories illustrate the relationship between the algorithms: the multiple-shooting intervals of GNMS(5) and iLQR-GNMS(5) start with states and controls lying on the respective GNMS trajectories. For GNMS(5), the system is simulated open-loop, the controls are identical to GNMS, and the state trajectories on the multiple-shooting intervals start to diverge.
By contrast, for iLQR-GNMS(5), in every multiple-shooting interval both state and control trajectories converge asymptotically towards the simulated iLQR state and control trajectories.
For the hybrid variants, a defect occurs every 0.6s, for GNMS, the defect is evenly distributed across all time intervals.
Due to the long shooting-intervals, GNMS(5) requires one iteration more to catch up with the other algorithms in terms of overall cost. Importantly, the control update plot shows that the asymptotic contraction rates, which are defined as
\begin{equation}
C=\lim_{k\rightarrow \infty} \nicefrac{|\mathbf U^{[k+1]}-\mathbf U^*|_2}{|\mathbf U^{[k]}-\mathbf U^*|_2}
\label{eq:asymptotic_contr_rate}
\end{equation}
are not the same. In this example, GNMS and GNMS(5) show better contraction than iLQR. Asymptotic contraction rates are investigated in more detail in Section~\ref{sec:contraction_rates}.

\subsection{Quadruped Trot Optimization Example}
\label{sec:quadrupedTO}

The quadrupedal robot `HyQ'~\cite{semini:2011:hyqjournal} is an 18~DoF, floating-base underactuated robot subject to contacts with the environment, c.f. Fig.~\ref{fig:hyqs}.
In this paper, the contacts are not
\begin{wrapfigure}{r}{0.49\columnwidth}
\includegraphics[width=0.49\columnwidth]{pics/hyqs}
\caption{The quadruped HyQ}
\label{fig:hyqs}
\vspace{-1em}
\end{wrapfigure}
 incorporated as constraints, but added to the system dynamics using an explicit contact model.
We employ a static, plain environment and a `soft' contact model, consisting of a nonlinear spring in surface-normal direction and a nonlinear damping term. The contact model is detailed in~\cite{Giftthaler2017autodiff}. Using this formulation, the contact force is a function of the current robot state only.
It is clear that such a soft contact model presents only a rough approximation of the complicated physics of contact, and also introduces a number of potential disadvantages such as increased stiffness and nonlinearity of the combined system dynamics.
However, the contact model allows a straight-forward computation of derivatives~\cite{Giftthaler2017autodiff}, which creates an ideal test-bed for comparing our shooting algorithms.
We obtain exact discrete sensitivities $\mathbf A_n$ and $\mathbf B_n$ through evaluating a sensitivity differential equation on the multiple-shooting intervals~\cite{dickinson1976sensitivity}.

The example task considered is the optimization of a periodic trotting gait. To achieve the trotting gait, we impose a time-varying quadratic penalty on the leg joint positions.
Furthermore, we penalize the intermediate and final position of the robot's trunk and the intermediate and final velocities of the leg joints. For an in-depth description of the cost modelling to achieve different gait patterns the reader is referred to~\cite{neunert:2017:ral}.
In the following, the trotting gait optimization is used to compare the algorithms developed in this paper.
For a meaningful comparison, we initialize all algorithms with identical state trajectories and control policies. The initial guess corresponds to standing still in a steady state. We optimize over 36 states, 12 control inputs and a total time horizon of 2.5~seconds with $N=2500$.
\begin{figure}
\centering
\includegraphics[width=0.99\columnwidth]{pics/HyQ_trot.pdf}
\caption{Comparing different algorithms from the family of Gauss-Newton shooting methods w.r.t. cost descent, total defects and control update norm, illustrated with the example of a quadruped trot optimization problem.}
\label{fig:hyq_trot}
\end{figure}

Fig.~\ref{fig:hyq_trot} compares iLQR, GNMS, and iLQR-GNMS($M$) with three different numbers of multiple-shooting intervals in terms of cost descent, control update norms and total defects.
Note that SS and GNMS($M$) are unstable due to the strong instability of the system. All remaining algorithms converge to the same minimum within 20~iterations.
iLQR and iLQR-GNMS(25) show short phases of increasing cost, which we accept in this simulation example.
Since the provided initial guess is dynamically consistent, the initial defects are zero. GNMS, having the largest number of multiple-shooting intervals~(2500), also shows the largest total defect sum after the first iteration. The multiple-shooting iLQR methods, all having significantly fewer continuity constraints to enforce, feature a lower total defect.

As expected for a Gauss-Newton method, all approaches show linear convergence.
Considering the control update norms, we see that GNMS and iLQR feature a similar contraction rate for this example. In fact, the contraction rate of GNMS is slightly better, which is visually hard to distinguish here, but is detailed in following example.
For the hybrid multiple-shooting iLQR variants, we observe a significantly better contraction rate than for both iLQR and GNMS. When applying an identical termination criterion based on the relative change of the cost function and a defect threshold to all algorithms, all lifted methods converge in fewer iterations than iLQR. Furthermore, all displayed iLQR-GNMS($M$) variants converge notably faster than GNMS.
Screen recordings of the optimized trotting motions are provided in the video attachment~\cite{videoAttachment}.

\subsection{Local Contraction Rates for Quadruped Trot Tracking}
\label{sec:contraction_rates}
While Section~\ref{sec:quadrupedTO} gives an optimization example for a single motion, starting with an initial guess far from the optimal solution, we now show a comparison based on statistical data from~$1000$ runs: the trotting gait from Section~\ref{sec:quadrupedTO} is now considered in a tracking MPC problem. All algorithms are initialized with an optimal, dynamically consistent solution, but the initial state is locally perturbed. The state perturbations are sampled from the hardware-experiments detailed in~\cite{neunert2017mpc}.
For every perturbation, we let different algorithms iterate until convergence.
Fig.~\ref{fig:hyq_contraction} compares average asymptotic contraction rates for four different algorithms. It shows the normalized difference between a fully converged optimal feedforward trajectory and trajectories obtained at previous iterations. Furthermore it shows first-order regressions approximating the local contraction rates, in terms of the slopes of the difference norms in the semi-logarithmic plot.
It can be seen that GNMS outperforms iLQR in terms of local contraction rate.
GNMS(50) shows a contraction rate similar to GNMS.
The example indicates better local convergence for iLQR-GNMS(50) than for classical iLQR, GNMS and GNMS(50).

\begin{figure}
\centering
\includegraphics[width=0.99\columnwidth]{pics/contractionRatesHyQ.pdf}
\caption{
A statistical comparison of local contraction rates for four different algorithms. This plot shows the normalized difference between solutions to perturbed optimal control problems and a fully converged reference solution. The normalization is w.r.t. the iLQR control trajectory. The data is averaged from 1000 samples, the corresponding standard deviations are plotted as error-bars. Estimates for the contraction rates are indicated by straight lines.
GNMS outperforms iLQR in terms of local contraction rate, and gets closer to the true optimal solution in fewer iterations.
In contrast to the example in Section~\ref{sec:quadrupedTO}, GNMS(50) is stable and shows a contraction rate similar to GNMS. iLQR-GNMS(50) clearly outperforms all other algorithms with significantly better local contraction rate.
After 4~iterations, it is on average 0.1\% away from the fully converged reference solution, while iLQR is on average 14\% away.}
\label{fig:hyq_contraction}
\end{figure}

Fig.~\ref{fig:asymptotic_contraction_rates} generalizes the result from Fig.~\ref{fig:hyq_contraction} for a range of multiple-shooting intervals~$M$, showing numerically approximated asymptotic contraction rates, Equation~\eqref{eq:asymptotic_contr_rate}, as a function of~$M$. Again, GNMS($M$) is unstable for overly long multiple-shooting intervals, similar to the limiting case open-loop single shooting.
For closed-loop shooting, the asymptotic contraction rates for all multiple-shooting variants are better than for iLQR, and the contraction rates for the hybrid variants outperform the limiting case GNMS. In this example, the relative improvement over iLQR is up to a factor two.
Note that the resulting iLQR-GNMS($M$) contraction rates differ slightly from the ones in Fig.~\ref{fig:hyq_trot} for $M=25, 50, 100$, which is due to the different problem setting. However, both experiments exhibit the same trend.
\begin{figure}
\centering
\includegraphics[width=0.99\columnwidth]{pics/asymptoticContractionRates.pdf}
\caption{
Comparing asymptotic contraction rates as a function of the number of multiple-shooting intervals~$M$. For GNMS($M$), the integration on the multiple-shooting intervals is only stable for $M\geq20$. For this range of `stable' multiple-shooting intervals, the contraction rates are almost constant and similar to the limiting case GNMS.
The iLQR multiple-shooting methods are stable for all $M$. Here, the contraction rates for intermediate numbers of multiple-shooting intervals are distinctively better than for the limit cases iLQR and GNMS.
}
\label{fig:asymptotic_contraction_rates}
\end{figure}

\subsection{Nonlinear MPC on HyQ}
The suitability of iLQR for nonlinear model-predictive control (NMPC) in robotics applications has been shown many times before, in~\cite{koenemann2015whole,neunert16hexrotor,giftthaler2017efficient}.
In this section, we show that GNMS and its hybrid variants are even more promising for NMPC applications. First, they converge faster to the optimal solution, c.f. Section~\ref{sec:contraction_rates}. A second advantage of the multiple-shooting variants of the presented algorithms is that the forward integrations can be parallelized. Therefore, the achievable MPC cycle time decreases approximately linearly with the number of available CPU cores. By combining faster update rates with better contraction rate, our multiple-shooting algorithms outperform classical iLQR-NMPC.

In the following simulation example, we compare the NMPC performance of our Gauss-Newton shooting algorithms against iLQR-NMPC in a HyQ simulation environment. In each NMPC cycle, we run an adapted version of the main iteration in Algorithm~\ref{alg:general_algorithm} and `warm-start' it with the previous solution. In such a setting, we can separate an NMPC iteration into a `preparation' and a `feedback' phase~\cite{DIEHL2002}, thus minimizing the latency between receiving a state-measurement and sending an updated policy to the control system. Our NMPC loop is described in Algorithm~\ref{alg:gnms_mpc_algorithm}.
\begin{algorithm}[tpb]
\caption{iLQR-GNMS($M$)-NMPC Algorithm}
\label{alg:gnms_mpc_algorithm}
\begin{algorithmic}
\scriptsize
\STATE \textbf{Given}
\STATE - cost function~\eqref{eq:nonlinear_cost} and system dynamics~\eqref{eq:nonlinear_dynamics}.
\STATE - receding MPC time horizon $N$
\STATE - number of multiple-shooting intervals $M$ with length $l$
\STATE - initial state and control trajectories \mbox{$\mathbf X = \{ \mathbf x_0, \mathbf x_1, \ldots, \mathbf x_N\}$}
\STATE \hspace{1em}\mbox{$\mathbf U = \{ \mathbf u_0, \mathbf u_1, \ldots, \mathbf u_{N-1}\}$}, state-feedback controller $\mathbf u_n(\mathbf x)$ of form~\eqref{eq:optimal_control_input}
\STATE \textbf{Repeat Online:}
\STATE \textit{Feedback phase}
\STATE - get state measurement $\mathbf x_{meas}$.
\STATE - forward integrate system dynamics~\eqref{eq:nonlinear_dynamics} with \mbox{$\mathbf x_0 = \mathbf x_{meas}$} on the first
\STATE \hspace{1em}multiple-shooting interval, compute $\mathbf A_{0,\ldots,l-1}, \ \mathbf B_{0,\ldots,l-1}$, defect $\mathbf d_{l-1}$
\STATE - quadratize cost function~\eqref{eq:nonlinear_cost} around $\mathbf X$ and $\mathbf U$ for control stages $1,\ldots, l-1$.
\STATE - solve LQ optimal control problem using a Riccati backward sweep
\STATE - retrieve updated control policy $\mathbf u_n^{+}(\mathbf x)$ and updated trajectories $\mathbf U^+$, $\mathbf X^+$.
\STATE - send policy $\mathbf u_n^{+}(\mathbf x)$ and $\mathbf X^+$ to the control system
\STATE \textit{Preparation phase}
\STATE - update: $\mathbf u_n(\mathbf x) \leftarrow \mathbf u_n^{+}(\mathbf x)$, $\mathbf X \leftarrow \mathbf X^+$, $\mathbf U \leftarrow \mathbf U^+$
\STATE - forward integrate system dynamics~\eqref{eq:nonlinear_dynamics} for the multiple-shooting
\STATE \hspace{1em}intervals 1 to $M$, obtain sensitivities $\mathbf A_l,\ldots \mathbf A_{N-1}$, $\mathbf B_l,\ldots, \mathbf B_{N-1}$
\STATE \hspace{1em} and defects $\mathbf d_{l,\ldots, N-1}$
\STATE - quadratize cost function~\eqref{eq:nonlinear_dynamics} around $\mathbf X$, $\mathbf U$ for multiple-shooting intervals $l$ to~$N$.
\end{algorithmic}
\end{algorithm}

In this experiment, we run a trotting gait on HyQ, in closed-loop MPC in a simulation environment~\cite{Schaal__2009}. For the NMPC optimal control problem, we choose a time-step size of 4~ms and \mbox{$N=125$}. We parallelize the integration of all multiple-shooting intervals and the sensitivity computation on four cores, and run both simulator and MPC controller on the same notebook equipped with an Intel Core~i7 (2.8~GHz) processor.
For four different algorithmic combinations, we record the executed trot under identical conditions for 18~seconds and compute the resulting accumulated intermediate cost. A summary of the achieved average cost and NMPC frequencies is given in Fig.~\ref{fig:hyq_mpc}. In these experiments, iLQR results in the highest accumulated cost. The multiple-shooting variants outperform iLQR, with relative cost differences up to 5\%. At the same time, due to shorter runtimes, the multiple-shooting variants achieve up to 40\% higher MPC frequencies, with a maximum of 103~Hz.

In our simulation, all four algorithm variants run in a stable and robust fashion. The relatively small cost difference is an indicator of better convergence, but the main reason why the multiple-shooting variants should be preferred over iLQR in real-world applications, is the superior control bandwidth.
The algorithms in this paper have been validated in hardware experiments on two different quadruped platforms, where a variety of motions and gaits was implemented. However, a in-depth description of the experimental setups, the optimized computational framework and practical tuning considerations are beyond the scope of this paper. The interested reader is therefore referred to~\cite{neunert2017mpc}, where we apply the GNMS-algorithm for full-body NMPC on the quadrupeds HyQ and ANYmal~\cite{hutter2016anymal}, explain the robotic setup in detail and present a variety of hardware experiments. As outlook, a video sequence of GNMS-NMPC running on hardware is provided in the attachment~\cite{videoAttachment}.

\begin{figure}
\centering
\includegraphics[width=0.99\columnwidth]{pics/hyqMpc.pdf}
\caption{Comparing four different solvers for HyQ-MPC. The left plot compares costs accumulated over 18.0 seconds of trotting-MPC execution. iLQR shows the highest accumulated cost. The multiple-shooting variants outperform iLQR, but the relative cost difference is small. GNMS runs at 96.3\% of the iLQR cost, GNMS(25) at 95.9\% and iLQR-GNMS(25) at 95.0\%.
Regarding the achieved average MPC frequencies, shown on the right, we find up to 40\% higher frequencies for the multiple-shooting variants. The maximum average MPC frequency, 103 Hz, is achieved using GNMS.}
\label{fig:hyq_mpc}
\end{figure}

\section{Conclusion and Outlook}
\label{sec:Discussion}
In this paper, we have shown how the well-known iLQR algorithm can be lifted and transformed into Gauss-Newton Multiple Shooting, GNMS. We have generalized the concept to form a family of Gauss-Newton shooting algorithms, which can be distinguished into sequential and simultaneous algorithms and closed and open-loop algorithms. Some algorithms partially overwrite decision variables by means of a numeric forward integration. All presented variants have approximately the same computational cost and feature linear time-complexity. Furthermore, all discussed algorithms share a large number of computational routines, and it is not difficult to implement all of the presented variants in a single software framework.

We have compared the performance of the algorithms in different simulation experiments, which indicate that the lifted algorithms can outperform classical iLQR.
While not included in this paper for reasons of compactness, similar results were obtained for other rigid-body dynamic systems including a 6~DoF fixed-base arm model.
A more fundamental investigation for formalizing the conditions that result in improved convergence rates for GNMS($M$) and iLQR-GNMS($M$) is subject to ongoing work.

In the application examples, we limited the comparison to full-step variants of all considered algorithms. However, for even more nonlinear dynamics or cost functions, where the LQ optimal control problem is a bad approximation to the nonlinear problem, a globalization strategy may be required. For single-shooting methods, a straight-forward solution is to employ a line-search scheme. This is simple to implement, as it is sufficient to search over the cost for different control update step-sizes. For multiple-shooting approaches, however, there are additional continuity constraints, and we need to line-search over a merit-function which trades off the costs and defects. It is typically required to introduce additional tuning variables or to implement a complex filter-scheme~\cite{nocedal}.
Our open-source reference implementation~\cite{adrlCT} provides a line-search scheme using a simple merit function.

For complex robot trajectory optimization problems, we do not recommend to generally prioritize one of the presented algorithms over another. While the multiple-shooting algorithms allow for advanced initialization strategies and are more robust w.r.t. bad initial guesses, they may require slightly more tuning efforts when the full-step algorithm is not sufficient.
By contrast, in NMPC applications with well-defined cost functions and using warm-starting, additional globalization steps are rarely required at all. Here, our multiple-shooting algorithms offer significant advantages, better local contraction rates and much shorter runtimes.

The focus of this paper is on unconstrained optimal control problems without general (in)equality path constraints. It is obvious that the lifting approach naturally transfers to equality-constrained variants of iLQR, such as~\cite{sideris2011riccati}.
The inclusion of general (in)equality path constraints is part of ongoing work. One option to include them in the existing framework is to replace the standard Riccati backward sweep with a dedicated solver for constrained LQ optimal control problems~\cite{frisonalgorithms}. In this way, general (in)equality path constraints can be included while keeping linear time-complexity.

While this work treats algorithms using a Gauss-Newton Hessian approximation, it similarly transfers to exact-Hessian approaches, resulting in a multiple-shooting DDP algorithm combining the advantages of simultaneous methods, quadratic convergence and closed-loop integration.

\section*{Acknowledgements}
\small
The authors would like to thank Dimitris Kouzoupis, Gianluca Frison, Mario Zanon and Timothy Sandy for fruitful discussions.
This research was supported by the Swiss National Science Foundation through the NCCR Digital Fabrication, the NCCR Robotics and a Professorship Award to Jonas Buchli.
Further, this research was supported by the EU via FP7-ITN-TEMPO (607 957) and H2020-ITN-AWESCO (642 682), by the Federal Ministry for Economic Affairs and Energy (BMWi) via eco4wind and DyConPV, and by DFG via Research Unit FOR 2401.

\bibliographystyle{ieeetr}