\section{Introduction}

Small satellites are well-suited for formation flying missions, where
multiple satellites operate together in a cluster or predefined geometry
to accomplish the task of a single conventionally large satellite.
Application of swarms of hundreds to thousands of femtosatellites
($100$-gram-class satellites) for synthetic aperture applications
is discussed in \cite{Ref:Hadaegh13}. In this paper, we introduce
an inhomogeneous Markov chain approach to develop probabilistic swarm
guidance algorithms for constructing and reconfiguring a multi-agent
network comprised of a large number of autonomous agents or spacecraft.

Analogous to fluid mechanics, the traditional view of guidance of
multi-agent systems is \textit{Lagrangian}, as it deals with an indexed
collection of agents \nocite{Ref:Chung12,Ref:Mesbahi01,Ref:Bullo08,Ref:Arslan07,Ref:Rus12,Ref:How02}\cite{Ref:Chung12}--\cite{Ref:How02}.
Note that such deterministic, Lagrangian approaches tend to perform
poorly with a large number ($100$s--$1000$s) of agents. In this
paper, we adopt an \textit{Eulerian} view, as we control the swarm
density distribution of a large number of index-free agents over disjoint
bins \cite{Ref:Egerstedt10,Ref:Menon04}. A centralized approach for
controlling the swarm, using optimal control of partial differential
equations, is discussed in \cite{Ref:Milutinovic06,Ref:Ferrari14}.
Distributed control of the density distribution, using region-based
shape controllers or attraction-repulsion forcing functions, are discussed
in \cite{Ref:Slotine09,Ref:MKumar11}.

In this paper, \textit{guidance} refers to both motion planning and
open--loop control that generate a desired trajectory for each agent
\cite{Ref:Scarf03}. Instead of allocating specific positions to individual
agents a priori, a probabilistic guidance algorithm is concerned with
achieving the desired swarm distribution across bins \nocite{Ref:Hespanha08,Ref:Kumar09,Ref:Chattopadhyay09,Ref:Acikmese12}\cite{Ref:Hespanha08}--\cite{Ref:Acikmese12}.
Each autonomous agent or robot independently determines its transition
from one bin to another following a synthesized Markov chain so that
the overall swarm converges to the desired formation. Because this
Markovian approach automatically fills deficient bins, the resulting
algorithm is robust to external damages to the formation. The main
limitation of probabilistic guidance algorithm using homogeneous Markov
chains, where the Markov matrix $M$ is fixed over time, is that the
agents are not allowed to settle down even after the swarm reaches
the desired steady-state distribution resulting in significant wastage
of control effort (e.g., fuel).

The main contribution of this paper is to develop a probabilistic
swarm guidance algorithm using inhomogeneous Markov chains (PSG--IMC)
to address this limitation and minimize the number of transitions
for achieving and maintaining the formation. Our key concept, which
was first presented in \cite{Ref:Bandyopadhyay13SFFMT}, is to develop
time-varying Markov matrices $M_{k}^{j}$ with $\lim_{k\rightarrow\infty}M_{k}^{j}=\mathbf{I}$
to ensure that the agents settle down after the desired formation
is achieved, thereby minimizing the number of unnecessary transitions
across the bins. The inhomogeneous Markov transition matrix for an
agent at each time instant depends on the current swarm distribution,
the agent's current bin location, the time-varying motion constraints,
and the agent's choice of distance-based parameters. We derive proofs
of convergence based on analysis of inhomogeneous Markov chains (e.g.,
\nocite{Ref:Hajnal56,Ref:Wolfowitz63,Ref:Chatterjee77,Ref:Seneta06,Ref:Touri11}\cite{Ref:Hajnal56}--\cite{Ref:Touri11}).

It is necessary that each agent communicates with its neighboring
agents to estimate the current swarm distribution. Consensus algorithms
are studied for formation control, sensor networks, and formation
flying applications \nocite{Ref:Chung12,Ref:Saber04,Ref:Tsitsiklis86,Ref:Jadbabaie03,Ref:Boyd04,Ref:Shamma07,Ref:Ren05TAC}\cite{Ref:Chung12,Ref:Saber04}--\cite{Ref:Ren05TAC}.
In this paper, by using a decentralized consensus algorithm on probability
distributions \cite{Ref:Bandyopadhyay13TAC}, the agents reach an
agreement on the current estimate of the swarm distribution.

Inter-agent collisions are not considered in this paper for concise
presentation since a collision-avoidance algorithm within each bin
can be combined with the proposed PSG-IMC algorithm (see \cite{Ref:Morgan14,Ref:Giri14}
for details). As an illustrative example, the guidance of swarms of
spacecraft orbiting Earth is presented in this paper.

\begin{figure}
\begin{centering}
\includegraphics[bb=0bp 110bp 960bp 455bp,clip,width=3in]{PGA_random_I_v4.pdf}
\par\end{centering}

\begin{centering}
\vspace{-10pt}\protect\caption{The PSG--IMC independently determines each agent's trajectory so that
the overall swarm converges to the desired formation (here letter
``I''), starting from any initial distribution. Here, the state
space is partitioned into $25$ bins and the desired formation $\boldsymbol{\pi}$
is given by $\frac{1}{7}[0,0,0,0,0,\thinspace0,1,1,1,0,\thinspace0,0,1,0,0,\thinspace0,1,1,1,0,\thinspace0,0,0,0,0]$.
\label{fig:PSG-IMC-example}}

\par\end{centering}

\vspace{-15pt}
\end{figure}

\textit{Notation:} The \textit{time index} is denoted by a right subscript
and the \textit{agent index} is denoted by a lower-case right superscript.
The symbol $\mathbb{P}(\cdot)$ refers to the probability of an event.
The graph $\mathcal{G}_{k}$ represents the directed time-varying
communication network topology at the $k^{\textrm{th}}$ time instant.
Let $\mathcal{N}_{k}^{j}$ denote the neighbors of the $j^{\textrm{th}}$
agent at the $k^{\textrm{th}}$ time instant from which it receives
information. The set of inclusive neighbors of the $j^{\textrm{th}}$
agent is denoted by $\mathcal{J}_{k}^{j}:=\mathcal{N}_{k}^{j}\cup\{j\}$.
Let $\mathbb{N}$, $\mathbb{Z}^{*}$, and $\mathbb{R}$ be the sets
of natural numbers (positive integers), nonnegative integers, and
real numbers respectively. Let $\sigma$ represent the singular value
of a matrix. Let $\mathrm{diag}(\boldsymbol{\alpha})$ denote the
diagonal matrix of appropriate size with $\boldsymbol{\alpha}$ as
its diagonal elements. Let $\min{}^{+}$ refer to the minimum of the
positive elements. Let $\mathbf{1}=[1,1,\ldots,1]^{T}$, $\mathbf{I}$,
$\mathbf{0}$, and $\emptyset$ be the ones (column) vector, the identity
matrix, the zero matrix of appropriate sizes, and the empty set respectively.
Let $\|\cdot\|_{p}$ denote the $\ell_{p}$ vector norm. The symbols
$\left|\cdot\right|$ and $\left\lceil \cdot\right\rceil $ denote
the absolute value or the cardinality of a set and the ceiling function
respectively.

\section{Problem Statement and Overview of PSG--IMC \label{sec:Problem-Statement}}

Let $\mathcal{R}\subset\mathbb{R}^{n_{x}}$ denote the $n_{x}$-dimensional
compact physical space over which the swarm is distributed. The region
$\mathcal{R}$ is partitioned into $n_{\textrm{cell}}$ disjoint bins
represented by $R[i],\thinspace i=1,\ldots,n_{\textrm{cell}}$ so
that $\bigcup_{i=1}^{n_{\textrm{cell}}}R[i]=\mathcal{R}$ and $R[i]\cap R[q]=\emptyset$,
if $i\not=q$.

Let $m\in\mathbb{N}$ agents belong to the swarm. Note that we assume
$m\gg n_{\textrm{cell}}$, since we deal with the swarm density distribution
over these bins. Let the row vector $\boldsymbol{r}_{k}^{j}$ represent
the bin in which the $j^{\textrm{th}}$ agent is actually present
at the $k^{\textrm{th}}$ time instant. If $\boldsymbol{r}_{k}^{j}[i]=1$,
then the $j^{\textrm{th}}$ agent is inside the $R[i]$ bin at the
$k^{\textrm{th}}$ time instant; otherwise $\boldsymbol{r}_{k}^{j}[i]=0$.
The current swarm distribution ($\mathcal{F}_{k}^{\star}$) is given
by the ensemble mean of actual agent positions, i.e., $\mathcal{F}_{k}^{\star}:=\frac{1}{m}\sum_{j=1}^{m}\boldsymbol{r}_{k}^{j}$.
Let us now define the desired formation.

\begin{definition} \label{def:Desired-formation} \textit{(Desired
Formation $\boldsymbol{\pi}$)} Let the desired formation shape be
represented by a probability (row) vector $\boldsymbol{\pi}\in\mathbb{R}^{n_{\textrm{cell}}}$
over the bins in $\mathcal{R}$, i.e., $\boldsymbol{\pi}\mathbf{1}=1$.
Note that $\boldsymbol{\pi}$ can be any arbitrary probability vector,
but it is the same for all agents within the swarm. In the presence
of motion constraints, $\boldsymbol{\pi}$ needs to satisfy Assumption
\ref{assump_Pi} discussed in Section \ref{sec:Handling-Motion-Constraints}.
For example, $\boldsymbol{\pi}$ is the shape of the letter ``I''
in Fig. \ref{fig:PSG-IMC-example}. Note that any guidance algorithm,
using a swarm of $m$ agents, can only achieve the best quantized
representation of the desired formation $\boldsymbol{\pi}$, where
$\frac{1}{m}$ is the quantization factor. \hfill $\Box$ \end{definition}

The objectives of probabilistic swarm guidance using inhomogeneous
Markov chains (PSG--IMC) running on board each agent are as follows:
\begin{enumerate}
\item Determine the agent's trajectory using a Markov chain, which obeys
motion constraints, so that the overall swarm converges to a desired
formation ($\boldsymbol{\pi}$).
\item Reduce the number of transitions for achieving and maintaining the
formation in order to reduce control effort (e.g., fuel).
\item Maintain the swarm distribution and automatically detect and repair
damages to the formation.
\end{enumerate}
The key idea of the proposed PSG--IMC is to synthesize inhomogeneous
Markov chains for each agent so that each agent can independently
determine its trajectory while the swarm distribution converges to
the desired formation. The pseudo code for the algorithm is given
in \textbf{Algorithm} \textbf{\ref{alg:PGA-inhomo-motion-const}}.

\begin{algorithm}[th]
\protect\caption{Probabilistic swarm guidance algorithm using inhomogeneous Markov
chains (PSG--IMC) \label{alg:PGA-inhomo-motion-const}}

\begin{singlespace}
{\small{}}%
\begin{tabular}{cll}
{\small{}1:} & \multicolumn{2}{l}{{\small{}(one cycle of $j^{\textrm{th}}$ agent during $k^{\textrm{th}}$
time instant)}}\tabularnewline
{\small{}2:} & \multicolumn{2}{l}{{\small{}Agent determines its present bin, e.g., $\boldsymbol{r}_{k}^{j}[i]=1$}}\tabularnewline
{\small{}3:} & {\small{}Set $n_{\textrm{loop}}$, the weighting factors $a_{k}^{j\ell}$ } & {\small{}$\}$ Theorem \ref{thm:Bayesian_Consensus_Filtering_Static_balanced_graphs}}\tabularnewline
{\small{}4:} & \textbf{\small{}for }{\small{}$\nu=1$ to $n_{\textrm{loop}}$} & \multirow{6}{*}{{\small{}$\left\} \begin{array}{c}
\\
\\
\textrm{Consensus}\\
\textrm{Stage}\\
\\
\\
\end{array}\right.$}}\tabularnewline
{\small{}5:} & \textbf{\small{}$\:\:\:$if $\nu=1$ then }{\small{}Set $\mathcal{F}_{k,0}^{j}$
from $\boldsymbol{r}_{k}^{j}$ }\textbf{\small{}end if} & \tabularnewline
{\small{}6:} & \textbf{\small{}$\:\:\:$}{\small{}Transmit the pmf $\mathcal{F}_{k,\nu-1}^{j}$
to other agents} & \tabularnewline
{\small{}7:} & \textbf{\small{}$\:\:\:$}{\small{}Obtain the pmfs $\mathcal{F}_{k,\nu-1}^{\ell},\forall\ell\in\mathcal{J}_{k}^{j}$ } & \tabularnewline
{\small{}8:} & \textbf{\small{}$\:\:\:$}{\small{}Compute the new pmf $\mathcal{F}_{k,\nu}^{j}$
using LinOP} & \tabularnewline
{\small{}9:} & \textbf{\small{}end for} & \tabularnewline
{\small{}10:} & {\small{}Compute the tuning parameter $\xi_{k}^{j}$ } & {\small{}$\}$ Eq. (\ref{eq:tuning_eqn})}\tabularnewline
{\small{}11:} & \textbf{\small{}if}{\small{} }\textbf{\small{}$R[i]\not\in\Pi$ then
}{\small{}Go to bin $R[\ell]$} & {\small{}$\}$ Eq. (\ref{eq:trapping_region})}\tabularnewline
{\small{}12:} & \textbf{\small{}else }{\small{}Compute the $\boldsymbol{\alpha}_{k}^{j}$
vector} & {\small{}$\}$ Eq. (\ref{eq:alpha_vector})}\tabularnewline
{\small{}13:} & \textbf{\small{}$\:\:\:$}{\small{}Compute the Markov matrix $M_{k}^{j}$ } & {\small{}$\}$ Prop. \ref{thm:Markov_matrix_new}}\tabularnewline
14: & {\small{}$\:\:\:$Modify the Markov matrix $\tilde{M}_{k}^{j}$} & {\small{}$\}$ Prop. \ref{thm:Markov_matrix_modified}}\tabularnewline
{\small{}15:} & \textbf{\small{}$\:\:\:$}{\small{}Generate a random number $z\in\textrm{unif}[0;1]$} & \multirow{3}{*}{{\small{}$\left\} \begin{array}{c}
\\
\textrm{Random}\\
\textrm{sampling}
\end{array}\right.$}}\tabularnewline
{\small{}16:} & \textbf{\small{}$\:\:\:$}{\small{}Go to bin $R[q]$ such that } & \tabularnewline
 & \textbf{\small{}$\:\:\:$$\:\:\:$}{\small{}$\sum_{\ell=1}^{q-1}\tilde{M}_{k}^{j}[i,\ell]\leq z<\sum_{\ell=1}^{q}\tilde{M}_{k}^{j}[i,\ell]$ } & \tabularnewline
{\small{}17:} & \textbf{\small{}end if} & \tabularnewline
\end{tabular}\end{singlespace}
\end{algorithm}

The first step (line 2) involves determining the bin in which the
agent is located. For example, $\boldsymbol{r}_{k}^{j}[i]=1$. During
the second step (lines 3--9), each agent estimates the current swarm
distribution ($\mathcal{F}_{k}^{\star}$) by reaching an agreement
across the network using the consensus algorithm \cite{Ref:Bandyopadhyay13TAC}.
This is elucidated in Section \ref{sec:Bayesian-Consensus-Filtering}.

In this paper we use the difference between the estimated swarm distribution
($\mathcal{\hat{F}}_{k,n_{\textrm{loop}}}^{j}$) and the desired formation
($\boldsymbol{\pi}$) to dictate the motion of agents in the swarm.
The Hellinger distance (HD) is a symmetric measure of the difference
between two probability distributions and it is upper bounded by $1$
\cite{Ref:Torgerson91}. As discussed in Section \ref{sec:Bayesian-Consensus-Filtering},
the tuning parameter ($\xi_{k}^{j}$) computed in line 10 is the HD
between the current swarm distribution and the desired formation.

It may be desired that the agents in a particular bin can only transition
to some bins, while they cannot transition to other bins. Each agent
checks if it is currently in the trapping region or a transient bin
in line 11 and then it transitions to the bin $R[\ell]$ which is
best-suited to reach the formation. The concept of trapping region
and the method for handling motion constraints are presented in Section
\ref{sec:Handling-Motion-Constraints}.

The next step (lines 12--14) involves designing a family of row stochastic
Markov transition matrices $\tilde{M}_{k}^{j}$ with $\boldsymbol{\pi}$
as their stationary distributions, which is presented in Sections
\ref{sec:Designing-Markov-Matrix} and \ref{sec:Handling-Motion-Constraints}.
When the HD between the estimated swarm distribution and the desired
formation is large, each agent propagates its position in a statistically-independent
manner so that the swarm asymptotically tends to the desired formation.
As this HD decreases, the Markov matrices also tend to an identity
matrix and each agent holds its own position. In lines 15--16, random
sampling of the Markov matrix generates the next location of the agent.
The stability and convergence guarantees for PSG--IMC are presented
in Section \ref{sec:Convergence-of-Inhomogeneous}.

The guidance or motion planning of spacecraft in a swarm is discussed
in Section \ref{sec:Guidance-Spacecraft}. Strategies for implementing
PSG--IMC algorithms are demonstrated with numerical examples in Sections
\ref{sec:Handling-Motion-Constraints} and \ref{sec:Guidance-Spacecraft}.

\section{Consensus Estimation of Swarm Distribution \label{sec:Bayesian-Consensus-Filtering}}

In this section, we use the decentralized consensus algorithm \cite{Ref:Bandyopadhyay13TAC}
to estimate the current swarm distribution, as illustrated in \textbf{Algorithm
\ref{alg:PGA-inhomo-motion-const}}. The objective of the consensus
stage is to estimate the current swarm distribution ($\mathcal{F}_{k}^{\star}$)
and maintain consensus across the network during each time step.

Let the row vector $\hat{\mathcal{F}}_{k,\nu}^{j}$ represent the
$j^{\textrm{th}}$ agent's estimate of the current swarm distribution
during the $\nu^{\textrm{th}}$ consensus loop at the $k^{\textrm{th}}$
time instant. At the beginning of the consensus stage of the $k^{\textrm{th}}$
time instant, the $j^{\textrm{th}}$ agent generates a row vector
of local estimate of the swarm distribution $\hat{\mathcal{F}}_{k,0}^{j}$
by only determining its present bin location: \vspace{-5pt}

\begin{equation}
\hat{\mathcal{F}}_{k,0}^{j}[i]=1\thinspace\textrm{ if }\boldsymbol{r}_{k}^{j}[i]=1,\thinspace\textrm{otherwise}\thinspace\thinspace0.
\end{equation}
In essence, the local estimate at the start of the consensus stage
is a discrete representation of the position of the $j^{\textrm{th}}$
agent in the space $\mathcal{R}$, i.e., $\hat{\mathcal{F}}_{k,0}^{j}=\boldsymbol{r}_{k}^{j}$.
Hence the current swarm distribution is also given by $\mathcal{F}_{k}^{\star}=\sum_{i=1}^{m}\frac{1}{m}\hat{\mathcal{F}}_{k,0}^{i}$,
which is equal to the ensemble mean of actual agent positions $\{\boldsymbol{r}_{k}^{j}\}_{j=1}^{m}$
over the bins in $\mathcal{R}$.

During the consensus stage, the agents recursively combine and update
their local distributions to reach an agreement across the network.
The Linear Opinion Pool (LinOP) of probability measures, which is
used for combining individual distributions \cite{Ref:DeGroot74,Ref:Genest86},
is given by: \vspace{-10pt}

\begin{align}
\hat{\mathcal{F}}_{k,\nu}^{j} & =\sum_{\ell\in\mathcal{J}_{k}^{j}}a_{k,\nu-1}^{\ell j}\hat{\mathcal{F}}_{k,\nu-1}^{j},\thinspace\forall j,\ell\in\{1,\ldots,m\},\forall\nu\in\mathbb{N},\label{eq:agreement_equation}
\end{align}
where $\sum_{\ell\in\mathcal{J}_{k}^{j}}a_{k,\nu-1}^{\ell j}=1$.
The updated distribution $\hat{\mathcal{F}}_{k,\nu}^{j}$ after the
$\nu^{\textrm{th}}$ consensus loop is a weighted average of the distributions
of the inclusive neighbors $\hat{\mathcal{F}}_{k,\nu-1}^{j},\forall\ell\in\mathcal{J}_{k}^{j}$
at $k^{\textrm{th}}$ time instant. Let $\mathbf{\mathcal{W}}_{k,\nu}=\left[\hat{\mathcal{F}}_{k,\nu}^{1},\ldots,\hat{\mathcal{F}}_{k,\nu}^{m}\right]$
be a row vector of pmf functions of the agents after the $\nu^{\textrm{th}}$
consensus loop. The LinOP (\ref{eq:agreement_equation}) can be expressed
concisely as $\mathbf{\mathcal{W}}_{k,\nu}=\mathbf{\mathcal{W}}_{k,\nu-1}P_{k,\nu-1},\thinspace\forall\nu\in\mathbb{N}$,
where $P_{k,\nu-1}$ is a matrix with entries $P_{k,\nu-1}[\ell,j]=a_{k,\nu-1}^{\ell j}$.

\begin{assumption} The communication network topology of the multi-agent
system $\mathcal{G}(k)$ is strongly connected (SC). The weighting
factors $a_{k,\nu-1}^{\ell j},\forall j,\ell\in\{1,\ldots,m\}$ and
the matrix $P_{k,\nu-1}$ have the following properties: (i) the weighting
factors are the same for all consensus loops within each time instants;
(ii) the matrix $P_{k}$ conforms with the graph $\mathcal{G}(k)$;
(iii) the matrix $P_{k}$ is column stochastic; and (iv) the weighting
factors $a_{k}^{\ell j}$ are balanced. \hfill $\Box$ \label{assump_weights2}
\end{assumption}

Since $m\gg n_{\textrm{cell}}$ and multiple agents are within the
same bin, it is guaranteed almost surely using Erd\"{o}s--R\'{e}nyi
random graphs or random nearest--neighbor graphs that the resulting
network is SC \cite{Ref:Jackson08,Ref:Jingjin14,Ref:Kumar04}. Moreover,
distributed algorithms exist for the agents to generate SC balanced
graphs \cite{Ref:Boyd04,Ref:Cortes12,Ref:Hadjicostis14}.

Let $\boldsymbol{\theta}_{k,\nu}=\left[\theta_{k,\nu}^{1},\ldots,\theta_{k,\nu}^{m}\right]$
be the disagreement vector, where $\theta_{k,\nu}^{j}$ is the $\mathcal{L}_{1}$
distances between $\hat{\mathbf{\mathcal{F}}}_{k,\nu}^{j}$ and $\mathbf{\mathcal{F}}_{k}^{\star}$,
i.e., $\theta_{k,\nu}^{j}=\sum_{R[i]\in\mathcal{R}}|\hat{\mathcal{F}}_{k,\nu}^{j}[i]-\mathcal{F}_{k}^{\star}[i]|$.
Since the $\mathcal{L}_{1}$ distances between pmfs is bounded by
$2$, the $\ell_{2}$ vector norm $\|\boldsymbol{\theta}_{k,\nu}\|_{2}$
is upper bounded by $2\sqrt{m}$.

\begin{theorem} \label{thm:Bayesian_Consensus_Filtering_Static_balanced_graphs}
\cite{Ref:Saber04}--\cite{Ref:Bandyopadhyay13TAC} \emph{(Consensus
using the LinOP on SC Balanced Digraphs)} Under Assumption \ref{assump_weights2},
using the LinOP (\ref{eq:agreement_equation}), each $\hat{\mathbf{\mathcal{F}}}_{k,\nu}^{j}$
globally exponentially converges to $\mathbf{\mathcal{F}}_{k}^{\star}=\sum_{i=1}^{m}\frac{1}{m}\hat{\mathbf{\mathcal{F}}}_{k,0}^{i}$
pointwise with a rate faster or equal to the second largest singular
value of $P_{k}$, i.e., $\sigma_{m-1}(P_{k})$. If for some consensus
error $\varepsilon_{\mathrm{cons}}>0$, the number of consensus loops
within each consensus stage is $n_{\mathrm{loop}}\geq\left\lceil \frac{\ln\left(\varepsilon_{\mathrm{cons}}/(2\sqrt{m})\right)}{\ln\sigma_{m-1}(P_{k})}\right\rceil $
then the $\ell_{2}$ norm of the disagreement vector at the end of
the consensus stage is less than $\varepsilon_{\mathrm{cons}}$, i.e.,
$\|\boldsymbol{\theta}_{k,n_{\mathrm{loop}}}\|_{2}\leq\varepsilon_{\mathrm{cons}}$.
\end{theorem}

In this paper, we assume $\varepsilon_{\mathrm{cons}}\leq\frac{1}{m}$
and $n_{\textrm{loop}}$ is computed using Theorem \ref{thm:Bayesian_Consensus_Filtering_Static_balanced_graphs}.
Note that, in order to transmit the pmf $\hat{\mathcal{F}}_{k,\nu}^{j}$
to another agent, the agent needs to transmit $n_{\textrm{cell}}$
real numbers bounded by $[0,1]$. For a practical scenario involving
spacecraft swarms, as discussed in Section \ref{sec:Guidance-Spacecraft},
it is feasible to execute multiple consensus loops within each time
step.%
\footnote{In order to obtain an estimate of the communication load, let us assume
that during each of the $20$ consensus loops, each agent needs to
transmits its estimated pmf $\hat{\mathcal{F}}_{k,\nu}^{j}$ to $100$
other agents and receives pmf estimates from $100$ neighboring agents.
The total transmission time using a $250$ Kbps XBee radio \cite{Ref:Brumbaugh12Master}
is $\frac{900\times8\times2\times100\times20}{250\times10^{3}}\approx2$
minutes. This is significantly less than the time step of $8-10$
minutes used for such missions. %
}

Next, we compute the tuning parameter ($\xi_{k}^{j}$) described in
line 10 of \textbf{Algorithm \ref{alg:PGA-inhomo-motion-const}}.

\begin{definition} \label{def:Hellinger-Distance} \textit{(Hellinger
Distance based Tuning Parameter $\xi_{k}^{j}$)} The HD is a symmetric
measure of the difference between two probability distributions and
it is upper bounded by $1$ \cite{Ref:Torgerson91,Ref:Cha07}. Each
agent chooses the tuning parameter ($\xi_{k}^{j}$), based on the
HD, using the following equation: \vspace{-10pt}

\begin{align}
\xi_{k}^{j} & \!=\! D_{H}(\boldsymbol{\pi},\hat{\mathcal{F}}_{k,n_{\textrm{loop}}}^{j})\!:=\!\frac{1}{\sqrt{2}}\sqrt{\sum_{i=1}^{n_{\textrm{cell}}}\left(\sqrt{\boldsymbol{\pi}[i]}-\sqrt{\hat{\mathcal{F}}_{k,n_{\textrm{loop}}}^{j}[i]}\right)^{2}}.\label{eq:tuning_eqn}
\end{align}
where $\hat{\mathcal{F}}_{k,n_{\textrm{loop}}}^{j}$ is the estimated
current swarm distribution and $\boldsymbol{\pi}$ is the desired
formation. \hfill $\Box$ \end{definition}

\begin{remark} In contrast with the standard $\mathcal{L}_{1}$ or
$\mathcal{L}_{2}$ distance metrics, the weight assigned by the HD
to the error in a bin is inversely proportional to the square root
of the desired distribution in that bin. For example, the $\mathcal{L}_{1}$
distances of $\mathcal{F}_{1}=[0.1,0,0.4,0,0.5]$ and $\mathcal{F}_{2}=[0,0,0.5,0,0.5]$
from the desired distribution $\boldsymbol{\pi}=[0,0,0.4,0,0.6]$
are equal to $0.2$. However, the agents in the first bin with $\mathcal{F}_{1}[1]=0.1$
should move to other bins with nonzero probability, which is better
encapsulated by the Hellinger distance (e.g., $D_{H}(\boldsymbol{\pi},\mathcal{F}_{1})=0.2286$
while $D_{H}(\boldsymbol{\pi},\mathcal{F}_{2})=0.0712$). \hfill
$\Box$ \end{remark}

\section{Family of Markov Transition Matrices \label{sec:Designing-Markov-Matrix}}

The key concept of  PSG--IMC is that each agent can independently
determine its trajectory from the evolution of an inhomogeneous Markov
chain with a desired stationary distribution. In this section, we
design the family of Markov transition matrices for a desired stationary
distribution, using the tuning parameter from Definition \ref{def:Hellinger-Distance}.

Let $\boldsymbol{x}_{k}^{j}\in\mathbb{R}^{n_{\textrm{cell}}}$ denote
the row vector of probability mass function (pmf) of the predicted
position of the $j^{\textrm{th}}$ agent at the $k^{\textrm{th}}$
time instant, i.e., $\boldsymbol{x}_{k}^{j}\mathbf{1}=1$. The $i^{\textrm{th}}$
element ($\boldsymbol{x}_{k}^{j}[i]$) is the probability of the event
that the $j^{\textrm{th}}$ agent is in $R[i]$ bin at the $k^{\textrm{th}}$
time instant: \vspace{-5pt}

\begin{equation}
\boldsymbol{x}_{k}^{j}[i]=\mathbb{P}(\boldsymbol{r}_{k}^{j}[i]=1),\quad\textrm{for all }i\in\{1,\ldots,n_{\textrm{cell}}\}\thinspace.\label{eq:def_prob_vector}
\end{equation}
The elements of the row stochastic Markov transition matrix $M_{k}^{j}\in\mathbb{R}^{n_{\textrm{cell}}\times n_{\textrm{cell}}}$
are the transition probabilities of the $j^{\textrm{th}}$ agent at
the $k^{\textrm{th}}$ time instant: \vspace{-5pt}

\begin{equation}
M_{k}^{j}[i,\ell]:=\mathbb{P}\left(\boldsymbol{r}_{k+1}^{j}[\ell]=1|\boldsymbol{r}_{k}^{j}[i]=1\right)\thinspace.
\end{equation}
In other words, the probability that the $j^{\textrm{th}}$ agent
in $R[i]$ bin at the $k^{\textrm{th}}$ time instant will transition
to $R[\ell]$ bin at the $(k+1)^{\textrm{th}}$ time instant is given
by $M_{k}^{j}[i,\ell]$. The Markov transition matrix $M_{k}^{j}$
determines the time evolution of the pmf row vector $\boldsymbol{x}_{k}^{j}$
by: \vspace{-15pt}

\begin{equation}
\boldsymbol{x}_{k+1}^{j}=\boldsymbol{x}_{k}^{j}M_{k}^{j},\quad\textrm{for all }k\in\mathbb{Z}^{*}\thinspace.\label{eq:time_evolution_single}
\end{equation}

Lines 12--14 of \textbf{Algorithm \ref{alg:PGA-inhomo-motion-const}}
involves designing a family of Markov transition matrices for each
agent $M_{k}^{j}$, with $\boldsymbol{\pi}$ as their stationary distributions.
The following theorem is used by each agent to find these Markov matrices
at each time instant.

\begin{proposition} \label{thm:Markov_matrix_new} \textit{(Family
of Markov transition matrices for a desired stationary distribution)}
Let $\boldsymbol{\alpha}_{k}^{j}\in\mathbb{R}^{n_{\mathrm{cell}}}$
be a nonnegative bounded column vector with $\|\boldsymbol{\alpha}_{k}^{j}\|_{\infty}\leq1$.
For given $\xi_{k}^{j}$ from (\ref{eq:tuning_eqn}), the following
parametrized family of row stochastic Markov matrices $M_{k}^{j}$
have $\boldsymbol{\pi}$ as their stationary distribution (i.e., $\boldsymbol{\pi}M_{k}^{j}=\boldsymbol{\pi}$)
: \vspace{-10pt}

\begin{align}
M_{k}^{j} & =\boldsymbol{\alpha}_{k}^{j}\frac{\xi_{k}^{j}}{\boldsymbol{\pi}\boldsymbol{\alpha}_{k}^{j}}\boldsymbol{\pi}\mathrm{diag}(\boldsymbol{\alpha}_{k}^{j})+\mathbf{I}-\xi_{k}^{j}\mathrm{diag}(\boldsymbol{\alpha}_{k}^{j}),\label{eq:Markov_matrix_family_new}
\end{align}
where $\boldsymbol{\pi}\boldsymbol{\alpha}_{k}^{j}\not=0$ and $\sup_{k}\xi_{k}^{j}\|\boldsymbol{\alpha}_{k}^{j}\|_{\infty}\leq1$.
\end{proposition}
\begin{IEEEproof}
For a valid first term in (\ref{eq:Markov_matrix_family_new}), we
need $\boldsymbol{\pi}\boldsymbol{\alpha}_{k}^{j}\not=0$. We first
show that $M_{k}^{j}$ is a row stochastic matrix. Right multiplying
both sides of (\ref{eq:Markov_matrix_family_new}) with $\boldsymbol{1}$
gives: \vspace{-5pt}

\[
M_{k}^{j}\boldsymbol{1}=\xi_{k}^{j}\boldsymbol{\alpha}_{k}^{j}\tfrac{\boldsymbol{\pi}\boldsymbol{\alpha}_{k}^{j}}{\boldsymbol{\pi}\boldsymbol{\alpha}_{k}^{j}}+\boldsymbol{1}-\xi_{k}^{j}\mathrm{diag}(\boldsymbol{\alpha}_{k}^{j})\boldsymbol{1}=\boldsymbol{1}\thinspace.
\]
Next, we show that $M_{k}^{j}$ is a Markov matrix with $\boldsymbol{\pi}$
as its stationary distribution, as $\boldsymbol{\pi}$ is the left
eigenvector corresponding to its largest eigenvalue $1$, i.e., $\boldsymbol{\pi}M_{k}^{j}=\boldsymbol{\pi}$.
Left multiplying both sides of (\ref{eq:Markov_matrix_family_new})
with $\boldsymbol{\pi}$ gives: \vspace{-12pt}

\[
\boldsymbol{\pi}M_{k}^{j}=\tfrac{\boldsymbol{\pi}\boldsymbol{\alpha}_{k}^{j}}{\boldsymbol{\pi}\boldsymbol{\alpha}_{k}^{j}}\boldsymbol{\pi}\xi_{k}^{j}\mathrm{diag}(\boldsymbol{\alpha}_{k}^{j})+\boldsymbol{\pi}-\boldsymbol{\pi}\xi_{k}^{j}\mathrm{diag}(\boldsymbol{\alpha}_{k}^{j})=\boldsymbol{\pi}\thinspace.
\]
In order to ensure that all the elements in the matrix $M_{k}^{j}$
are nonnegative, we enforce that $\mathbf{I}-\xi_{k}^{j}\mathrm{diag}(\boldsymbol{\alpha}_{k}^{j})\geq0$
which results in the condition $\sup_{k}\xi_{k}^{j}\|\boldsymbol{\alpha}_{k}^{j}\|_{\infty}\leq1$.
\end{IEEEproof}
The additional degrees of freedom due to $\boldsymbol{\alpha}_{k}^{j}$
vector allows us to capture the physical distance between bins while
designing the Markov matrix.

\begin{definition} \label{def-alpha-vec} \textit{(Physical distance
based $\boldsymbol{\alpha}_{k}^{j}$ vector)} The $j^{\textrm{th}}$
agent at the $k^{\textrm{th}}$ time instant selects bin $R[c]$ independent
of its current location. Then each element of the $\boldsymbol{\alpha}_{k}^{j}$
vector is determined using the physical distance between bins in the
following manner for all $\ell\in\{1,\ldots,n_{\textrm{cell}}\}$:
\vspace{-10pt}

\begin{align}
\boldsymbol{\alpha}_{k}^{j}[\ell] & :=1-\frac{\textrm{dis}(R[\ell],R[c])}{\max_{q\in\{1,\ldots,n_{\textrm{cell}}\}}\textrm{dis}(R[q],R[c])+1}\thinspace,\label{eq:alpha_vector}
\end{align}
where $\textrm{dis}(R[\ell],R[c])$ is the $\ell_{1}$ distance between
the bins $R[\ell]$ and $R[c]$. If $\boldsymbol{\kappa}[\ell]\in\mathbb{R}^{n_{x}}$
denotes the location of the centroid of the bin $R[\ell]$, then $\textrm{dis}(R[\ell],R[c])=\|\boldsymbol{\kappa}[\ell]-\boldsymbol{\kappa}[c]\|_{1}$.
Irrespective of the distribution of $\boldsymbol{\pi}$, the condition
$\boldsymbol{\pi}\boldsymbol{\alpha}_{k}^{j}\not=0$ is satisfied
because $\boldsymbol{\alpha}_{k}^{j}[\ell]>0,\thinspace\forall\ell\in\{1,\ldots,n_{\textrm{cell}}\}$.
Moreover, the condition $\sup_{k}\xi_{k}^{j}\|\boldsymbol{\alpha}_{k}^{j}\|_{\infty}\leq1$
is satisfied as $\xi_{k}^{j}=D_{H}(\boldsymbol{\pi},\hat{\mathcal{F}}_{k,n_{\textrm{loop}}}^{j})\leq1$
in (\ref{eq:tuning_eqn}) and $\|\boldsymbol{\alpha}_{k}^{j}\|_{\infty}=1$
in (\ref{eq:alpha_vector}). \hfill $\Box$ \end{definition}

The evolution of the agent's location during each time step is based
on random sampling of the Markov transition matrix, as shown in line
15--16 of \textbf{Algorithm \ref{alg:PGA-inhomo-motion-const}}. An
agent is said to have undergone a transition if it jumps from bin
$R[i]$ to bin $R[\ell],\thinspace\ell\not=i$ during a given time
step. If $\xi_{k}^{j}=0$ in (\ref{eq:Markov_matrix_family_new}),
then $M_{k}^{j}=\mathbf{I}$ and no transitions occur as the transition
probabilities from a bin to any bin (other than itself) goes to zero.
If $\xi_{k}^{j}$ is large (close to $1$), the agents vigorously
transition from one bin to another. In this paper, we seek to minimize
unnecessary bin-to-bin transitions during each time step while maintaining
or reconfiguring the formation.

\begin{remark} \textit{(Dynamics based $\boldsymbol{\alpha}_{k}^{j}$
vector)} The agent dynamics can also be captured using the $\boldsymbol{\alpha}_{k}^{j}$
vector. At the $k^{\textrm{th}}$ time instant, the $\boldsymbol{\alpha}_{k}^{j}[\ell]$
element is the likelihood of the $j^{\textrm{th}}$ agent transitioning
to the bin $R[\ell]$ due to its dynamics and is independent of its
current location. In order to obtain valid solutions, it is necessary
that $\boldsymbol{\alpha}_{k}^{j}[\ell]>0,\thinspace\forall\ell\in\{1,\ldots,n_{\textrm{cell}}\}$
and $\|\boldsymbol{\alpha}_{k}^{j}\|_{\infty}=1$ for all time instants.
\hfill $\Box$ \end{remark}

In the probabilistic guidance algorithm (PGA) \textit{\cite{Ref:Chattopadhyay09,Ref:Acikmese12},}
the same homogeneous Markov transition matrix is used by all agents
for all time, i.e., $M_{k}^{j}=M$ for all $j\in\{1,\ldots,m\}$ and
$k\in\mathbb{Z}^{*}$. As the agents do not realize whether the desired
formation has already been achieved, they continue to transition for
all time steps. In contrast, in this paper, each agent executes a
different time-varying Markov matrix $M_{k}^{j}$ at each time step.
We show that the inhomogeneous Markov chain not only guides the agents
so that the swarm distribution converges to the desired steady-state
distribution but also reduces the number of transitions across the
bins.

\section{Convergence Analysis of Inhomogeneous Markov Chains \label{sec:Convergence-of-Inhomogeneous}}

In this section, we provide asymptotic guarantees for the proposed
PSG--IMC algorithm without motion constraints, and study its stability
and convergence characteristics. Theorem \ref{thm:Convergence-of-imhomo-MC}
states that each agent's pmf vector $\boldsymbol{x}_{k}^{j}$ asymptotically
converges pointwise to the desired formation $\boldsymbol{\pi}$ while
Theorem \ref{thm:Convergence-swarm-dist} states that the swarm distribution
$\mathcal{F}_{k}^{\star}$ also asymptotically converges pointwise
to the desired formation $\boldsymbol{\pi}$, when the number of agents
tends to infinity. For practical implementation of this algorithm,
a lower bound on the number of agents is provided in Theorem \ref{thm:Convergence-swarm-dist-practical}.
Finally, Remark \ref{rem:Markov_identity} states that if the consensus
error tends to zero, then the Markov transition matrix $M_{k}^{j}$
asymptotically converges to an identity matrix. This means that the
agents stop moving after the desired formation is achieved, resulting
in significant savings of control effort compared to the homogeneous
Markov chain case. Therefore, the first two objectives of PSG--IMC,
stated in Section \ref{sec:Problem-Statement}, are achieved.

The time evolution of the pmf vector $\boldsymbol{x}_{k}^{j}$, defined
in (\ref{eq:def_prob_vector}), from an initial condition ($\boldsymbol{x}_{0}^{j}$)
to the $k^{\textrm{th}}$ time instant is given by the inhomogeneous
Markov chain: \vspace{-5pt}

\begin{equation}
\boldsymbol{x}_{k}^{j}=\boldsymbol{x}_{0}^{j}U_{0,k}^{j},\quad\textrm{for all }k\in\mathbb{N},\label{eq:time_evolution_overall}
\end{equation}
where $U_{0,k}^{j}=M_{0}^{j}M_{1}^{j}\ldots M_{k-2}^{j}M_{k-1}^{j}$
and each $M_{k}^{j}$ is a $\mathbb{R}^{n_{\textrm{cell}}\times n_{\textrm{cell}}}$
row stochastic matrix obtained using Proposition \ref{thm:Markov_matrix_new}.

We first focus on the convergence of each agent's pmf vector. In the
following proof, we first show that the inhomogeneous Markov chain
is strongly ergodic and then show that the limit is the desired distribution
$\boldsymbol{\pi}$. Moreover, we assume that the swarm has not converged
if at least two or more agents are not in the correct location. If
at most one agent is not in its correct location,%
\footnote{The probability of the event, that at most one agent is not in its
correct location, is upper bounded by $(\frac{1}{n_{\textrm{rec}}})^{m-1}$.
For the simulation example discussed in Section \ref{sub:Numerical-Example-1},
it is approximately $10^{-7000}$.%
} then we state that the swarm has already converged and no further
convergence is necessary.

\begin{theorem} \label{thm:Convergence-of-imhomo-MC} \textit{(Convergence
of inhomogeneous Markov chains)} Each agent's time evolution of the
pmf vector $\boldsymbol{x}_{k}^{j}$, from any initial condition $\boldsymbol{x}_{0}^{j}\in\mathbb{R}^{n_{\textrm{cell}}}$,
is given by the inhomogeneous Markov chain (\ref{eq:time_evolution_overall}).
If each agent executes the PSG--IMC algorithm, then $\boldsymbol{x}_{k}^{j}$
asymptotically converges pointwise to the desired stationary distribution
$\boldsymbol{\pi}$, i.e., $\lim_{k\rightarrow\infty}\boldsymbol{x}_{k}^{j}=\boldsymbol{\pi}$
pointwise for all $j\in\{1,\ldots,m\}$. \end{theorem}
\begin{IEEEproof}
According to Proposition \ref{thm:Markov_matrix_new}, each matrix
$M_{k}^{j}$ is a function of the tuning parameter $\xi_{k}^{j}$
which is determined by (\ref{eq:tuning_eqn}) and bounded by $[0,1]$.
Let us define the bins with nonzero probabilities in $\boldsymbol{\pi}$
as recurrent bins. According to the design of the Markov matrix, if
$\xi_{k}^{j}>0$, then an agent can enter these recurrent bins from
all other bins. The bins which are not recurrent are called transient
bins and they have zero probabilities in $\boldsymbol{\pi}$. The
agents can only leave the transient bins when $\xi_{k}^{j}>0$, but
they can never enter a transient bin from any other bin due to the
structure of the Markov matrix in (\ref{eq:Markov_matrix_family_new}).
In this proof, we first consider the special case where all the bins
are recurrent and show that each agent's pmf vector converges to $\boldsymbol{\pi}$.
We next consider the general case where transient bins are also present
and show that this situation converges to the special case geometrically
fast.

\textbf{Case 1: all bins are recurrent, }i.e., $\boldsymbol{\pi}[i]>0,\thinspace\textrm{for all }i\in\{1,\ldots,n_{\textrm{cell}}\}$.

For $\xi_{k}^{j}>0$, Proposition \ref{thm:Markov_matrix_new} guarantees
that the matrix $M_{k}^{j}$ is positive, strongly connected and primitive.
We next prove that $\lim_{k\rightarrow\infty}U_{0,k}^{j}$ is also
a primitive matrix. Forward multiplication of two row stochastic Markov
matrices ($M_{s}^{j},\thinspace M_{s+1}^{j}$ with $\xi_{s}^{j},\thinspace\xi_{s+1}^{j}>0$
and $s\in\mathbb{Z}^{*}$) obtained using Proposition \ref{thm:Markov_matrix_new}
yields the positive matrix $M_{s}^{j}M_{s+1}^{j}$ which is a primitive
matrix.

If $\xi_{k}^{j}=0$, then $M_{k}^{j}=\mathbf{I}$ from (\ref{eq:Markov_matrix_family_new}).
The matrix product $U_{0,k}^{j}$ can be decomposed into two parts;
(a) the tuning parameter ($\xi_{s}^{j}$) of the matrices in the first
part is always zero and (b) the second part contains the remaining
Markov matrices with $\xi_{s}^{j}>0$. \vspace{-15pt}

\begin{align*}
U_{0,k}^{j} & =M_{0}^{j}(\xi_{0}^{j}>0)\thinspace M_{1}^{j}(\xi_{1}^{j}>0)\thinspace M_{2}^{j}(\xi_{2}^{j}=0)\thinspace M_{3}^{j}(\xi_{3}^{j}>0)\ldots\\
 & \quad M_{k-3}^{j}(\xi_{k-3}^{j}>0)\thinspace M_{k-2}^{j}(\xi_{k-2}^{j}=0)\thinspace M_{k-1}^{j}(\xi_{k-1}^{j}>0)\thinspace,\\
 & =\underbrace{\left(\mathbf{I}\ldots\mathbf{I}\right)}_{\text{1st part}}\cdot\underbrace{\left(M_{0}^{j}M_{1}^{j}M_{3}^{j}\ldots M_{k-3}^{j}M_{k-1}^{j}\right)}_{\text{2nd part}}\thinspace.
\end{align*}
Since the first part results in an identity matrix, the matrix product
$U_{0,k}^{j}$ can be completely defined by the second part containing
Markov matrices with $\xi_{k}^{j}>0$. In the matrix product $\lim_{k\rightarrow\infty}U_{0,k}^{j}$,
the tuning parameters ($\xi_{k}^{j}$) for the first few Markov matrices
are nonzero because the swarm starts from initial conditions that
are different from the desired formation. Hence there exists at least
one matrix in the second part of $\lim_{k\rightarrow\infty}U_{0,k}^{j}$.
Thus we can prove (by induction) that the matrix product $\lim_{k\rightarrow\infty}U_{0,k}^{j}$
is a primitive matrix.

Next, we prove that the matrix product $\lim_{k\rightarrow\infty}U_{0,k}^{j}$
is asymptotically homogeneous and strongly ergodic (See Definition
\ref{def:Strong-ergodic-asymp-homo} in Appendix). Let us find a positive
$\gamma$ independent of $k$ such that $\gamma\leq\min^{+}M_{k}^{j}[i,\ell]$
for all $i,\ell\in\{1,\ldots,n_{\mathrm{cell}}\}$. Let $\varepsilon_{\mathrm{cons}}\leq\frac{1}{m}$
and $n_{\textrm{loop}}$ is computed using Theorem \ref{thm:Bayesian_Consensus_Filtering_Static_balanced_graphs}.
If at least two agents are not in the correct location, i.e., $\sum_{R[i]\in\mathcal{R}}|\mathcal{F}_{k}^{\star}[i]-\boldsymbol{\pi}[i]|\geq\frac{2}{m}$;
then due to the quantization of the pmf by the number of agents $m$,
the smallest positive tuning parameter $\xi_{\min}$ is given by:
\vspace{-10pt}

\begin{equation}
\xi_{\min}=\frac{1}{2^{\frac{3}{2}}m}\leq\min{}_{j\in\{1,\ldots,m\},\thinspace k\in\mathbb{Z}^{*}}^{+}\xi_{k}^{j}\thinspace.
\end{equation}
The smallest positive element in the $\boldsymbol{\alpha}_{k}^{j}$
vector in (\ref{eq:alpha_vector}) is given by: \vspace{-10pt}

\begin{align}
\alpha_{\min} & =\left(1-\frac{\max_{i,q\in\{1,\ldots,n_{\textrm{cell}}\}}\textrm{dis}(R[q],R[i])}{\max_{i,q\in\{1,\ldots,n_{\textrm{cell}}\}}\textrm{dis}(R[q],R[i])+1}\right)\thinspace.
\end{align}
Finally, the smallest positive element in the stationary distribution
$\boldsymbol{\pi}$ is given by $\pi_{\min}=$ $\left(\min{}_{i\in\{1,\ldots,n_{\mathrm{cell}}\}}^{+}\boldsymbol{\pi}[i]\right)$.
The diagonal and off-diagonal elements of the Markov matrix $M_{k}^{j}$
designed using (\ref{eq:Markov_matrix_family_new}) are given by:
\vspace{-15pt}

\begin{align}
M_{k}^{j}[i,i] & =1-\xi_{k}^{j}\boldsymbol{\alpha}_{k}^{j}[i]+\tfrac{\xi_{k}^{j}}{\boldsymbol{\pi\alpha}_{k}^{j}}(\boldsymbol{\alpha}_{k}^{j}[i])^{2}\boldsymbol{\pi}[i]\thinspace,\\
M_{k}^{j}[i,\ell] & =\tfrac{\xi_{k}^{j}}{\boldsymbol{\pi\alpha}_{k}^{j}}\boldsymbol{\alpha}_{k}^{j}[i]\boldsymbol{\pi}[\ell]\boldsymbol{\alpha}_{k}^{j}[\ell]\thinspace,\textrm{ where }i\not=\ell.\label{eq:off_diagonal_term}
\end{align}
Irrespective of the choice of $\xi_{k}^{j}$, the diagonal elements
of $M_{k}^{j}$ will not tend to zero. But the off-diagonal elements
of $M_{k}^{j}$ will tend to zero if $\xi_{k}^{j}\rightarrow0$. Moreover,
the largest possible value of $\boldsymbol{\pi\alpha}_{k}^{j}=1$,
when $\boldsymbol{\alpha}_{k}^{j}=\mathbf{1}$. Hence the smallest
nonzero element in $M_{k}^{j}$ is lower bounded by the smallest possible
values of the terms in (\ref{eq:off_diagonal_term}).

Thus we get a positive $\gamma$ that is independent of $k$, i.e.,
$\gamma=\xi_{\min}\alpha_{\min}^{2}\pi_{\min}$. If $\xi_{k}^{j}=0$,
then $\min{}_{i,\ell\in\{1,\ldots,n_{\mathrm{cell}}\}}^{+}M_{k}^{j}[i,\ell]=1>\gamma$.
Since $M_{k}^{j}$ is row stochastic, $M_{k}^{j}[i,\ell]\leq1$ for
all $i,\ell\in\{1,\ldots,n_{\mathrm{cell}}\}$ and $k\in\mathbb{Z}^{*}$.
Thus each Markov matrix satisfies the condition (\ref{eq:condition_C})
in Lemma \ref{thm:Asymptotic-Homogeneity-Strong-Ergodicity} given
in Appendix. Since $\boldsymbol{\psi}=\boldsymbol{\pi}$ in (\ref{eq:condition_asymtotic_homogeneity}),
it follows from Lemma \ref{thm:Asymptotic-Homogeneity-Strong-Ergodicity}
that the matrix product $\lim_{k\rightarrow\infty}U_{0,k}^{j}$ is
asymptotically homogeneous and strongly ergodic.

Note that $M_{k}^{j}$ for all $k\in\mathbb{Z}^{*}$, by Proposition
\ref{thm:Markov_matrix_new}, have $\boldsymbol{\pi}$ as their left
eigenvector for the eigenvalue $1$. Hence $\boldsymbol{v}_{0}^{j}=\boldsymbol{\pi}$
is a set of absolute probability vectors for $\lim_{k\rightarrow\infty}U_{0,k}^{j}$.
According to Lemma \ref{thm:Uniqueness-Absolute-Probability-Vectors}
given in Appendix, $\boldsymbol{v}_{0}^{j}=\boldsymbol{\pi}$ is the
unique set of absolute probability vectors for $\lim_{k\rightarrow\infty}U_{0,k}^{j}$
\cite{Ref:Blackwell45} and $\boldsymbol{w}_{0}^{j}=\boldsymbol{\pi}$
in (\ref{eq:strong_ergodicity}). Hence the individual pmfs asymptotically
converge to: \vspace{-10pt}

\begin{equation}
\lim_{k\rightarrow\infty}\boldsymbol{x}_{k}^{j}=\lim_{k\rightarrow\infty}\boldsymbol{x}_{0}^{j}U_{0,k}^{j}=\boldsymbol{x}_{0}^{j}\mathbf{1}\boldsymbol{\pi}=\boldsymbol{\pi}\thinspace.
\end{equation}

\textbf{Case 2: both transient and recurrent bins are present, }i.e.,
$\boldsymbol{\pi}[i]>0$ for all $i\in\{1,\ldots,n_{\textrm{rec}}\}$
and $\boldsymbol{\pi}[i]=0$ for all $i\in\{n_{\textrm{rec}}+1,\ldots,n_{\textrm{cell}}\}$.\textbf{ }

Without loss of generality, let us reorder the bins such that the
first $n_{\textrm{rec}}$ bins are recurrent, and the remaining bins
are transient. Hence the pmf vector $\boldsymbol{x}_{k}^{j}$ is also
reordered so that $\boldsymbol{x}_{k}^{j}[i],\thinspace i\in\{1,\ldots,n_{\textrm{rec}}\}$
is the probability of the event that the $j^{\textrm{th}}$ agent
is in the $R[i]$ recurrent bin at the $k^{\textrm{th}}$ time instant.
It is known that the agents leave the transient bins geometrically
fast \cite[Theorem 4.3, pp. 120]{Ref:Seneta06}. Once all the agents
are in the recurrent bins, then the situation is similar to Case 1
with $n_{\textrm{rec}}$ bins. The submatrix $M_{k,\textrm{sub}}^{j}\in\mathbb{R}^{n_{\textrm{rec}}\times n_{\textrm{rec}}}$
of the original Markov matrix, as shown in Fig. \ref{fig:Submatrix},
is primitive when $\xi_{k}^{j}>0$. Then, it follows from Case 1 that
$\lim_{k\rightarrow\infty}\boldsymbol{x}_{k}^{j}=\boldsymbol{\pi}.$
\end{IEEEproof}
\begin{figure}
\begin{centering}
\includegraphics[bb=80bp 190bp 910bp 415bp,clip,width=3in]{Markov_submatrix.pdf}
\par\end{centering}

\protect\caption{Submatrix $M_{k,\textrm{sub}}^{j}\in\mathbb{R}^{n_{\textrm{rec}}\times n_{\textrm{rec}}}$
of only the recurrent bins in the original Markov matrix $M_{k}^{i}$.
\label{fig:Submatrix}}
\end{figure}

Since each agent's pmf vector converges to $\boldsymbol{\pi}$, we
now focus on the convergence of the current swarm distribution.

\begin{theorem} \label{thm:Convergence-swarm-dist} \textit{(Convergence
of swarm distribution to desired formation)} If the number of agents,
executing the PSG--IMC algorithm, tends to infinity; then the current
swarm distribution ($\mathcal{F}_{k}^{\star}=\frac{1}{m}\sum_{j=1}^{m}\boldsymbol{r}_{k}^{j}$)
asymptotically converges pointwise to the desired stationary distribution
$\boldsymbol{\pi}$, i.e., $\lim_{m\rightarrow\infty}\lim_{k\rightarrow\infty}\mathcal{F}_{k}^{\star}=\boldsymbol{\pi}$
pointwise. \end{theorem}
\begin{IEEEproof}
Let $X_{k}^{j}[i]$ denote the independent Bernoulli random variable
representing the event that the $j^{\textrm{th}}$ agent is actually
located in bin $R[i]$ at the $k^{\textrm{th}}$ time instant, i.e.,
$X_{k}^{j}[i]=1$ if $\boldsymbol{r}_{k}^{j}[i]=1$ and $X_{k}^{j}[i]=0$
otherwise. Let $X_{\infty}^{j}[i]$ denote the random variable $\lim_{k\rightarrow\infty}X_{k}^{j}[i]$.
Theorem \ref{thm:Convergence-of-imhomo-MC} implies that the success
probability of $X_{\infty}^{j}[i]$ is given by $\mathbb{P}\left(X_{\infty}^{j}[i]=1\right)=\lim_{k\rightarrow\infty}x_{k}^{j}[i]=\boldsymbol{\pi}[i]$.
Hence $\mathbb{E}\left[X_{\infty}^{j}[i]\right]=\boldsymbol{\pi}[i]\cdot1+(1-\boldsymbol{\pi}[i])\cdot0=\boldsymbol{\pi}[i]$,
where $\mathbb{E}[\cdot]$ is the expected value of the random variable.
Let $S_{\infty}^{m}[i]=X_{\infty}^{1}[i]+\ldots+X_{\infty}^{m}[i]$.
As the random variables $X_{\infty}^{j}[i]$ are independent and identically
distributed, the strong law of large numbers (cf. \cite[pp. 85]{Ref:Billingsley95})
states that: \vspace{-5pt}

\begin{equation}
\mathbb{P}\left(\lim_{m\rightarrow\infty}\frac{S_{\infty}^{m}[i]}{m}=\boldsymbol{\pi}[i]\right)=1\thinspace.\label{eq:SLLN_1}
\end{equation}
The final swarm distribution is also given by $\lim_{k\rightarrow\infty}\mathcal{F}_{k}^{\star}[i]=\frac{1}{m}\sum_{j=1}^{m}\boldsymbol{r}_{k}^{j}[i]=\frac{S_{\infty}^{m}[i]}{m}$.
Hence (\ref{eq:SLLN_1}) implies that $\lim_{m\rightarrow\infty}\lim_{k\rightarrow\infty}\mathcal{F}_{k}^{\star}=\boldsymbol{\pi}$
pointwise. It follows from Scheff$\acute{\textrm{e}}$'s theorem \cite[pp. 84]{Ref:Durrett05}
that the measure induced by $\mathcal{F}_{k}^{\star}$ on the $\sigma$-algebra
of $\mathcal{R}$ converges in total variation to the measure induced
by $\boldsymbol{\pi}$.
\end{IEEEproof}
In practical scenarios, the number of agents is finite, hence we need
to specify a convergence error threshold. The following theorem gives
the minimum number of agents needed to establish $\varepsilon$-convergence
of the swarm.

\begin{theorem} For some acceptable convergence error $\varepsilon_{\mathrm{conv}}>0$
and $\varepsilon_{\mathrm{bin}}>0$, if the number of agents is at
least $m\geq\frac{1}{4\varepsilon_{\mathrm{bin}}^{2}\varepsilon_{\mathrm{conv}}}$,
then the pointwise error probability for each bin is bounded by $\varepsilon_{\mathrm{conv}}$,
i.e., $\mathbb{P}\left(\left|\frac{S_{\infty}^{m}[i]}{m}-\boldsymbol{\pi}[i]\right|>\varepsilon_{\mathrm{bin}}\right)\leq\varepsilon_{\mathrm{conv}},\thinspace\forall i\in\{1,\ldots,n_{\mathrm{cell}}\}$.
\label{thm:Convergence-swarm-dist-practical} \end{theorem}
\begin{IEEEproof}
The variance of the independent random variable from Theorem \ref{thm:Convergence-swarm-dist}
is $\textrm{Var}(X_{\infty}^{j}[i])=\boldsymbol{\pi}[i](1-\boldsymbol{\pi}[i])$,
hence $\textrm{Var}(\frac{S_{\infty}^{m}[i]}{m})=\frac{\boldsymbol{\pi}[i](1-\boldsymbol{\pi}[i])}{m}$.
The Chebychev\textquoteright s inequality (cf. \cite[Theorem 1.6.4, pp. 25]{Ref:Durrett05})
implies that for any $\varepsilon_{\mathrm{bin}}>0$, the pointwise
error probability for each bin is bounded by: \vspace{-10pt}

\[
\mathbb{P}\left(\left|\frac{S_{\infty}^{m}[i]}{m}-\boldsymbol{\pi}[i]\right|\!>\!\varepsilon_{\mathrm{bin}}\right)\!\leq\!\frac{\boldsymbol{\pi}[i](1-\boldsymbol{\pi}[i])}{m\varepsilon_{\mathrm{bin}}^{2}}\!\leq\!\frac{1}{4m\varepsilon_{\mathrm{bin}}^{2}}.
\]
Hence, the minimum number of agents is given by $\frac{1}{4m\varepsilon_{\mathrm{bin}}^{2}}\leq\varepsilon_{\mathrm{conv}}$.
\end{IEEEproof}
We now study the convergence of the tuning parameter and the Markov
matrix.

\begin{remark} \label{rem:Markov_identity} \textit{(Convergence
of Markov matrix to identity matrix)} If the consensus error tends
to zero, i.e., $\varepsilon_{\mathrm{cons}}\rightarrow0$, then $\hat{\mathcal{F}}_{k,n_{\mathrm{loop}}}^{j}=\mathcal{F}_{k}^{\star}$.
If the number of agents, executing the PSG--IMC algorithm, tends to
infinity then Theorem \ref{thm:Convergence-swarm-dist} implies that
the limiting tuning parameter (\ref{eq:tuning_eqn}) is given by:
\vspace{-15pt}

\begin{align*}
 & \lim_{m\rightarrow\infty}\lim_{k\rightarrow\infty}\lim_{n_{\mathrm{loop}}\rightarrow\infty}\xi_{k}^{j}=\lim_{m\rightarrow\infty}\lim_{k\rightarrow\infty}\lim_{n_{\mathrm{loop}}\rightarrow\infty}D_{H}(\boldsymbol{\pi},\hat{\mathcal{F}}_{k,n_{\textrm{loop}}}^{j})\\
 & \quad=\lim_{m\rightarrow\infty}\lim_{k\rightarrow\infty}D_{H}(\boldsymbol{\pi},\mathcal{F}_{k}^{\star})=D_{H}(\boldsymbol{\pi},\boldsymbol{\pi})=0\thinspace.
\end{align*}
Therefore Proposition \ref{thm:Markov_matrix_new} implies that the
Markov transition matrix $\lim_{m\rightarrow\infty}\lim_{k\rightarrow\infty}\lim_{n_{\mathrm{loop}}\rightarrow\infty}M_{k}^{j}=\mathbf{I}$.
\hfill $\Box$ \end{remark}

In practical scenarios, $n_{\mathrm{loop}}$ is finite, hence $\xi_{k}^{j}$
may not converge to zero. A practical work-around to avoid transitions
due to nonzero $\xi_{k}^{j}$ is to set $M_{k}^{j}=\mathbf{I}$ when
$\xi_{k}^{j}$ is sufficiently small and has leveled out. In the next
section, we use the above theorems to prove convergence of the PSG--IMC
algorithm with motion constraints.

\section{Motion Constraints \label{sec:Handling-Motion-Constraints}}

In this section, we introduce additional constraints on the motion
of agents and study their effects on the convergence of the swarm.
We first introduce the motion constraints and the corresponding trapping
problem. Next, we discuss the strategy for leaving the trapping region
and an additional condition on the desired formation. Finally, Theorem
\ref{thm:Convergence-of-imhomo-MC-motion-const} shows that each agent's
pmf vector $\boldsymbol{x}_{k}^{j}$ asymptotically converges pointwise
to the desired formation $\boldsymbol{\pi}$, if it executes the PSG--IMC
algorithm with motion constraints.

The agents in a particular bin can only transition to some bins but
cannot transition to other bins because of the dynamics or physical
constraints. These (possibly time-varying) motion constraints are
specified in a matrix $A_{k}^{j}$ as follows: \vspace{-15pt}

\begin{align}
A_{k}^{j}[i,\ell] & =\begin{cases}
1 & \textrm{if the \ensuremath{j^{\textrm{th}}}agent can transition to }R[\ell]\\
0 & \textrm{if the \ensuremath{j^{\textrm{th}}}agent cannot transition to }R[\ell]
\end{cases},\nonumber \\
 & \thinspace\textrm{where }\boldsymbol{r}_{k}^{j}[i]=1,\thinspace\textrm{for all }\ell\in\{1,\ldots,n_{\textrm{cell}}\}\thinspace.\label{eq:motion_constraints}
\end{align}
\begin{assumption} The matrix $A_{k}^{j}$ is symmetric and the graph
conforming to the $A_{k}^{j}$ matrix is strongly connected. Moreover,
an agent can always choose to remain in its present bin, i.e., $A_{k}^{j}[i,i]=1$
for all $i\in\{1,\ldots,n_{\textrm{cell}}\}$ and $k\in\mathbb{Z}^{*}$.
\hfill $\Box$ \label{assump_SC} \end{assumption}

In this paper, we introduce a simple, intuitive method for handling
motion constraints, such that the convergence results in Section \ref{sec:Convergence-of-Inhomogeneous}
are not affected. The key idea is to modify the Markov matrix $M_{k}^{j}$
designed in Proposition \ref{thm:Markov_matrix_new} to capture the
motion constraints in (\ref{eq:motion_constraints}).

\begin{proposition} \label{thm:Markov_matrix_modified}\textit{ }Let
$\tilde{M}_{k}^{j}$ represent the modified Markov matrix that satisfies
motion constraints, which is obtained from the original Markov matrix
$M_{k}^{j}$ given by Proposition \ref{thm:Markov_matrix_new}. For
each transition that is not allowed by the motion constraint, i.e.,
$A_{k}^{j}[i,\ell]=0$, the corresponding transition probability in
$M_{k}^{j}$ is added to the diagonal element in $\tilde{M}_{k}^{j}$.
First, let $\tilde{M}_{k}^{j}=M_{k}^{j}$. Then set: \vspace{-10pt}

\[
\tilde{M}_{k}^{j}[i,i]=M_{k}^{j}[i,i]+\sum_{\ell\in\{1,\ldots,n_{\mathrm{cell}}:A_{k}^{j}[i,\ell]=0\}}M_{k}^{j}[i,\ell]\thinspace.
\]
Finally, for all $i,\ell\in\{1,\ldots,n_{\textrm{cell}}\}$: \vspace{-10pt}

\begin{align*}
\textrm{if}\quad & A_{k}^{j}[i,\ell]=0,\textrm{ then set }\tilde{M}_{k}^{j}[i,\ell]=0\thinspace.
\end{align*}
The resulting row stochastic Markov matrix $\tilde{M}_{k}^{j}$ has
$\boldsymbol{\pi}$ as its stationary distribution (i.e., $\boldsymbol{\pi}\tilde{M}_{k}^{j}=\boldsymbol{\pi}$).
\end{proposition}
\begin{IEEEproof}
The modified Markov matrix $\tilde{M}_{k}^{j}$ is row stochastic
since $\tilde{M}_{k}^{j}\mathbf{1}=M_{k}^{j}\mathbf{1}=\mathbf{1}$.
Moreover, $\tilde{M}_{k}^{j}$ also has $\boldsymbol{\pi}$ as its
stationary distribution because of the reversible property of $M_{k}^{j}$,
i.e. $\boldsymbol{\pi}[\ell]M_{k}^{j}[\ell,i]=\boldsymbol{\pi}[i]M_{k}^{j}[i,\ell]$.
Hence $\tilde{M}_{k}^{j}$ is indeed a valid Markov matrix.
\end{IEEEproof}
For a bin $R[i]$, let us define $\mathcal{A}_{k}^{j}(R[i])$ as the
set of all bins that the $j^{\textrm{th}}$ agent can transition to
at the $k^{\textrm{th}}$ time instant: \vspace{-10pt}

\begin{equation}
\mathcal{A}_{k}^{j}(R[i]):=\left\{ R[\ell]:\ell\in\{1,\dots,n_{\textrm{cell}}\};\thinspace A_{k}^{j}[i,\ell]=1\right\} .
\end{equation}
Similarly, let us define $\Pi$ as the set of all bins that have nonzero
probabilities in the desired formation ($\boldsymbol{\pi}$): \vspace{-10pt}

\begin{equation}
\Pi:=\left\{ R[\ell]\thinspace:\thinspace\ell\in\{1,\dots,n_{\textrm{cell}}\};\textrm{ and }\boldsymbol{\pi}[\ell]>0\right\} \thinspace.
\end{equation}
As defined before in the proof of Theorem \ref{thm:Convergence-of-imhomo-MC},
the bins in the set $\Pi$ are called recurrent bins while those not
in $\Pi$ are called transient bins.

\begin{remark} \textit{(Trapping Problem)} If the $j^{\textrm{th}}$
agent is actually located in bin $R[i]$ and we observe that $\mathcal{A}_{k}^{j}(R[i])\cap\Pi=\emptyset$
for all $k\in\mathbb{Z}^{*}$, then the $j^{\textrm{th}}$ agent is
trapped in the bin $R[i]$ forever. Let us define $\mathcal{T}_{k}^{j}$
as the set of all bins that satisfy this trapping condition: \vspace{-10pt}

\begin{equation}
\mathcal{T}_{k}^{j}:=\bigcup{}_{i\in\{1,\ldots,n_{\textrm{cell}}\}}\left\{ R[i]\thinspace:\thinspace\mathcal{A}_{k}^{j}(R[i])\cap\Pi=\emptyset\right\} \thinspace.\label{eq:trapping_region}
\end{equation}
To avoid this trapping problem, we enforce a secondary condition on
the $j^{\textrm{th}}$ agent if it is actually located in a bin belonging
to the set $\mathcal{T}_{k}^{j}$. For each bin $R[i]\in\mathcal{T}_{k}^{j}$,
we chose another bin $\Psi(R[i])$; where $\Psi(R[i])\in\mathcal{A}_{k}^{j}(R[i])$
and either $\Psi(R[i])\not\in\mathcal{T}_{k}^{j}$ or $\Psi(R[i])$
is close to other bins which are not in $\mathcal{T}_{k}^{j}$. The
secondary condition on the $j^{\textrm{th}}$ agent is that it will
transition to bin $\Psi(R[i])$ during the $k^{\textrm{th}}$ time
step. Moreover, the agents in the transient bins, which are not in
the trapping region, eventually transition to the recurrent bins.
We also speed up this process. The information about exiting the trapping
region and the transient bins is captured by the matrix $C_{k}^{j}\in\mathbb{R}^{n_{\textrm{cell}}\times n_{\textrm{cell}}}$
given by: \vspace{-10pt}

\begin{align}
C_{k}^{j}[i,\ell] & =\begin{cases}
1 & \textrm{if }R[i]\in\mathcal{T}_{k}^{j}\textrm{ and }R[\ell]=\Psi(R[i])\\
\frac{1}{\left|\mathcal{A}_{k}^{j}(R[i])\cap\Pi\right|} & \textrm{else if }R[i]\not\in\mathcal{T}_{k}^{j}\textrm{ and }R[i]\not\in\Pi\\
 & \thinspace\thinspace\thinspace\textrm{and }R[\ell]\in\mathcal{A}_{k}^{j}(R[i])\cap\Pi\\
0 & \textrm{otherwise}
\end{cases},\nonumber \\
 & \quad\textrm{for all }i,\ell\in\{1,\ldots,n_{\textrm{cell}}\}\thinspace,\label{eq:C_matrix}
\end{align}
which is used instead of the Markov matrix in (\ref{eq:time_evolution_single}).
Note that this secondary condition would not cause an infinite loop
as the graph conforming to the $A_{k}^{j}$ matrix is strongly connected.
\hfill $\Box$ \end{remark}

\begin{figure}
\begin{centering}
\begin{tabular}{cc}
\includegraphics[bb=360bp 160bp 615bp 415bp,clip,width=1in]{Pi_example.pdf} & \includegraphics[bb=0bp 0bp 285bp 284bp,clip,width=1in]{Desired_distribution_Illini_logo_with_motion_constraint_v6.pdf}\tabularnewline
(a) & (b)\tabularnewline
\end{tabular}
\par\end{centering}

\begin{centering}
\protect\caption{(a) The red region denotes the set $\Pi$. The motion constraints
are such that the agents can only transition to the immediate neighboring
bins. Note that each of the four corners form four subsets of $\Pi$,
such that any agent cannot transition from one subset to another subset
without exiting the set $\Pi$. (b) The UIUC logo, marked in red,
is the desired formation $\boldsymbol{\pi}$. Although $\mathcal{R}$
is partitioned into $900$ bins, the set $\Pi$ contains only $262$
bins. The trapping region $\mathcal{T}_{k}^{j}$ for all $k\in\mathbb{N}$
is marked in green. \label{fig:Pi-example} }

\par\end{centering}

\vspace{-20pt}
\end{figure}

It is possible that the set of all the bins that have nonzero probabilities
in the desired formation ($\Pi$) can be decomposed into subsets,
such that any agent cannot transition from one subset to another subset
without exiting the set $\Pi$ (for example see Fig. \ref{fig:Pi-example}(a)).
Since our proposed algorithm suggests that the agents will always
transition within $\Pi$ after entering it, the agents in one such
subset will never transition to the other subsets. Hence, the proposed
algorithms will not be able to achieve the desired formation, as the
agents in each subset of $\Pi$ are trapped within that subset. In
order to avoid such situations, we need the following assumption on
$\Pi$ and $A_{k}^{j}$.

\begin{assumption} The set of all bins that have nonzero probabilities
in the desired formation ($\Pi$) and the matrix specifying the motion
constraints ($A_{k}^{j}$) are such that each agent can transition
from any bin in $\Pi$ to any other bin in $\Pi$, without exiting
the set $\Pi$ while satisfying the motion constraints. \hfill $\Box$
\label{assump_Pi} \end{assumption}

For the $j^{\textrm{th}}$ agent in bin $R[i]$, random sampling of
the Markov chain selects the bin $R[q]$ in line 15--16 of \textbf{Algorithm
\ref{alg:PGA-inhomo-motion-const}}. Then the time evolution of the
pmf vector $\boldsymbol{x}_{k}^{j}$ can be written as: \vspace{-15pt}

\begin{equation}
\boldsymbol{x}_{k+1}^{j}=\boldsymbol{x}_{k}^{j}B_{k}^{j},\thinspace\textrm{where }B_{k}^{j}=\begin{cases}
C_{k}^{j} & \textrm{ if }R[i]\not\in\Pi\\
\tilde{M}_{k}^{j} & \textrm{ otherwise }
\end{cases}\thinspace.\label{eq:time_evolution_motion_constraint}
\end{equation}
Note that the Markov matrix $\tilde{M}_{k}^{j}$ in (\ref{eq:time_evolution_motion_constraint})
is computed using Proposition \ref{thm:Markov_matrix_new} and then
modified using Proposition \ref{thm:Markov_matrix_modified}. Let
$V_{0,k}^{j}$ denote the row stochastic matrix defined by the forward
matrix multiplication: \vspace{-15pt}

\begin{equation}
V_{0,k}^{j}=B_{0}^{j}B_{1}^{j}\ldots B_{k-2}^{j}B_{k-1}^{j},\quad\textrm{for all }k\in\mathbb{N}\thinspace,\label{eq:matrix_product-1}
\end{equation}
where each $B_{k}^{j}$ is given by (\ref{eq:time_evolution_motion_constraint}).
Similar to (\ref{eq:time_evolution_overall}), the evolution of the
probability vector $\boldsymbol{x}_{k}^{j}$ from an initial condition
to any $k^{\textrm{th}}$ time instant is given by $\boldsymbol{x}_{k}^{j}=\boldsymbol{x}_{0}^{j}V_{0,k}^{j}$
for all $k\in\mathbb{\mathbb{N}}$.

\begin{theorem} \label{thm:Convergence-of-imhomo-MC-motion-const}
\textit{(Convergence of inhomogeneous Markov chains with motion constraints)}
Under Assumptions \ref{assump_SC} and \ref{assump_Pi}, each agent's
time evolution of the pmf vector $\boldsymbol{x}_{k}^{j}$, from any
initial condition $\boldsymbol{x}_{0}^{j}\in\mathbb{R}^{n_{\textrm{cell}}}$,
is given by the inhomogeneous Markov chain $\boldsymbol{x}_{k}^{j}=\boldsymbol{x}_{0}^{j}V_{0,k}^{j}$.
If each agent executes the PSG--IMC algorithm, then $\boldsymbol{x}_{k}^{j}$
asymptotically converges pointwise to the desired stationary distribution
$\boldsymbol{\pi}$, i.e., $\lim_{k\rightarrow\infty}\boldsymbol{x}_{k}^{j}=\boldsymbol{\pi}$
pointwise for all $j\in\{1,\ldots,m\}$. \end{theorem}
\begin{IEEEproof}
We first show that there are finitely many occurrences of the $C_{k}^{j}$
matrix in the matrix product $\lim_{k\rightarrow\infty}V_{0,k}^{j}$.
Due to the design of the Markov matrix in Proposition \ref{thm:Markov_matrix_new},
the bins in the set $\Pi$ are absorbing; i.e., if an agent enters
any of the bins in the set $\Pi$, then it cannot leave the set $\Pi$.
Next we notice that if $R[\ell]$ is a transient bin but not in the
trapping region, then the only possible transitions are to the bins
in $\Pi$. Moreover, once the agent exits the set $\mathcal{T}_{k}^{j}$,
it cannot enter it again. Finally, the number of steps inside the
transient bins is limited by the total number of bins $n_{\textrm{cell}}$.
Hence the $C_{k}^{j}$ can only occur finite number of times in the
matrix product $\lim_{k\rightarrow\infty}V_{0,k}^{j}$. The new initial
condition of the agent is obtained by forward multiplying the previous
initial condition with the $C_{k}^{j}$ matrices: \vspace{-10pt}

\begin{equation}
\tilde{\boldsymbol{x}}_{0}^{j}=\boldsymbol{x}_{0}^{j}C_{0}^{j}C_{1}^{j}\ldots C_{s-1}^{j}C_{s}^{j}\thinspace,
\end{equation}
where $s$ is the maximum number of steps that the $j^{\textrm{th}}$
agent makes in the transient bins. Hence the overall time evolution
of the pmf $\boldsymbol{x}_{k}^{j}$ can be written as $\lim_{k\rightarrow\infty}\boldsymbol{x}_{k}^{j}=\lim_{k\rightarrow\infty}\boldsymbol{x}_{0}^{j}V_{0,k}^{j}=\lim_{k\rightarrow\infty}\tilde{\boldsymbol{x}}_{0}^{j}U_{0,k}^{j}$.
Here the matrix product $\lim_{k\rightarrow\infty}U_{0,k}^{j}$ is
the product of modified Markov matrices obtained using Proposition
\ref{thm:Markov_matrix_modified}.

Once the agent exits the transient bins, the situation is exactly
similar to that discussed in Case 1 of the proof of Theorem \ref{thm:Convergence-of-imhomo-MC}.
Hence, we now focus on proving that the submatrix $\tilde{M}_{k,\textrm{sub}}^{j}\in\mathbb{R}^{n_{\textrm{rec}}\times n_{\textrm{rec}}}$
of the original Markov matrix $\tilde{M}_{k}^{j}$, similar to that
shown in Fig. \ref{fig:Submatrix}, is primitive when $\xi_{k}^{j}>0$.
The modified Markov submatrix $\tilde{M}_{k,\textrm{sub}}^{j}$ is
a nonnegative matrix due to the motion constraints $A_{k}^{j}$. But
$\tilde{M}_{k,\textrm{sub}}^{j}$ is strongly connected and irreducible
when $\xi_{k}^{j}>0$ due to Assumptions \ref{assump_SC} and \ref{assump_Pi}.
Since $(\tilde{M}_{k,\textrm{sub}}^{j})^{n_{\textrm{rec}}}>0$, the
primitive matrix theorem \cite[Theorem 8.5.2, pp. 516]{Ref:Horn85}
implies that $\tilde{M}_{k,\textrm{sub}}^{j}$ is a primitive matrix.
Then it follows from Case 1 that the product of submatrices $\lim_{k\rightarrow\infty}U_{0,k,\textrm{sub}}^{j}$
is also a primitive matrix. Finally, it follows from the proof of
Theorem \ref{thm:Convergence-of-imhomo-MC} that $\lim_{k\rightarrow\infty}\boldsymbol{x}_{k}^{j}=\boldsymbol{\pi}$
pointwise for all $j\in\{1,\ldots,m\}$.
\end{IEEEproof}
Note that Theorem \ref{thm:Convergence-swarm-dist} and Remark \ref{rem:Markov_identity}
can be directly applied to satisfy the first two objectives of PSG--IMC,
even under motion constraints.

\begin{figure}
\begin{centering}
\begin{tabular}{cc}
\includegraphics[bb=0bp 0bp 529bp 398bp,clip,width=1.3in]{HD_num_trans_Illini_with_motion_const_v75.png} & \includegraphics[bb=0bp 0bp 527bp 398bp,clip,width=1.3in]{HD_num_trans_Illini_with_motion_const_v76.png}\tabularnewline
(a) & (b)\tabularnewline
\end{tabular}
\par\end{centering}

\begin{centering}
\protect\caption{The PSG--IMC algorithm is compared with the homogeneous PGA algorithm
using Monte Carlo simulations. The figures show (a) convergence of
the swarm to the desired formation in terms of HD and (b) the number
of agents transitioning per time step, along with their $3\sigma$
error bars. \label{fig:Desired-dist-Convergence-with-motion-const}}

\par\end{centering}

\vspace{-10pt}
\end{figure}

\vspace{-10pt}

\subsection{Numerical Example \label{sub:Numerical-Example-1}}

In this example, the PSG--IMC algorithm with motion constraints is
used to guide a swarm containing $m=3000$ agents to the desired formation
$\boldsymbol{\pi}$ associated with the UIUC logo shown in Fig. \ref{fig:Pi-example}(b).
Monte Carlo simulations were performed to compare the PSG--IMC algorithm
with the homogeneous PGA algorithm and the cumulative results from
$50$ runs are shown in Fig. \ref{fig:Desired-dist-Convergence-with-motion-const}.
As shown in Fig. \ref{fig:Color-plot-of-agents-with-motion-const},
at $k=1$, each simulation starts with the agents uniformly distributed
across $\mathcal{R}\subset\mathbb{R}^{2}$, which is partitioned into
$30\times30$ bins. Each agent independently executes the PSG--IMC
with motion constraints, illustrated in \textbf{Algorithm \ref{alg:PGA-inhomo-motion-const}}.
During the consensus stage, each agent is allowed to communicate with
those agents which are at most $10$ steps away. The communication
matrix $P_{k}$ is generated using Metropolis weights \cite{Ref:Boyd04},
hence it satisfies Assumption \ref{assump_weights2}. Moreover, each
agent is allowed to transition to only those bins which are at most
$5$ steps away.%
\footnote{This simulation video is available at \url{http://youtu.be/KFFCYHgLfvw}.%
}

As shown in the HD graph in Fig. \ref{fig:Desired-dist-Convergence-with-motion-const}(a),
the desired formation is almost achieved within the first $200$ time
steps for all simulation runs. After the $500^{\textrm{th}}$ time
step, the swarm is externally damaged by eliminating approximately
$460\pm60$ agents from the middle section of the formation. This
can been seen by comparing the images for the $500^{\textrm{th}}$
and $501^{\textrm{st}}$ time step in Fig. \ref{fig:Color-plot-of-agents-with-motion-const}
and the discontinuity during the $501^{\textrm{st}}$ time step in
Fig. \ref{fig:Desired-dist-Convergence-with-motion-const}. Note that
the swarm always recovers from this damage and attains the desired
formation within another $100$ time steps. Thus the third objective
of PSG--IMC, stated in Section \ref{sec:Problem-Statement}, is also
achieved. From the correlation between the plots of HD and number
of transitions in Fig. \ref{fig:Desired-dist-Convergence-with-motion-const}(b),
we can infer that the agents transition only when necessary.

During the first $500$ time steps, each agent in the PSG--IMC algorithm
undergoes just $10$ transitions, compared to approximately $60$
transitions in the homogeneous PGA algorithm. Moreover, the swarm
of agents executing the PSG--IMC algorithm converges faster to the
desired formation during this stage as seen in Fig. \ref{fig:Desired-dist-Convergence-with-motion-const}(a).
Hence, it is evident from the cumulative results in Fig. \ref{fig:Desired-dist-Convergence-with-motion-const}
that the PSG--IMC algorithm is more efficient than the homogeneous
PGA algorithm.

\begin{figure}[h]
\begin{centering}
\includegraphics[bb=0bp 0bp 405bp 285bp,clip,width=3in]{six_diff_views_of_Illini_with_motion_const_v6.pdf}
\par\end{centering}

\vspace{-5pt}

\begin{centering}
\protect\caption{Histogram plots of the swarm distribution at different time instants
for $3000$ agents, in a sample run of the Monte Carlo simulation.
The colorbar represents the pmf of the swarm distribution. Each agent
is allowed to transition to only those bins which are at most $5$
steps away. \label{fig:Color-plot-of-agents-with-motion-const}}

\par\end{centering}

\vspace{-15pt}
\end{figure}

\section{Guidance of Spacecraft Swarms \label{sec:Guidance-Spacecraft}}

Since it is technically feasible to develop and deploy swarms ($100$s--$1000$s)
of femtosatellites \cite{Ref:Hadaegh13}, in this section we solve
the guidance problem for such swarms of spacecraft in Earth orbit.

Assume that the spacecraft are located in the Local Vertical Local
Horizontal (LVLH) frame rotating around Earth. As shown in Fig. \ref{fig:Changing-bin-locations-HCW}(a),
for some initial conditions and no further control input, each spacecraft
is in a closed elliptical orbit in the LVLH frame, called passive
relative orbit (PRO) \cite{Ref:Morgan12}. If time-varying bins ($R_{k}[i],\thinspace\forall i\in\{1,\ldots,n_{\textrm{cell}}\}$)
are designed so that the spacecraft continues to coast along the PRO
in the LVLH frame, as shown in Fig. \ref{fig:Changing-bin-locations-HCW}(a),
then no control input is required to transition from $R_{k}[i]$ to
$R_{k+1}[i]$.

Motion constraints could arise from the spacecraft dynamics, as it
might be infeasible for the spacecraft in a certain bin to transition
to another bin within a single time step due to the large distance
between these bins. Some constraints could also arise from a limit
on the amount of fuel that can be consumed in a single time step or
limited control authority. A time-varying motion constraints matrix
$A_{k}^{j}$ is designed to handle such motion constraints due to
spacecraft dynamics. If a spacecraft is currently in $R_{k}[i]$ bin,
then it can only transition to the light blue cells in Fig. \ref{fig:Changing-bin-locations-HCW}(b)
during the $(k+1)^{\textrm{th}}$ time instant due to the motion constraints.
We have already shown that if each spacecraft executes the PSG--IMC
algorithm, then each spacecraft satisfies these motion constraints,
the swarm converges to the desired formation, and the Markov matrices
converge to the identity matrix.

\begin{center}
\begin{figure}
\begin{centering}
\begin{tabular}{cc}
\includegraphics[bb=310bp 70bp 700bp 430bp,clip,width=1.3in]{HCW_motion_constraint_v5.pdf} & \includegraphics[bb=315bp 0bp 831bp 510bp,clip,width=1.3in]{Guidance_20steps_only_bins_v8.pdf}\tabularnewline
(a) & (b)\tabularnewline
\end{tabular}
\par\end{centering}

\begin{centering}
\protect\caption{(a) The PRO through $R_{k}[i]$ bin and the time-varying bin locations
in the LVLH frame. (b) Location of the bins at different time steps
in the LVLH frame along with the respective swarm distributions. \label{fig:Changing-bin-locations-HCW}}

\par\end{centering}

\vspace{-15pt}
\end{figure}

\par\end{center}

\vspace{-25pt}

We now extend the example discussed in Section \ref{sub:Numerical-Example-1}
to a swarm of spacecraft in Earth orbit. If $\left(x[i],\thinspace y[i]\right)$
denotes the location of the bin $R[i]$ in the $30\times30$ grid,
then the time-varying location of the centroid of the bin on the PRO
in the LVLH frame is given by: \vspace{-10pt}

\begin{equation}
\boldsymbol{\kappa}_{k}[i]=\left(\begin{array}{c}
\frac{1}{2}(1+\frac{1}{15}x[i])\sin(\frac{\pi}{10}k+\frac{\pi}{300}y[i])\\
(1+\frac{1}{15}x[i])\cos(\frac{\pi}{10}k+\frac{\pi}{300}y[i])
\end{array}\right)\thinspace.
\end{equation}
Fig. \ref{fig:Changing-bin-locations-HCW}(b) shows the locations
of the time-varying bins at different time steps along with the respective
swarm distributions. Similar to the previous example, the swarm converges
to the desired formation and the spacecraft settle down after the
final formation is achieved.

\section{Conclusions}

This paper presents a new approach to shaping and reconfiguring a
large number of autonomous agents in a probabilistic framework. In
the proposed PSG--IMC algorithm, each agent independently determines
its own trajectory so that the overall swarm asymptotically converges
to the  desired formation, which is robust to external disturbances
or damages. Compared to prior work using homogeneous Markov chains
for all agents, the proposed algorithm using inhomogeneous Markov
chains helps the agents avoid unnecessary transitions when the swarm
converges to the desired formation. We present a novel technique for
constructing inhomogeneous Markov matrices in a distributed fashion,
thereby achieving and maintaining the desired swarm distribution while
satisfying motion constraints. Using a consensus algorithm along with
communication with neighboring agents, the agents estimate the current
swarm distribution and transition so that the Hellinger distance between
the estimated swarm distribution and the desired formation is minimized.
The application of PSG--IMC algorithm to guide a swarm of spacecraft
in Earth orbit is discussed. Results from multiple simulation runs
demonstrate the properties of self-repair capability, reliability
and improved convergence of the proposed PSG--IMC algorithm.