  We are motivated by the need for impromptu (or as-you-go) deployment of
multihop wireless networks, by human agents or robots; the agent moves along a
line, makes wireless link quality measurements at regular intervals, and makes
on-line placement decisions using these measurements. As a first step, we have
formulated such deployment along a line as a sequential decision problem. In
our earlier work, we proposed two possible deployment approaches: (i) the pure
as-you-go approach where the deployment agent can only move forward, and (ii)
the explore-forward approach where the deployment agent explores a few
successive steps and then selects the best relay placement location. The latter
was shown to provide better performance but at the expense of more measurements
and deployment time, which makes explore-forward impractical for quick
deployment by an energy constrained agent such as a UAV. Further, the
deployment algorithm should not require prior knowledge of the parameters of
the wireless propagation model. In [1] we, therefore, developed learning
algorithms for the explore-forward approach.
  The current paper provides deploy-and-learn algorithms for the pure as-you-go
approach. We formulate the sequential relay deployment problem as an average
cost Markov decision process (MDP), which trades off among power consumption,
link outage probabilities, and the number of deployed relay nodes. First we
show structural results for the optimal policy. Next, by exploiting the special
structure of the optimality equation and by using the theory of asynchronous
stochastic approximation, we develop two learning algorithms that
asymptotically converge to the set of optimal policies as deployment
progresses. Numerical results show reasonably fast speed of convergence, and
hence the model-free algorithms can be useful for practical, fast deployment of
emergency wireless networks.
