  The stochastic approximation (SA) algorithm is a widely used probabilistic
method for finding a zero or a fixed point of a vector-valued funtion, when
only noisy measurements of the function are available. In the literature to
date, one makes a distinction between ``synchronous'' updating, whereby every
component of the current guess is updated at each time, and ``asynchronous''
updating, whereby only one component is updated. In this paper, we study an
intermediate situation that we call ``batch asynchronous stochastic
approximation'' (BASA), in which, at each time instant, \textit{some but not
all} components of the current estimated solution are updated. BASA allows the
user to trade off memory requirements against time complexity. We develop a
general methodology for proving that such algorithms converge to the fixed
point of the map under study. These convergence proofs make use of weaker
hypotheses than existing results. Specifically, existing convergence proofs
require that the measurement noise is a zero-mean i.i.d\ sequence or a
martingale difference sequence. In the present paper, we permit biased
measurements, that is, measurement noises that have nonzero conditional mean.
Also, all convergence results to date assume that the stochastic step sizes
satisfy a probabilistic analog of the well-known Robbins-Monro conditions. We
replace this assumption by a purely deterministic condition on the
irreducibility of the underlying Markov processes.
  As specific applications to Reinforcement Learning, we analyze the temporal
difference algorithm $TD(\lambda)$ for value iteration, and the $Q$-learning
algorithm for finding the optimal action-value function. In both cases, we
establish the convergence of these algorithms, under milder conditions than in
the existing literature.
