  Developing efficient MCMC algorithms is indispensable in Bayesian inference.
In parallel tempering, multiple interacting MCMC chains run to more efficiently
explore the state space and improve performance. The multiple chains advance
independently through local moves, and the performance enhancement steps are
exchange moves, where the chains pause to exchange their current sample amongst
each other. To accelerate the independent local moves, they may be performed
simultaneously on multiple processors. Another problem is then encountered:
depending on the MCMC implementation and inference problem, local moves can
take a varying and random amount of time to complete. There may also be
infrastructure-induced variations, such as competing jobs on the same
processors, which arises in cloud computing. Before exchanges can occur, all
chains must complete the local moves they are engaged in to avoid introducing a
potentially substantial bias (Proposition 2.1). To solve this issue of randomly
varying local move completion times in multi-processor parallel tempering, we
adopt the Anytime Monte Carlo framework of Murray et al. (2016): we impose
real-time deadlines on the parallel local moves and perform exchanges at these
deadlines without any processor idling. We show our methodology for exchanges
at real-time deadlines does not introduce a bias and leads to significant
performance enhancements over the na\"ive approach of idling until every
processor's local moves complete. The methodology is then applied in an ABC
setting, where an Anytime ABC parallel tempering algorithm is derived for the
difficult task of estimating the parameters of a Lotka-Volterra predator-prey
model, and similar efficiency enhancements are observed.
