  Exact approximations of Markov chain Monte Carlo (MCMC) algorithms are a
general emerging class of sampling algorithms. One of the main ideas behind
exact approximations consists of replacing intractable quantities required to
run standard MCMC algorithms, such as the target probability density in a
Metropolis-Hastings algorithm, with estimators. Perhaps surprisingly, such
approximations lead to powerful algorithms which are exact in the sense that
they are guaranteed to have correct limiting distributions. In this paper we
discover a general framework which allows one to compare, or order, performance
measures of two implementations of such algorithms. In particular, we establish
an order with respect to the mean acceptance probability, the first
autocorrelation coefficient, the asymptotic variance and the right spectral
gap. The key notion to guarantee the ordering is that of the convex order
between estimators used to implement the algorithms. We believe that our convex
order condition is close to optimal, and this is supported by a counter-example
which shows that a weaker variance order is not sufficient. The convex order
plays a central role by allowing us to construct a martingale coupling which
enables the comparison of performance measures of Markov chain with differing
invariant distributions, contrary to existing results. We detail applications
of our result by identifying extremal distributions within given classes of
approximations, by showing that averaging replicas improves performance in a
monotonic fashion and that stratification is guaranteed to improve performance
for the standard implementation of the Approximate Bayesian Computation (ABC)
MCMC method.
