  Parallel Markov Chain Monte Carlo (pMCMC) algorithms generate clouds of
proposals at each step to efficiently resolve a target probability
distribution. We build a rigorous foundational framework for pMCMC algorithms
that situates these methods within a unified `extended phase space'
measure-theoretic formalism. Drawing on our recent work that provides a
comprehensive theory for reversible single proposal methods, we herein derive
general criteria for multiproposal acceptance mechanisms which yield unbiased
chains on general state spaces. Our formulation encompasses a variety of
methodologies, including proposal cloud resampling and Hamiltonian methods,
while providing a basis for the derivation of novel algorithms. In particular,
we obtain a top-down picture for a class of methods arising from `conditionally
independent' proposal structures. As an immediate application, we identify
several new algorithms including a multiproposal version of the popular
preconditioned Crank-Nicolson (pCN) sampler suitable for high- and
infinite-dimensional target measures which are absolutely continuous with
respect to a Gaussian base measure. To supplement our theoretical results, we
carry out a selection of numerical case studies that evaluate the efficacy of
these novel algorithms. First, noting that the true potential of pMCMC
algorithms arises from their natural parallelizability, we provide a limited
parallelization study using TensorFlow and a graphics processing unit to scale
pMCMC algorithms that leverage as many as 100k proposals at each step. Second,
we use our multiproposal pCN algorithm (mpCN) to resolve a selection of
problems in Bayesian statistical inversion for partial differential equations
motivated by fluid measurement. These examples provide preliminary evidence of
the efficacy of mpCN for high-dimensional target distributions featuring
complex geometries and multimodal structures.
