  We propose the Bayesian bridge estimator for regularized regression and
classification. Two key mixture representations for the Bayesian bridge model
are developed: (1) a scale mixture of normals with respect to an alpha-stable
random variable; and (2) a mixture of Bartlett--Fejer kernels (or triangle
densities) with respect to a two-component mixture of gamma random variables.
Both lead to MCMC methods for posterior simulation, and these methods turn out
to have complementary domains of maximum efficiency. The first representation
is a well known result due to West (1987), and is the better choice for
collinear design matrices. The second representation is new, and is more
efficient for orthogonal problems, largely because it avoids the need to deal
with exponentially tilted stable random variables. It also provides insight
into the multimodality of the joint posterior distribution, a feature of the
bridge model that is notably absent under ridge or lasso-type priors. We prove
a theorem that extends this representation to a wider class of densities
representable as scale mixtures of betas, and provide an explicit inversion
formula for the mixing distribution. The connections with slice sampling and
scale mixtures of normals are explored. On the practical side, we find that the
Bayesian bridge model outperforms its classical cousin in estimation and
prediction across a variety of data sets, both simulated and real. We also show
that the MCMC for fitting the bridge model exhibits excellent mixing
properties, particularly for the global scale parameter. This makes for a
favorable contrast with analogous MCMC algorithms for other sparse Bayesian
models. All methods described in this paper are implemented in the R package
BayesBridge. An extensive set of simulation results are provided in two
supplemental files.
