  We consider the constrained sampling problem where the goal is to sample from
a distribution $\pi(x)\propto e^{-f(x)}$ and $x$ is constrained on a convex
body $\mathcal{C}\subset \mathbb{R}^d$. Motivated by penalty methods from
optimization, we propose penalized Langevin Dynamics (PLD) and penalized
Hamiltonian Monte Carlo (PHMC) that convert the constrained sampling problem
into an unconstrained one by introducing a penalty function for constraint
violations. When $f$ is smooth and the gradient is available, we show
$\tilde{\mathcal{O}}(d/\varepsilon^{10})$ iteration complexity for PLD to
sample the target up to an $\varepsilon$-error where the error is measured in
terms of the total variation distance and $\tilde{\mathcal{O}}(\cdot)$ hides
some logarithmic factors. For PHMC, we improve this result to
$\tilde{\mathcal{O}}(\sqrt{d}/\varepsilon^{7})$ when the Hessian of $f$ is
Lipschitz and the boundary of $\mathcal{C}$ is sufficiently smooth. To our
knowledge, these are the first convergence rate results for Hamiltonian Monte
Carlo methods in the constrained sampling setting that can handle non-convex
$f$ and can provide guarantees with the best dimension dependency among
existing methods with deterministic gradients. We then consider the setting
where unbiased stochastic gradients are available. We propose PSGLD and PSGHMC
that can handle stochastic gradients without Metropolis-Hasting correction
steps. When $f$ is strongly convex and smooth, we obtain an iteration
complexity of $\tilde{\mathcal{O}}(d/\varepsilon^{18})$ and
$\tilde{\mathcal{O}}(d\sqrt{d}/\varepsilon^{39})$ respectively in the
2-Wasserstein distance. For the more general case, when $f$ is smooth and
non-convex, we also provide finite-time performance bounds and iteration
complexity results. Finally, we test our algorithms on Bayesian LASSO
regression and Bayesian constrained deep learning problems.
