  We study momentum-based first-order optimization algorithms in which the
iterations utilize information from the two previous steps and are subject to
an additive white noise. This class of algorithms includes Polyak's heavy-ball
and Nesterov's accelerated methods as special cases and noise accounts for
uncertainty in either gradient evaluation or iteration updates. For strongly
convex quadratic problems, we use the steady-state variance of the error in the
optimization variable to quantify noise amplification and identify fundamental
stochastic performance tradeoffs. Our approach utilizes the Jury stability
criterion to provide a novel geometric characterization of conditions for
linear convergence, and it clarifies the relation between the noise
amplification and convergence rate as well as their dependence on the condition
number and the constant algorithmic parameters. This geometric insight leads to
simple alternative proofs of standard convergence results and allows us to
establish analytical lower bounds on the product between the settling time and
noise amplification that scale quadratically with the condition number. Our
analysis also identifies a key difference between the gradient and iterate
noise models: while the amplification of gradient noise can be made arbitrarily
small by sufficiently decelerating the algorithm, the best achievable variance
amplification for the iterate noise model increases linearly with the settling
time in decelerating regime. Furthermore, we introduce two parameterized
families of algorithms that strike a balance between noise amplification and
settling time while preserving order-wise Pareto optimality for both noise
models. Finally, by analyzing a class of accelerated gradient flow dynamics,
whose suitable discretization yields the two-step momentum algorithm, we
establish that stochastic performance tradeoffs also extend to continuous time.
