  We show that convex-concave Lipschitz stochastic saddle point problems (also
known as stochastic minimax optimization) can be solved under the constraint of
$(\epsilon,\delta)$-differential privacy with \emph{strong (primal-dual) gap}
rate of $\tilde O\big(\frac{1}{\sqrt{n}} + \frac{\sqrt{d}}{n\epsilon}\big)$,
where $n$ is the dataset size and $d$ is the dimension of the problem. This
rate is nearly optimal, based on existing lower bounds in differentially
private stochastic optimization. Specifically, we prove a tight upper bound on
the strong gap via novel implementation and analysis of the recursive
regularization technique repurposed for saddle point problems. We show that
this rate can be attained with
$O\big(\min\big\{\frac{n^2\epsilon^{1.5}}{\sqrt{d}}, n^{3/2}\big\}\big)$
gradient complexity, and $\tilde{O}(n)$ gradient complexity if the loss
function is smooth. As a byproduct of our method, we develop a general
algorithm that, given a black-box access to a subroutine satisfying a certain
$\alpha$ primal-dual accuracy guarantee with respect to the empirical
objective, gives a solution to the stochastic saddle point problem with a
strong gap of $\tilde{O}(\alpha+\frac{1}{\sqrt{n}})$. We show that this
$\alpha$-accuracy condition is satisfied by standard algorithms for the
empirical saddle point problem such as the proximal point method and the
stochastic gradient descent ascent algorithm. Further, we show that even for
simple problems it is possible for an algorithm to have zero weak gap and
suffer from $\Omega(1)$ strong gap. We also show that there exists a
fundamental tradeoff between stability and accuracy. Specifically, we show that
any $\Delta$-stable algorithm has empirical gap $\Omega\big(\frac{1}{\Delta
n}\big)$, and that this bound is tight. This result also holds also more
specifically for empirical risk minimization problems and may be of independent
interest.
