  We study the problem of computing an approximate Nash equilibrium of
continuous-action game without access to gradients. Such game access is common
in reinforcement learning settings, where the environment is typically treated
as a black box. To tackle this problem, we apply zeroth-order optimization
techniques that combine smoothed gradient estimators with equilibrium-finding
dynamics. We model players' strategies using artificial neural networks. In
particular, we use randomized policy networks to model mixed strategies. These
take noise in addition to an observation as input and can flexibly represent
arbitrary observation-dependent, continuous-action distributions. Being able to
model such mixed strategies is crucial for tackling continuous-action games
that lack pure-strategy equilibria. We evaluate the performance of our method
using an approximation of the Nash convergence metric from game theory, which
measures how much players can benefit from unilaterally changing their
strategy. We apply our method to continuous Colonel Blotto games, single-item
and multi-item auctions, and a visibility game. The experiments show that our
method can quickly find high-quality approximate equilibria. Furthermore, they
show that the dimensionality of the input noise is crucial for performance. To
our knowledge, this paper is the first to solve general continuous-action games
with unrestricted mixed strategies and without any gradient information.
