  We consider two-player zero-sum stochastic games and propose a two-timescale
$Q$-learning algorithm with function approximation that is payoff-based,
convergent, rational, and symmetric between the two players. In two-timescale
$Q$-learning, the fast-timescale iterates are updated in spirit to the
stochastic gradient descent and the slow-timescale iterates (which we use to
compute the policies) are updated by taking a convex combination between its
previous iterate and the latest fast-timescale iterate. Introducing the slow
timescale as well as its update equation marks as our main algorithmic novelty.
In the special case of linear function approximation, we establish, to the best
of our knowledge, the first last-iterate finite-sample bound for payoff-based
independent learning dynamics of these types. The result implies a polynomial
sample complexity to find a Nash equilibrium in such stochastic games.
  To establish the results, we model our proposed algorithm as a two-timescale
stochastic approximation and derive the finite-sample bound through a
Lyapunov-based approach. The key novelty lies in constructing a valid Lyapunov
function to capture the evolution of the slow-timescale iterates. Specifically,
through a change of variable, we show that the update equation of the
slow-timescale iterates resembles the classical smoothed best-response
dynamics, where the regularized Nash gap serves as a valid Lyapunov function.
This insight enables us to construct a valid Lyapunov function via a
generalized variant of the Moreau envelope of the regularized Nash gap. The
construction of our Lyapunov function might be of broad independent interest in
studying the behavior of stochastic approximation algorithms.
