  In general-sum games, the interaction of self-interested learning agents
commonly leads to socially worse outcomes, such as defect-defect in the
iterated stag hunt (ISH). Previous works address this challenge by sharing
rewards or shaping their opponents' learning process, which require too strong
assumptions. In this paper, we demonstrate that agents trained to optimize
expected returns are more likely to choose a safe action that leads to
guaranteed but lower rewards. However, there typically exists a risky action
that leads to higher rewards in the long run only if agents cooperate, e.g.,
cooperate-cooperate in ISH. To overcome this, we propose using action value
distribution to characterize the decision's risk and corresponding potential
payoffs. Specifically, we present Adaptable Risk-Sensitive Policy (ARSP). ARSP
learns the distributions over agent's return and estimates a dynamic
risk-seeking bonus to discover risky coordination strategies. Furthermore, to
avoid overfitting training opponents, ARSP learns an auxiliary opponent
modeling task to infer opponents' types and dynamically alter corresponding
strategies during execution. Empirically, agents trained via ARSP can achieve
stable coordination during training without accessing opponent's rewards or
learning process, and can adapt to non-cooperative opponents during execution.
To the best of our knowledge, it is the first method to learn coordination
strategies between agents both in iterated prisoner's dilemma (IPD) and
iterated stag hunt (ISH) without shaping opponents or rewards, and can adapt to
opponents with distinct strategies during execution. Furthermore, we show that
ARSP can be scaled to high-dimensional settings.
