  Multi-agent value-based approaches recently make great progress, especially
value decomposition methods. However, there are still a lot of limitations in
value function factorization. In VDN, the joint action-value function is the
sum of per-agent action-value function while the joint action-value function of
QMIX is the monotonic mixing of per-agent action-value function. To some
extent, QTRAN reduces the limitation of joint action-value functions that can
be represented, but it has unsatisfied performance in complex tasks. In this
paper, in order to extend the class of joint value functions that can be
represented, we propose a novel actor-critic method called NQMIX. NQMIX
introduces an off-policy policy gradient on QMIX and modify its network
architecture, which can remove the monotonicity constraint of QMIX and
implement a non-monotonic value function factorization for the joint
action-value function. In addition, NQMIX takes the state-value as the learning
target, which overcomes the problem in QMIX that the learning target is
overestimated. Furthermore, NQMIX can be extended to continuous action space
settings by introducing deterministic policy gradient on itself. Finally, we
evaluate our actor-critic methods on SMAC domain, and show that it has a
stronger performance than COMA and QMIX on complex maps with heterogeneous
agent types. In addition, our ablation results show that our modification of
mixer is effective.
