  While achieving tremendous success in various fields, existing multi-agent
reinforcement learning (MARL) with a black-box neural network architecture
makes decisions in an opaque manner that hinders humans from understanding the
learned knowledge and how input observations influence decisions. Instead,
existing interpretable approaches, such as traditional linear models and
decision trees, usually suffer from weak expressivity and low accuracy. To
address this apparent dichotomy between performance and interpretability, our
solution, MIXing Recurrent soft decision Trees (MIXRTs), is a novel
interpretable architecture that can represent explicit decision processes via
the root-to-leaf path and reflect each agent's contribution to the team.
Specifically, we construct a novel soft decision tree to address partial
observability by leveraging the advances in recurrent neural networks, and
demonstrate which features influence the decision-making process through the
tree-based model. Then, based on the value decomposition framework, we linearly
assign credit to each agent by explicitly mixing individual action values to
estimate the joint action value using only local observations, providing new
insights into how agents cooperate to accomplish the task. Theoretical analysis
shows that MIXRTs guarantees the structural constraint on additivity and
monotonicity in the factorization of joint action values. Evaluations on the
challenging Spread and StarCraft II tasks show that MIXRTs achieves competitive
performance compared to widely investigated methods and delivers more
straightforward explanations of the decision processes. We explore a promising
path toward developing learning algorithms with both high performance and
interpretability, potentially shedding light on new interpretable paradigms for
MARL.
