  Multi-agent reinforcement learning methods such as VDN, QMIX, and QTRAN that
adopt centralized training with decentralized execution (CTDE) framework have
shown promising results in cooperation and competition. However, in some
multi-agent scenarios, the number of agents and the size of action set actually
vary over time. We call these unshaped scenarios, and the methods mentioned
above fail in performing satisfyingly. In this paper, we propose a new method
called Unshaped Networks for Multi-Agent Systems (UNMAS) that adapts to the
number and size changes in multi-agent systems. We propose the self-weighting
mixing network to factorize the joint action-value. Its adaption to the change
in agent number is attributed to the nonlinear mapping from each-agent Q value
to the joint action-value with individual weights. Besides, in order to address
the change in action set, each agent constructs an individual action-value
network that is composed of two streams to evaluate the constant
environment-oriented subset and the varying unit-oriented subset. We evaluate
UNMAS on various StarCraft II micro-management scenarios and compare the
results with several state-of-the-art MARL algorithms. The superiority of UNMAS
is demonstrated by its highest winning rates especially on the most difficult
scenario 3s5z_vs_3s6z. The agents learn to perform effectively cooperative
behaviors while other MARL algorithms fail in. Animated demonstrations and
source code are provided in https://sites.google.com/view/unmas.
