  Research in model-based reinforcement learning has made significant progress
in recent years. Compared to single-agent settings, the exponential dimension
growth of the joint state-action space in multi-agent systems dramatically
increases the complexity of the environment dynamics, which makes it infeasible
to learn an accurate global model and thus necessitates the use of agent-wise
local models. However, during multi-step model rollouts, the prediction of one
local model can affect the predictions of other local models in the next step.
As a result, local prediction errors can be propagated to other localities and
eventually give rise to considerably large global errors. Furthermore, since
the models are generally used to predict for multiple steps, simply minimizing
one-step prediction errors regardless of their long-term effect on other models
may further aggravate the propagation of local errors. To this end, we propose
Models as AGents (MAG), a multi-agent model optimization framework that
reversely treats the local models as multi-step decision making agents and the
current policies as the dynamics during the model rollout process. In this way,
the local models are able to consider the multi-step mutual affect between each
other before making predictions. Theoretically, we show that the objective of
MAG is approximately equivalent to maximizing a lower bound of the true
environment return. Experiments on the challenging StarCraft II benchmark
demonstrate the effectiveness of MAG.
