  Future Internet involves several emerging technologies such as 5G and beyond
5G networks, vehicular networks, unmanned aerial vehicle (UAV) networks, and
Internet of Things (IoTs). Moreover, future Internet becomes heterogeneous and
decentralized with a large number of involved network entities. Each entity may
need to make its local decision to improve the network performance under
dynamic and uncertain network environments. Standard learning algorithms such
as single-agent Reinforcement Learning (RL) or Deep Reinforcement Learning
(DRL) have been recently used to enable each network entity as an agent to
learn an optimal decision-making policy adaptively through interacting with the
unknown environments. However, such an algorithm fails to model the
cooperations or competitions among network entities, and simply treats other
entities as a part of the environment that may result in the non-stationarity
issue. Multi-agent Reinforcement Learning (MARL) allows each network entity to
learn its optimal policy by observing not only the environments, but also other
entities' policies. As a result, MARL can significantly improve the learning
efficiency of the network entities, and it has been recently used to solve
various issues in the emerging networks. In this paper, we thus review the
applications of MARL in the emerging networks. In particular, we provide a
tutorial of MARL and a comprehensive survey of applications of MARL in next
generation Internet. In particular, we first introduce single-agent RL and
MARL. Then, we review a number of applications of MARL to solve emerging issues
in future Internet. The issues consist of network access, transmit power
control, computation offloading, content caching, packet routing, trajectory
design for UAV-aided networks, and network security issues.
