  In multi-agent reinforcement learning, each agent acts to maximize its
individual accumulated rewards. Nevertheless, individual accumulated rewards
could not fully reflect how others perceive them, resulting in selfish
behaviors that undermine global performance. The externality theory, defined as
``the activities of one economic actor affect the activities of another in ways
that are not reflected in market transactions,'' is applicable to analyze the
social dilemmas in MARL. One of its most profound non-market solutions,
``Pigovian Tax'', which internalizes externalities by taxing those who create
negative externalities and subsidizing those who create positive externalities,
could aid in developing a mechanism to resolve MARL's social dilemmas. The
purpose of this paper is to apply externality theory to analyze social dilemmas
in MARL. To internalize the externalities in MARL, the \textbf{L}earning
\textbf{O}ptimal \textbf{P}igovian \textbf{T}ax method (LOPT), is proposed,
where an additional agent is introduced to learn the tax/allowance allocation
policy so as to approximate the optimal ``Pigovian Tax'' which accurately
reflects the externalities for all agents. Furthermore, a reward shaping
mechanism based on the approximated optimal ``Pigovian Tax'' is applied to
reduce the social cost of each agent and tries to alleviate the social
dilemmas. Compared with existing state-of-the-art methods, the proposed LOPT
leads to higher collective social welfare in both the Escape Room and the
Cleanup environments, which shows the superiority of our method in solving
social dilemmas.
