  The field of safe multi-agent reinforcement learning, despite its potential
applications in various domains such as drone delivery and vehicle automation,
remains relatively unexplored. Training agents to learn optimal policies that
maximize rewards while considering specific constraints can be challenging,
particularly in scenarios where having a central controller to coordinate the
agents during the training process is not feasible. In this paper, we address
the problem of multi-agent policy optimization in a decentralized setting,
where agents communicate with their neighbors to maximize the sum of their
cumulative rewards while also satisfying each agent's safety constraints. We
consider both peak and average constraints. In this scenario, there is no
central controller coordinating the agents and both the rewards and constraints
are only known to each agent locally/privately. We formulate the problem as a
decentralized constrained multi-agent Markov Decision Problem and propose a
momentum-based decentralized policy gradient method, DePAint, to solve it. To
the best of our knowledge, this is the first privacy-preserving fully
decentralized multi-agent reinforcement learning algorithm that considers both
peak and average constraints. We also provide theoretical analysis and
empirical evaluation of our algorithm in various scenarios and compare its
performance to centralized algorithms that consider similar constraints.
