  Multi-Agent Reinforcement Learning (MARL) has emerged as a foundational
approach for addressing diverse, intelligent control tasks, notably in
autonomous driving within the Internet of Vehicles (IoV) domain. However, the
widely assumed existence of a central node for centralized, federated
learning-assisted MARL might be impractical in highly dynamic environments.
This can lead to excessive communication overhead, potentially overwhelming the
IoV system. To address these challenges, we design a novel
communication-efficient and policy collaboration algorithm for MARL under the
frameworks of Soft Actor-Critic (SAC) and Decentralized Federated Learning
(DFL), named RSM-MASAC, within a fully distributed architecture. In particular,
RSM-MASAC enhances multi-agent collaboration and prioritizes higher
communication efficiency in dynamic IoV system by incorporating the concept of
segmented aggregation in DFL and augmenting multiple model replicas from
received neighboring policy segments, which are subsequently employed as
reconstructed referential policies for mixing. Distinctively diverging from
traditional RL approaches, with derived new bounds under Maximum Entropy
Reinforcement Learning (MERL), RSM-MASAC adopts a theory-guided mixture metric
to regulate the selection of contributive referential policies to guarantee the
soft policy improvement during communication phase. Finally, the extensive
simulations in mixed-autonomy traffic control scenarios verify the
effectiveness and superiority of our algorithm.
