  Traffic congestion in metropolitan areas is a world-wide problem that can be
ameliorated by traffic lights that respond dynamically to real-time conditions.
Recent studies applying deep reinforcement learning (RL) to optimize single
traffic lights have shown significant improvement over conventional control.
However, optimization of global traffic condition over a large road network
fundamentally is a cooperative multi-agent control problem, for which
single-agent RL is not suitable due to environment non-stationarity and
infeasibility of optimizing over an exponential joint-action space. Motivated
by these challenges, we propose QCOMBO, a simple yet effective multi-agent
reinforcement learning (MARL) algorithm that combines the advantages of
independent and centralized learning. We ensure scalability by selecting
actions from individually optimized utility functions, which are shaped to
maximize global performance via a novel consistency regularization loss between
individual utility and a global action-value function. Experiments on diverse
road topologies and traffic flow conditions in the SUMO traffic simulator show
competitive performance of QCOMBO versus recent state-of-the-art MARL
algorithms. We further show that policies trained on small sub-networks can
effectively generalize to larger networks under different traffic flow
conditions, providing empirical evidence for the suitability of MARL for
intelligent traffic control.
