  Connected and automated vehicles (CAVs) have recently gained prominence in
traffic research, thanks to the advancements in communication technology and
autonomous driving. A variety of longitudinal control strategies for CAVs have
been developed to enhance traffic efficiency, stability, and safety in
mixed-autonomy scenarios. Deep reinforcement learning (DRL) is one promising
strategy for mixed-autonomy platoon control since it can tackle complex
scenarios in real-time. However, there are three research gaps for DRL-based
mixed-autonomy platoon control. First, incorporating safety considerations into
DRL typically relies on designing collision avoidance-based reward functions,
which lack collision-free guarantees. Second, current DRL-based-control
approaches for mixed traffic only consider the safety of CAVs, with little
attention paid to the surrounding HDVs. To address the research gaps, we
introduce a differentiable safety layer that converts DRL actions into safe
actions with collision-free guarantees. This process relies on solving a
differentiable quadratic programming problem that incorporates control barrier
function-based (CBF) safety constraints for both CAV and its following HDVs to
achieve system-level safety. Moreover, constructing CBF constraints needs
system dynamics for the following HDVs, and thus we employ an online system
identification module to estimate the car-following dynamics of the surrounding
HDVs. The proposed safe reinforcement learning approach explicitly integrates
system-level safety constraints into the training process and enables our
method to adapt to varying safety-critical scenarios. Simulation results
demonstrate that our proposed method effectively ensures CAV safety and
improves HDV safety in mixed platoon environments while simultaneously
enhancing traffic capacity and string stability.
