  High-density, unsignalized intersection has always been a bottleneck of
efficiency and safety. The emergence of Connected Autonomous Vehicles (CAVs)
results in a mixed traffic condition, further increasing the complexity of the
transportation system. Against this background, this paper aims to study the
intricate and heterogeneous interaction of vehicles and conflict resolution at
the high-density, mixed, unsignalized intersection. Theoretical insights about
the interaction between CAVs and Human-driven Vehicles (HVs) and the
cooperation of CAVs are synthesized, based on which a novel cooperative
decision-making framework in heterogeneous mixed traffic is proposed.
Normalized Cooperative game is concatenated with Level-k game (NCL game) to
generate a system optimal solution. Then Lattice planner generates the optimal
and collision-free trajectories for CAVs. To reproduce HVs in mixed traffic,
interactions from naturalistic human driving data are extracted as prior
knowledge. Non-cooperative game and Inverse Reinforcement Learning (IRL) are
integrated to mimic the decision making of heterogeneous HVs. Finally, three
cases are conducted to verify the performance of the proposed algorithm,
including the comparative analysis with different methods, the case study under
different Rates of Penetration (ROP) and the interaction analysis with
heterogeneous HVs. It is found that the proposed cooperative decision-making
framework is beneficial to the driving conflict resolution and the traffic
efficiency improvement of the mixed unsignalized intersection. Besides, due to
the consideration of driving heterogeneity, better human-machine interaction
and cooperation can be realized in this paper.
