  We consider a rate-splitting multiple access (RSMA)-based communication and
radar coexistence (CRC) system. The proposed system allows an RSMA-based
communication system to share spectrum with multiple radars. Furthermore, RSMA
enables flexible and powerful interference management by splitting messages
into common parts and private parts to partially decode interference and
partially treat interference as noise. The RSMA-based CRC system thus
significantly improves spectral efficiency, energy efficiency and quality of
service (QoS) of communication users (CUs). However, the RSMA-based CRC system
raises new challenges. Due to the spectrum sharing, the communication network
and the radars cause interference to each other, which reduces the
signal-to-interference-plus-noise ratio (SINR) of the radars as well as the
data rate of the CUs in the communication network. Therefore, a major problem
is to maximize the sum rate of the CUs while guaranteeing their QoS
requirements of data transmissions and the SINR requirements of multiple
radars. To achieve these objectives, we formulate a problem that optimizes i)
the common rate allocation to the CUs, transmit power of common messages and
transmit power of private messages of the CUs, and ii) transmit power of the
radars. The problem is non-convex with multiple decision parameters, which is
challenging to be solved. We propose two algorithms. The first sequential
quadratic programming (SQP) can quickly return a local optimal solution, and
has been known to be the state-of-the-art in nonlinear programming methods. The
second is an additive approximation scheme (AAS) which solves the problem
globally in a reasonable amount of time, based on the technique of applying
exhaustive enumeration to a modified instance. The simulation results show the
improvement of the AAS compared with the SQP in terms of sum rate.
