  A novel integrated sensing and communication (ISAC) system is proposed, where
a dual-functional base station is utilized to transmit the superimposed
non-orthogonal multiple access (NOMA) communication signal for serving
communication users and sensing targets simultaneously. Furthermore, a new
reconfigurable intelligent surface (RIS)-aided-sensing structure is also
proposed to address the significant path loss or blockage of LoS links for the
sensing task. Based on this setup, the beampattern gain at the RIS for the
radar target is derived and adopted as a sensing metric. The objective of this
paper is to maximize the minimum beampattern gain by jointly optimizing active
beamforming, power allocation coefficients and passive beamforming. To tackle
the non-convexity of the formulated optimization problem, the beampattern gain
and constraints are first transformed into more tractable forms. Then, an
iterative block coordinate descent (IBCD) algorithm is proposed by employing
successive convex approximation (SCA), Schur complement, semidefinite
relaxation (SDR) and sequential rank-one constraint relaxation (SRCR) methods.
To reduce the complexity of the proposed IBCD algorithm, a low-complexity
iterative alternating optimization (IAO) algorithm is proposed. Particularly,
the active beamforming is optimized by solving a semidefinite programming (SDP)
problem and the closed-form solutions of the power allocation coefficients are
derived. Numerical results show that: i) the proposed RIS-NOMA-ISAC system
always outperforms the RIS-ISAC system without NOMA in beampattern gain and
illumination power; ii) the low-complexity IAO algorithm achieves a comparable
performance to that achieved by the IBCD algorithm. iii) high beampattern gain
can be achieved by the proposed joint optimization algorithms in underloaded
and overloaded communication scenarios.
