  Aiming at providing wireless communication systems with
environment-perceptive capacity, emerging integrated sensing and communication
(ISAC) technologies face multiple difficulties, especially in balancing the
performance trade-off between the communication and radar functions. In this
paper, we introduce a reconfigurable intelligent surface (RIS) to assist both
data transmission and target detection in a dual-functional ISAC system. To
formulate a general optimization framework, diverse communication performance
metrics have been taken into account including famous capacity maximization and
mean-squared error (MSE) minimization. Whereas the target detection process is
modeled as a general likelihood ratio test (GLRT) due to the practical
limitations, and the monotonicity of the corresponding detection probability is
proved. For the single-user and single-target (SUST) scenario, the minimum
transmit power of the ISAC transceiver has been revealed. By exploiting the
optimal conditions of the BS design, we validate that the BS is able to realize
the maximum power allocation scheme and derive the optimal BS precoder in a
semi-closed form. Moreover, an alternating direction method of multipliers
(ADMM) based RIS design is proposed to address the optimization of unit-modulus
RIS phase shifts. For the sake of further enhancing computational efficiency,
we also develop a low-complexity RIS design based on Riemannian gradient
descent. Furthermore, the ISAC transceiver design for the multiple-users and
multiple-targets (MUMT) scenario is also investigated, where a zero-forcing
(ZF) radar receiver is adopted to cancel the interferences. Then optimal BS
precoder is derived under the maximum power allocation scheme, and the RIS
phase shifts can be optimized by extending the proposed ADMM-based RIS design.
Numerical simulation results verify the performance of our proposed transceiver
designs.
