  Intelligent reflecting surface (IRS) is a promising solution to enhance the
wireless communication capacity both cost-effectively and energy-efficiently,
by properly altering the signal propagation via tuning a large number of
passive reflecting units. In this paper, we aim to characterize the fundamental
capacity limit of IRS-aided point-to-point multiple-input multiple-output
(MIMO) communication systems with multi-antenna transmitter and receiver in
general, by jointly optimizing the IRS reflection coefficients and the MIMO
transmit covariance matrix. First, we consider narrowband transmission under
frequency-flat fading channels, and develop an efficient alternating
optimization algorithm to find a locally optimal solution by iteratively
optimizing the transmit covariance matrix or one of the reflection coefficients
with the others being fixed. Next, we consider capacity maximization for
broadband transmission in a general MIMO orthogonal frequency division
multiplexing (OFDM) system under frequency-selective fading channels, where
transmit covariance matrices can be optimized for different subcarriers while
only one common set of IRS reflection coefficients can be designed to cater to
all subcarriers. To tackle this more challenging problem, we propose a new
alternating optimization algorithm based on convex relaxation to find a
high-quality suboptimal solution. Numerical results show that our proposed
algorithms achieve substantially increased capacity compared to traditional
MIMO channels without the IRS, and also outperform various benchmark schemes.
In particular, it is shown that with the proposed algorithms, various key
parameters of the IRS-aided MIMO channel such as channel total power, rank, and
condition number can be significantly improved for capacity enhancement.
