  Conventional multi-user multiple-input multiple-output (MU-MIMO) mainly
focused on Gaussian signaling, independent and identically distributed (IID)
channels, and a limited number of users. It will be laborious to cope with the
heterogeneous requirements in next-generation wireless communications, such as
various transmission data, complicated communication scenarios, and massive
user access. Therefore, this paper studies a generalized MU-MIMO (GMU-MIMO)
system with more practical constraints, i.e., non-Gaussian signaling, non-IID
channel, and massive users and antennas. These generalized assumptions bring
new challenges in theory and practice. For example, there is no accurate
capacity analysis for GMU-MIMO. In addition, it is unclear how to achieve the
capacity optimal performance with practical complexity.
  To address these challenges, a unified framework is proposed to derive the
GMU-MIMO capacity and design a capacity optimal transceiver, which jointly
considers encoding, modulation, detection, and decoding. Group asymmetry is
developed to make a tradeoff between user rate allocation and implementation
complexity. Specifically, the capacity region of group asymmetric GMU-MIMO is
characterized by using the celebrated mutual information and minimum
mean-square error (MMSE) lemma and the MMSE optimality of orthogonal
approximate message passing (OAMP)/vector AMP (VAMP). Furthermore, a
theoretically optimal multi-user OAMP/VAMP receiver and practical multi-user
low-density parity-check (MU-LDPC) codes are proposed to achieve the capacity
region of group asymmetric GMU-MIMO. Numerical results verify that the gaps
between theoretical detection thresholds of the proposed framework with
optimized MU-LDPC codes and QPSK modulation and the sum capacity of GMU-MIMO
are about 0.2 dB. Moreover, their finite-length performances are about 1~2 dB
away from the associated sum capacity.
