  Low-complexity beamformer design with practical constraints is an attractive
research area for hybrid analog/digital systems in mm-wave massive
multiple-input multiple-output (MIMO). This paper investigates
interference-aware pre-beamformer (analog beamformer) design for joint spatial
division and multiplexing (JSDM) which is a user-grouping based two-stage
beamforming method. Single-carrier frequency domain equalization (SC-FDE) is
employed in uplink frequency-selective channels. First, unconstrained slowly
changing statistical analog beamformer of each group, namely, generalized
eigenbeamformer (GEB) which has strong interference suppression capability is
designed where the mutual information in reduced dimension is maximized. Then,
constant-modulus constrained approximations of unconstrained beamformer are
obtained by utilizing alternating minimization algorithms for fully connected
arrays and fixed subarrays. In addition, a dynamic subarray algorithm is
proposed where the connections between radio frequency (RF) chains and antennas
are changed with changing channel statistics. Convergence of the proposed
alternating minimization-based algorithms are provided along with their
complexity analysis. It is observed that additional complexity of proposed
algorithms is insignificant for the overall system design. Although most of the
interference is suppressed with the help of proposed constrained beamformers,
there may be some residual interference after the analog beamforming stage.
Therefore, linear minimum mean square error (LMMSE) type digital beamformers,
which take the residual interference in reduced dimension into account, are
proposed instead of zero-forcing (ZF) type. Simulation results verify the
superiority of the proposed interference-aware constrained design over existing
approaches in terms of beampattern, spectral efficiency, outage capacity and
channel estimation accuracy.
