  Intelligent reflecting surface (IRS) is envisioned to be widely applied in
future wireless networks. In this paper, we investigate a multi-user
communication system assisted by cooperative IRS devices with the capability of
energy harvesting. Aiming to maximize the long-term average achievable system
rate, an optimization problem is formulated by jointly designing the transmit
beamforming at the base station (BS) and discrete phase shift beamforming at
the IRSs, with the constraints on transmit power, user data rate requirement
and IRS energy buffer size. Considering time-varying channels and stochastic
arrivals of energy harvested by the IRSs, we first formulate the problem as a
Markov decision process (MDP) and then develop a novel multi-agent Q-mix (MAQ)
framework with two layers to decouple the optimization parameters. The higher
layer is for optimizing phase shift resolutions, and the lower one is for phase
shift beamforming and power allocation. Since the phase shift optimization is
an integer programming problem with a large-scale action space, we improve MAQ
by incorporating the Wolpertinger method, namely, MAQ-WP algorithm to achieve a
sub-optimality with reduced dimensions of action space. In addition, as MAQ-WP
is still of high complexity to achieve good performance, we propose a policy
gradient-based MAQ algorithm, namely, MAQ-PG, by mapping the discrete phase
shift actions into a continuous space at the cost of a slight performance loss.
Simulation results demonstrate that the proposed MAQ-WP and MAQ-PG algorithms
can converge faster and achieve data rate improvements of 10.7% and 8.8% over
the conventional multi-agent DDPG, respectively.
