  In this paper, an unmanned aerial vehicle (UAV)-assisted wireless network is
considered in which a battery-constrained UAV is assumed to move towards
energy-constrained ground nodes to receive status updates about their observed
processes. The UAV's flight trajectory and scheduling of status updates are
jointly optimized with the objective of minimizing the normalized weighted sum
of Age of Information (NWAoI) values for different physical processes at the
UAV. The problem is first formulated as a mixed-integer program. Then, for a
given scheduling policy, a convex optimization-based solution is proposed to
derive the UAV's optimal flight trajectory and time instants on updates.
However, finding the optimal scheduling policy is challenging due to the
combinatorial nature of the formulated problem. Therefore, to complement the
proposed convex optimization-based solution, a finite-horizon Markov decision
process (MDP) is used to find the optimal scheduling policy. Since the state
space of the MDP is extremely large, a novel neural combinatorial-based deep
reinforcement learning (NCRL) algorithm using deep Q-network (DQN) is proposed
to obtain the optimal policy. However, for large-scale scenarios with numerous
nodes, the DQN architecture cannot efficiently learn the optimal scheduling
policy anymore. Motivated by this, a long short-term memory (LSTM)-based
autoencoder is proposed to map the state space to a fixed-size vector
representation in such large-scale scenarios. A lower bound on the minimum
NWAoI is analytically derived which provides system design guidelines on the
appropriate choice of importance weights for different nodes. The numerical
results also demonstrate that the proposed NCRL approach can significantly
improve the achievable NWAoI per process compared to the baseline policies,
such as weight-based and discretized state DQN policies.
