text: In this paper, we study the problem of dynamic channel allocation for URLLC traffic in a multiuser multi-channel wireless network where urgent packets have to be successfully received in a timely manner. We formulate the problem as a finite-horizon Markov Decision Process with a stochastic constraint related to the QoS requirement, defined as the packet loss rate for each user. We propose a novel weighted formulation that takes into account both the total expected reward (number of successfully decoded packets) and the risk which we define as the QoS requirement violation. First, we use the value iteration algorithm to find the optimal policy, which assumes a perfect knowledge of the controller of all the parameters, namely the channel statistics. We then propose a Q-learning algorithm where the controller learns the optimal policy without having knowledge of neither the CSI nor the channel statistics. We illustrate the performance of our algorithms with numerical studies.
  @cite_18: 
  @cite_14: This paper studies Ultra-Reliable Low-Latency Communications (URLLC), an important service class of emerging 5G networks. In this class, multiple unreliable transmissions must be combined to achieve reliable latency: a user experiences a frame success when the entire L bits are received correctly within a deadline, and its latency performance is reliable when the frame success rate is above a threshold. When jointly serving multiple users, a natural URLLC scheduling question arises: given the uncertainty of the wireless channel, can we find a scheduling policy that allows all users to meet a target reliable latency objective? This is called the URLLC SLA Satisfaction (USS) problem. The USS problem is an infinite horizon constrained Markov Decision Process, for which, after establishing a convenient property, we are able to derive an optimal policy based on dynamic programming. Our policy suffers from the curse of dimensionality, hence for large instances we propose a class of knapsack-inspired computationally efficient — but not necessarily optimal — policies. We prove that every policy in that class becomes optimal in a fluid regime, where both the deadline and L scale to infinity, while our simulations show that the policies perform well even in small practical instances of the USS problem.
  @cite_4: We consider the problem of resource allocation in downlink OFDMA systems for multi service and unknown environment. Due to users' mobility and intercell interference, the base station cannot predict neither the Signal to Noise Ratio (SNR) of each user in future time slots nor their probability distribution functions. In addition, the traffic is bursty in general with unknown arrival. The probability distribution functions of the SNR, channel state and traffic arrival density are then unknown. Achieving a multi service Quality of Service (QoS) while optimizing the performance of the system (e.g. total throughput) is a hard and interesting task since it depends on the unknown future traffic and SNR values. In this paper we solve this problem by modeling the multiuser queuing system as a discrete time linear dynamic system. We develop a robust H∞ controller to regulate the queues of different users. The queues and Packet Drop Rates (PDR) are controlled by proposing a minimum data rate according to the demanded service type of each user. The data rate vector proposed by the controller is then fed as a constraint to an instantaneous resource allocation framework. This instantaneous problem is formulated as a convex optimization problem for instantaneous subcarrier and power allocation decisions. Simulation results show small delays and better fairness among users.
  @cite_8: In this paper, we study resource allocation in a downlink OFDMA system assuming imperfect channel state information (CSI) at the transmitter. To achieve the individual QoS of the users in OFDMA system, adaptive resource allocation is very important, and has therefore been an active area of research. However, in most of the the previous work perfect CSI at the transmitter is assumed which is rarely possible due to channel estimation error and feedback delay. In this paper, we study the effect of channel estimation error on resource allocation in a downlink OFDMA system. We assume that each user terminal estimates its channel by using an MMSE estimator and sends its CSI back to the base station through a feedback channel. We approach the problem by using convex optimization framework, provide an explicit closed form expression for the users' transmit power and then develop an optimal margin adaptive resource allocation algorithm. Our proposed algorithm minimizes the total transmit power of the system subject to constraints on users' average data rate. The algorithm has polynomial complexity and solves the problem with zero optimality gaps. Simulation results show that our algorithm highly improves the system performance in the presence of imperfect channel estimation.
  @cite_16: 
  @cite_13: In OFDMA downlink resource allocation, the base station exploits knowledge of the users' channel realizations in order to opportunistically assign users to appropriate subchannels, as well as to optimize the rates and powers across those subchannels. Because reverse-link bandwidth is scarce, the base station's channel knowledge must be obtained via some form of limited feedback. While the typical assumption for this feedback is that it comes in the form of heavily quantized SNR estimates computed at the user terminals, we propose to use ACK NAK feedback that is already provided by higher-layer ARQ. Towards this aim, we propose a greedy resource allocation scheme, based on distributional (rather than point) estimates of SNR. We also show how these SNR distributions can be updated recursively for Markov time-varying channels.1
  @cite_12: In this paper, we study resource allocation and relay selection in a decode-and-forward downlink OFDMA cooperative network assuming imperfect channel state information (CSI) at the base station (BS). We assume that due to feedback delay, the BS has only outdated CSI of BS to users and inter-user links. We propose a centralized optimization framework in which the BS takes decisions on the basis of available outdated CSI given the conditional probability distribution of the current CSI. We approach the problem by using dual optimization framework, decompose it into per-subcarrier optimization problem, and develop an optimal resource allocation and relay selection algorithm. The proposed algorithm minimizes the total transmit power of the system by jointly selecting the best relay node for each source-destination pair, and optimally allocating power and subcarriers to the source and relay nodes under constraints on users' conditional expected data rates. The duality gaps of the solution are virtually zero. Furthermore, we show that the per-subcarrier decomposition of the problem reduces the complexity of the solution from exponential to polynomial. Simulation results show that the system performance may be badly affected by feedback delay. Simulation results also show that users' cooperation significantly reduces the power consumption compared to non-cooperative communication.
  @cite_20: Next-generation cellular wireless communication networks aim to provide a variety of quality-of-service (QoS)-sensitive packet-based services to downlink users. Included among these are real-time multimedia services, which have stringent delay requirements. Downlink packet scheduling at the base station plays a key role in efficiently allocating system resources to meet the desired level of QoS for various users. In this paper, we employ dynamic programming (DP) to study the design of a downlink packet scheduler capable of supporting real-time multimedia applications. Under well-justified modeling reductions, we extensively characterize structural properties of the optimal control associated with the DP problem. We leverage intuition gained from these properties to propose a heuristic scheduling policy, namely, Channel-Aware Earliest Due Date (CA-EDD), which is based on a "quasi- static" approach to scheduling. The per-time-slot implementation complexity of CA-EDD is only O(K) for a system with K downlink users. Experimental results show that CA-EDD delivers up to 50 percent of performance gains over benchmark schedulers. CA-EDD achieves these performance gains by using channel and deadline information in conjunction with application layer information (relative importance of packets) in a systematic and unified way for scheduling.
summary: The issue of deadline-constrained traffic scheduling has been investigated by several works including @cite_20 @cite_18 @cite_14 @cite_16 . For example, in @cite_16 , the authors study the problem of dynamic channel allocation in a single user multi-channel system with service costs and deadline-constrained traffic. They propose online algorithms to enable the controller to learn the optimal policy based on Thompson sampling for multi-armed bandit problems. The MDP framework and reinforcement learning approaches for downlink packet scheduling are considered in @cite_20 @cite_14 @cite_13 @cite_4 @cite_8 @cite_12 . In @cite_20 , the authors propose an MDP for deadline-constrained packet scheduling problem and use dynamic programming to find the optimal scheduling policies. The authors do not consider QoS constraints in the scheduling problem.