  Next-generation wireless deployments are characterized by being dense and
uncoordinated, which often leads to inefficient use of resources and poor
performance. To solve this, we envision the utilization of completely
decentralized mechanisms to enable Spatial Reuse (SR). In particular, we focus
on dynamic channel selection and Transmission Power Control (TPC). We rely on
Reinforcement Learning (RL), and more specifically on Multi-Armed Bandits
(MABs), to allow networks to learn their best configuration. In this work, we
study the exploration-exploitation trade-off by means of the
$\varepsilon$-greedy, EXP3, UCB and Thompson sampling action-selection, and
compare their performance. In addition, we study the implications of selecting
actions simultaneously in an adversarial setting (i.e., concurrently), and
compare it with a sequential approach. Our results show that optimal
proportional fairness can be achieved, even when no information about
neighboring networks is available to the learners and Wireless Networks (WNs)
operate selfishly. However, there is high temporal variability in the
throughput experienced by the individual networks, specially for
$\varepsilon$-greedy and EXP3. These strategies, contrary to UCB and Thompson
sampling, base their operation on the absolute experienced reward, rather than
on its distribution. We identify the cause of this variability to be the
adversarial setting of our setup in which the set of most played actions
provide intermittent good/poor performance depending on the neighboring
decisions. We also show that learning sequentially, even if using a selfish
strategy, contributes to minimize this variability. The sequential approach is
therefore shown to effectively deal with the challenges posed by the
adversarial settings that are typically found in decentralized WNs.
