  Autonomous cyber and cyber-physical systems need to perform decision-making,
learning, and control in unknown environments. Such decision-making can be
sensitive to multiple factors, including modeling errors, changes in costs, and
impacts of events in the tails of probability distributions. Although
multi-agent reinforcement learning (MARL) provides a framework for learning
behaviors through repeated interactions with the environment by minimizing an
average cost, it will not be adequate to overcome the above challenges. In this
paper, we develop a distributed MARL approach to solve decision-making problems
in unknown environments by learning risk-aware actions. We use the conditional
value-at-risk (CVaR) to characterize the cost function that is being minimized,
and define a Bellman operator to characterize the value function associated to
a given state-action pair. We prove that this operator satisfies a contraction
property, and that it converges to the optimal value function. We then propose
a distributed MARL algorithm called the CVaR QD-Learning algorithm, and
establish that value functions of individual agents reaches consensus. We
identify several challenges that arise in the implementation of the CVaR
QD-Learning algorithm, and present solutions to overcome these. We evaluate the
CVaR QD-Learning algorithm through simulations, and demonstrate the effect of a
risk parameter on value functions at consensus.
