  Machine Ethics (ME) is concerned with the design of Artificial Moral Agents
(AMAs), i.e. autonomous agents capable of reasoning and behaving according to
moral values. Previous approaches have treated values as labels associated with
some actions or states of the world, rather than as integral components of
agent reasoning. It is also common to disregard that a value-guided agent
operates alongside other value-guided agents in an environment governed by
norms, thus omitting the social dimension of AMAs. In this blue sky paper, we
propose a new AMA paradigm grounded in moral and social psychology, where
values are instilled into agents as context-dependent goals. These goals
intricately connect values at individual levels to norms at a collective level
by evaluating the outcomes most incentivized by the norms in place. We argue
that this type of normative reasoning, where agents are endowed with an
understanding of norms' moral implications, leads to value-awareness in
autonomous agents. Additionally, this capability paves the way for agents to
align the norms enforced in their societies with respect to the human values
instilled in them, by complementing the value-based reasoning on norms with
agreement mechanisms to help agents collectively agree on the best set of norms
that suit their human values. Overall, our agent model goes beyond the
treatment of values as inert labels by connecting them to normative reasoning
and to the social functionalities needed to integrate value-aware agents into
our modern hybrid human-computer societies.
