  With the rapid development of artificial intelligence (AI), ethical issues
surrounding AI have attracted increasing attention. In particular, autonomous
vehicles may face moral dilemmas in accident scenarios, such as staying the
course resulting in hurting pedestrians or swerving leading to hurting
passengers. To investigate such ethical dilemmas, recent studies have adopted
preference aggregation, in which each voter expresses her/his preferences over
decisions for the possible ethical dilemma scenarios, and a centralized system
aggregates these preferences to obtain the winning decision. Although a useful
methodology for building ethical AI systems, such an approach can potentially
violate the privacy of voters since moral preferences are sensitive information
and their disclosure can be exploited by malicious parties. In this paper, we
report a first-of-its-kind privacy-preserving crowd-guided AI decision-making
approach in ethical dilemmas. We adopt the notion of differential privacy to
quantify privacy and consider four granularities of privacy protection by
taking voter-/record-level privacy protection and centralized/distributed
perturbation into account, resulting in four approaches VLCP, RLCP, VLDP, and
RLDP. Moreover, we propose different algorithms to achieve these privacy
protection granularities, while retaining the accuracy of the learned moral
preference model. Specifically, VLCP and RLCP are implemented with the data
aggregator setting a universal privacy parameter and perturbing the averaged
moral preference to protect the privacy of voters' data. VLDP and RLDP are
implemented in such a way that each voter perturbs her/his local moral
preference with a personalized privacy parameter. Extensive experiments on both
synthetic and real data demonstrate that the proposed approach can achieve high
accuracy of preference aggregation while protecting individual voter's privacy.
