  Recent work has demonstrated the viability of using crowdsourcing as a tool
for evaluating the truthfulness of public statements. Under certain conditions
such as: (1) having a balanced set of workers with different backgrounds and
cognitive abilities; (2) using an adequate set of mechanisms to control the
quality of the collected data; and (3) using a coarse grained assessment scale,
the crowd can provide reliable identification of fake news. However, fake news
are a subtle matter: statements can be just biased ("cherrypicked"), imprecise,
wrong, etc. and the unidimensional truth scale used in existing work cannot
account for such differences. In this paper we propose a multidimensional
notion of truthfulness and we ask the crowd workers to assess seven different
dimensions of truthfulness selected based on existing literature: Correctness,
Neutrality, Comprehensibility, Precision, Completeness, Speaker's
Trustworthiness, and Informativeness. We deploy a set of quality control
mechanisms to ensure that the thousands of assessments collected on 180
publicly available fact-checked statements distributed over two datasets are of
adequate quality, including a custom search engine used by the crowd workers to
find web pages supporting their truthfulness assessments. A comprehensive
analysis of crowdsourced judgments shows that: (1) the crowdsourced assessments
are reliable when compared to an expert-provided gold standard; (2) the
proposed dimensions of truthfulness capture independent pieces of information;
(3) the crowdsourcing task can be easily learned by the workers; and (4) the
resulting assessments provide a useful basis for a more complete estimation of
statement truthfulness.
