  Verifiability is a core content policy of Wikipedia: claims that are likely
to be challenged need to be backed by citations. There are millions of articles
available online and thousands of new articles are released each month. For
this reason, finding relevant sources is a difficult task: many claims do not
have any references that support them. Furthermore, even existing citations
might not support a given claim or become obsolete once the original source is
updated or deleted. Hence, maintaining and improving the quality of Wikipedia
references is an important challenge and there is a pressing need for better
tools to assist humans in this effort. Here, we show that the process of
improving references can be tackled with the help of artificial intelligence
(AI). We develop a neural network based system, called Side, to identify
Wikipedia citations that are unlikely to support their claims, and subsequently
recommend better ones from the web. We train this model on existing Wikipedia
references, therefore learning from the contributions and combined wisdom of
thousands of Wikipedia editors. Using crowd-sourcing, we observe that for the
top 10% most likely citations to be tagged as unverifiable by our system,
humans prefer our system's suggested alternatives compared to the originally
cited reference 70% of the time. To validate the applicability of our system,
we built a demo to engage with the English-speaking Wikipedia community and
find that Side's first citation recommendation collects over 60% more
preferences than existing Wikipedia citations for the same top 10% most likely
unverifiable claims according to Side. Our results indicate that an AI-based
system could be used, in tandem with humans, to improve the verifiability of
Wikipedia. More generally, we hope that our work can be used to assist fact
checking efforts and increase the general trustworthiness of information
online.
