Rule-based deduplication of article records from bibliographic databases

Yu Jiang, Can Lin, Weiyi Meng, Clement Yu, Aaron M. Cohen, Neil R. Smalheiser

Research output: Contribution to journalArticlepeer-review

15 Scopus citations

Abstract

We recently designed and deployed a metasearch engine, Metta, that sends queries and retrieves search results from five leading biomedical databases: PubMed, EMBASE, CINAHL, PsycINFO and the Cochrane Central Register of Controlled Trials. Because many articles are indexed in more than one of these databases, it is desirable to deduplicate the retrieved article records. This is not a trivial problem because data fields contain a lot of missing and erroneous entries, and because certain types of information are recorded differently (and inconsistently) in the different databases. The present report describes our rule-based method for deduplicating article records across databases and includes an open-source script module that can be deployed freely. Metta was designed to satisfy the particular needs of people who are writing systematic reviews in evidence-based medicine. These users want the highest possible recall in retrieval, so it is important to err on the side of not deduplicating any records that refer to distinct articles, and it is important to perform deduplication online in real time. Our deduplication module is designed with these constraints in mind. Articles that share the same publication year are compared sequentially on parameters including PubMed ID number, digital object identifier, journal name, article title and author list, using text approximation techniques. In a review of Metta searches carried out by public users, we found that the deduplication module was more effective at identifying duplicates than EndNote without making any erroneous assignments.

Original languageEnglish (US)
Article numberbat086
JournalDatabase
Volume2014
DOIs
StatePublished - 2014

ASJC Scopus subject areas

  • Information Systems
  • Biochemistry, Genetics and Molecular Biology(all)
  • Agricultural and Biological Sciences(all)

Fingerprint Dive into the research topics of 'Rule-based deduplication of article records from bibliographic databases'. Together they form a unique fingerprint.

Cite this
https://ohsu.pure.elsevier.com/en/publications/rule-based-deduplication-of-article-records-from-bibliographic-da-2
ohsu.pure.elsevier.com
crawl=CC-MAIN-2021-04
48601
application/http; msgtype=response
sha1:5XBUQHCTBVRJJMT3STSP2UGZEN5IB6Z7
<urn:uuid:5f84c9ec-2f67-4338-a256-b1a19b3563e5>
2021-01-24T13:14:09Z
3.90.122.189
text/html
sha1:DECIYFR2W52FKB42OXGME2FIAA5MNBLB
<urn:uuid:c4d9f751-36be-4c65-ad8f-fe29ae989c23>
https://ohsu.pure.elsevier.com/en/publications/rule-based-deduplication-of-article-records-from-bibliographic-da-2
response
<urn:uuid:b0eb7c74-5994-45a1-927b-427be71193c7>
isPartOf: CC-MAIN-2021-04
publisher: Common Crawl
description: Wide crawl of the web for January 2021
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-254.ec2.internal
software: Apache Nutch 1.17 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.2-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/