Journal article

Unbiased evaluation of ranking metrics reveals consistent performance in science and technology citation data

  • Xu, Shuqi Institute of Fundamental and Frontier Sciences, University of Electronic Science and Technology of China, Chengdu 610054, PR China
  • Mariani, Manuel Sebastian Institute of Fundamental and Frontier Sciences, University of Electronic Science and Technology of China, Chengdu 610054, PR China - URPP Social Networks, University of Zurich, 8050 Zurich, Switzerland
  • Lü, Linyuan Institute of Fundamental and Frontier Sciences, University of Electronic Science and Technology of China, Chengdu 610054, PR China - Alibaba Research Center for Complexity Sciences, Hangzhou Normal University, 311121 Hangzhou, PR China
  • Medo, Matúš Institute of Fundamental and Frontier Sciences, University of Electronic Science and Technology of China, Chengdu 610054, PR China - Department of Radiation Oncology, Inselspital, Bern University Hospital and University of Bern, 3010 Bern, Switzerland - Department of Physics, University of Fribourg, 1700 Fribourg, Switzerland
Show more…
    04.02.2020
Published in:
  • Journal of Informetrics. - 2020, vol. 14, no. 1, p. 101005
English Despite the increasing use of citation-based metrics for research evaluation purposes, we do not know yet which metrics best deliver on their promise to gauge the significance of a scientific paper or a patent. We assess 17 network-based metrics by their ability to identify milestone papers and patents in three large citation datasets. We find that traditional information-retrieval evaluation metrics are strongly affected by the interplay between the age distribution of the milestone items and age biases of the evaluated metrics. Outcomes of these metrics are therefore not representative of the metrics’ ranking ability. We argue in favor of a modified evaluation procedure that explicitly penalizes biased metrics and allows us to reveal metrics’ performance patterns that are consistent across the datasets. PageRank and LeaderRank turn out to be the best-performing ranking metrics when their age bias is suppressed by a simple transformation of the scores that they produce, whereas other popular metrics, including citation count, HITS and Collective Influence, produce significantly worse ranking results.
Faculty
Faculté des sciences et de médecine
Department
Département de Physique
Language
  • English
Classification
Physics
License
License undefined
Identifiers
Persistent URL
https://folia.unifr.ch/unifr/documents/308764
Statistics

Document views: 17 File downloads:
  • med_uer.pdf: 15
https://folia.unifr.ch/unifr/documents/308764
folia.unifr.ch
CC-MAIN-2023-50
25074
application/http; msgtype=response
sha1:KMA3EJILCOKQSAY3O5Y6EMXPLZWS3IJ4
<urn:uuid:a78ace65-ee5d-44d8-b814-2d275372477f>
2023-12-06T17:38:05Z
153.109.157.125
text/html
sha1:FZ6NND6F5C45T4M3E5ZKWKPDNZOF5XJW
<urn:uuid:280fa2ef-c733-4982-920b-5829eed0d337>
https://folia.unifr.ch/unifr/documents/308764
response
<urn:uuid:f6d69427-ecfc-46bd-b077-cbb723983ff2>
isPartOf: CC-MAIN-2023-50
publisher: Common Crawl
description: Wide crawl of the web for November/December 2023
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-150
software: Apache Nutch 1.19 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.5-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/