New method ranks quality of scientific journals by field

Feb 27, 2008

Worldwide, the number of scientists is increasing as is the number of scientific journals and published papers, the latter two thanks in large part to the rise of electronic publishing. Scientists and other researchers are finding it more difficult than ever to zero in on the published literature that is most valuable to them.

Now, much like Google locates electronic information likely to be relevant and of quality to the user, a team of researchers from Northwestern University has developed a mathematical method to rank scientific journals according to quality, an approach that will help scientists locate high-impact research papers to read and to cite in their own papers. The rankings also should be useful to university administrators and funding agencies who must evaluate the quality of a researcher’s work.

The team analyzed the citation data of nearly 23 million papers that appeared in 2,267 journals representing 200 academic fields and that spanned the years from 1955 to 2006; their analysis produced 200 separate tables of journal rankings by field. The results, including all the rankings, will be published online Feb. 27 in PLoS ONE, an international, peer-reviewed, open-access, online journal published by the Public Library of Science.

“Trying to find good information in the literature can be a problem for scientists,” said Luís A. Nunes Amaral, associate professor of chemical and biological engineering in Northwestern’s McCormick School of Engineering and Applied Science, who led the study. “Because we have quantified which journals are better, now we can definitively say that the journal in which one publishes provides reliable information about the paper’s quality. It is important to grasp a paper’s quality right away.”

Amaral, Michael J. Stringer, lead author of the paper and a graduate student in Amaral’s research group, and Marta Sales-Pardo, co-author and a postdoctoral fellow, developed methods to look at the enormous amount of published papers and to make sense of them. For each of the 2,267 journals, they charted the citations each paper received across a certain span of years and then developed a model of that data, which allowed the researchers to compare journals.

The researchers’ model produced bell curves for the distribution of “quality” of the papers published in each journal. For each field, all the bell curves for the journals then were compared, which resulted in the journal rankings. The field of ecology, for example, had 36 journals, with Ecology ranked highest and Natural History ranked lowest.

“The higher a journal is ranked, the higher the probability of finding a high-impact paper published in that journal,” said Amaral.

Amaral and his team found that the time scale for a published paper’s complete accumulation of citations -- a gauge for determining the full impact of the paper -- can range from less than one year to 26 years, depending on the journal. Using their new method, the Northwestern researchers can estimate the total number of citations a paper in a specific journal will get in the future and thus determine -- right now -- the paper’s likely impact in its field. This is the kind of information university administrators and funding agencies should find helpful when they are evaluating faculty members for tenure and researchers for grant awards.

“This study is just one example of how large datasets are allowing us to gain insight about systems that we could only speculate about before,” said Stringer. “Understanding how we use information and how it spreads is an important and interesting question. The most surprising thing to me about our study is that the data seem to suggest that the citation patterns in journals are actually simpler than we would have expected.”

The PLoS ONE paper is titled “Effectiveness of Journal Ranking Schemes as a Tool for Locating Information.” Upon publication, the paper will be available at www.plosone.org/doi/pone.0001683 .

Source: Northwestern University

Explore further: Professor quantifies how 'one thing leads to another'

add to favorites email to friend print save as pdf

Related Stories

Bone chemistry reveals royal lifestyle of Richard III

23 hours ago

A recent study by the British Geological Survey, in association with researchers at the University of Leicester, has delved into the bone and tooth chemistry of King Richard III and uncovered fascinating ...

After breakups, newspapers seek path forward

23 hours ago

Following an unprecedented series of spinoffs by major US media companies, the print news industry now faces a rocky future without financial support from deep-pocketed parent firms.

Recommended for you

Social inequality worsens in New Zealand

1 minute ago

Research by Dr Lisa Marriott, an associate professor in Victoria's School of Accounting and Commercial Law, and Dr Dalice Sim, Statistical Consultant in the School of Mathematics, Statistics and Operations Research, builds ...

Jurassic Welsh mammals were picky eaters, study finds

18 hours ago

For most people, mere mention of the word Jurassic conjures up images of huge dinosaurs chomping their way through lush vegetation – and each other. However, mammals and their immediate ancestors were also ...

User comments : 1

Adjust slider to filter visible comments by rank

Display comments: newest first

BigTone
3 / 5 (2) Feb 27, 2008
I think this is great and I like how they patterned relevance or as they put it - impact - of a paper with one Google-esque method of how many other publications reference it.

The only thing that scares me about this is a bias towards traditional consensus as opposed to new ideas. For example, many open minded scientists are taking a look at extremeophiles and are beginning to think the traditional Goldilocks Zone is not a hard requirement for life - especially microbial life - even as we know it and find it here on Earth. Thus, papers on Goldilocks Zone type research may get an unfair number of references because of its long standing history of most scientists concurring with its assumptions.

Although they are counterbalancing this effect with other criteria for impact, those counterbalances also seem biased by traditional assumptions that could be erroneous but mainstream (peer review is only as useful as what your peers generally believe) - to give a very extreme example - I wouldn't want the impact of my genetic research to peer reviewed by Neo-Nazi quasi scientists that are heavily opinionated and would dismiss or spin my findings to meet their own agenda...
http://phys.org/news123333975.html
phys.org
crawl=CC-MAIN-2014-35
62722
application/http; msgtype=response
sha1:UIE4M6TQOPKOSDVHYKWA4UYI4GIAITHI
<urn:uuid:e4f65bfb-75d0-49da-a081-1ee47bc29670>
2014-08-21T11:01:56Z
69.9.167.167
sha1:YNOR33CGQVHGX6B32H464DLFNR36OSEX
<urn:uuid:3fa2be09-4db8-48d0-81c3-da8d6c9057cc>
http://phys.org/news123333975.html
length
response
<urn:uuid:5afaa5df-966e-4855-a18e-7d7a52c70b79>
robots: classic
hostname: ip-10-180-136-8.ec2.internal
software: Nutch 1.6 (CC)/CC WarcExport 1.0
isPartOf: CC-MAIN-2014-35
operator: CommonCrawl Admin
description: Wide crawl of the web with URLs provided by Blekko for August 2014
publisher: CommonCrawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf