Skip to content

What impact is the impact factor measuring?

July 3, 2009

I recently read a very interesting article in PLoS ONE examining various measures of the scientific importance of particular journals:

Bollen, J., Van de Sompel, H., Hagberg, A., & Chute, R. (2009). A Principal Component Analysis of 39 Scientific Impact Measures PLoS ONE, 4 (6) DOI: 10.1371/journal.pone.0006022

The article isn’t breaking new ground in its criticism of the impact factor, calculated by Thomson Scientific.  However, the statistical analysis comparing multiple measures of importance sheds new light on the relationship between the various measures.

The authors analyzed 39 different impact measures that fall into two main groups: those that look at citation counts and those that look at online usage data (page views and downloads).  A few additional measures that take account of online social networks were also included.

Schematic representation of PCA analysis

Schematic representation of PCA analysis. JIF is the normal Journal Impact Factor from Thompson Scientific

In general, the usage measurements cluster more closely together than the citation measurements – they are measuring approximately the same thing.

As a result of this analysis, they were able to differentiate measures that looked at immediate (“rapid”) use vs. longer term use (“delayed”), and to distinguish measures that look at how popular a resource is vs. how prestigious a resource is.

All of this leads us to repeat the problem posited by the authors:  we don’t have an accepted definition of what “impact” really is.

Publications, institutions, and tenure committees all have different needs and requirements.  For example, the faculty at one large research institution may be more concerned about prestige, while another may need to market their programs and examine their popularity.  I think this analysis shows that folks can and should be a bit more choosy when selecting the measure they use to judge their competitors, their research, and their colleagues.

From → communication

2 Comments

Comments are closed.

Follow

Get every new post delivered to your Inbox.

Join 77 other followers

%d bloggers like this:
http://undergraduatesciencelibrarian.org/2009/07/03/what-impact-is-the-impact-factor-measuring/
undergraduatesciencelibrarian.org
crawl=CC-MAIN-2013-48
58037
application/http; msgtype=response
sha1:HDADL7X6QSHSW7LXT7K27PRXCUNE2BYM
<urn:uuid:9ae2bab0-e24b-443c-9ec0-a238fbb8902a>
2013-12-12T15:36:15Z
192.0.80.250
sha1:IPODPKQO2CDT33Q7MIPABBKHL3L7IZHO
<urn:uuid:2101c90e-7835-4992-9649-3271573f3fab>
http://undergraduatesciencelibrarian.org/2009/07/03/what-impact-is-the-impact-factor-measuring/
length
response
<urn:uuid:21218e87-369c-4ca6-807b-bf000ae3094f>
robots: classic
hostname: ip-10-33-133-15.ec2.internal
software: Nutch 1.6 (CC)/CC WarcExport 1.0
isPartOf: CC-MAIN-2013-48
operator: CommonCrawl Admin
description: Wide crawl of the web with URLs provided by Blekko for Winter 2013
publisher: CommonCrawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf