Evaluation based on scientific publishing: Home

Evaluation based on scientific publishing

Research activity is constantly being evaluated by both outside experts and in-house experts. An individual researcher or research group may evaluate the quality of their own research findings in relation to the research within their own discipline, and will be interested, for example, in the level of research produced by co-workers. The focus of the evaluation can be a single researcher, or a research group/institution, or the research activity of a whole country.  The volume and quality of research is a basis for allocating basic funding for the universities in Finland.

Before publishing the manuscript is peer reviewed. More information about peer reviewing. After publishing academic research can be evaluated using either bibliometric methods based on quantitative research or expert assessment based on qualitative methods.  Bibliometric methods are inexpensive because they can be performed by utilizing data in well known citation databases. Bibliometrics examines, for example, the amount of citations an article has received. More information about citation analysis. Different forms of publishing are characteristic of different disciplines so also the evaluation methods should vary.

The Oulu University Library offers bibliometric services for departments, faculties and research groups to support their decision making. Additionally, guidance and instructions related to  bibliometrics are provided for staff and students. Read more.

Differences between disciplines

The most common forms of scientific publication are journals, books, compilations and conference proceedings. Different forms of publishing are characteristic of different disciplines. However, all disciplines share the high regard for internationally peer reviewed journal articles. More information about peer reviewing.

The majority of scientific publishing is nowadays in English. Most disciplines, however, do have scientific journals that publish in their national languages.

Scientific disciplines can be divided into four main categories:

  1. Hard-pure
  2. Hard-applied
  3. Soft-applied
  4. Soft-pure disciplines

Disciplines within the main categories can differ from each other, and even within the same discipline there are differences, both nationally and between universities.

Between the main categories, there are differences in publishing practices. Different disciplines have differences in the lengths of publications, how often they publish and how long it takes for articles to be published. There are disciplines where almost all publications are collaborative publications, and other disciplines where there is usually only one author. There are also differences between disciplines with regard to who is credited as the author(s) of a publication and what order the names are written in.

In some fields, literature becomes outdated rapidly, and in others older literature gets cited. In some fields, research results are only understood by a narrow specialist group, and in other cases the language use is almost everyday language, and the results are often popularised.

Citation practices differ significantly according to discipline. In different disciplines citations are aimed at different aged literature and the average amounts of citations in articles vary: Times Higher Education  "average rates of citation by field for journal articles indexed by Thomson Reuters in its Essential Science Indicators database from 2000 to 2010".

The obsolescence of literature in different disciplines can be seen in the citations that publications get and the bibliographies of publications. Literature grows old fast, especially in Medicine and Biosciences, but not so fast in soft disciplines. However, in Mathematics, for example, older literature often gets cited. The obsolescence can be measured with Citing Half-life and Cited Half-life calculations. The Thomson Reuters produced Journal Citation Reports calculates Citing Half-life and Cited Half-life. Older literature ceases to be cited not only because the information goes out of date, but also because it becomes commonly accepted knowledge, and is then utilised but not marked as a citation in publications. Journals of different disciplines receive very different IF scores because of different citation norms, and yet they are calculated in the same way for all journals, counting two years worth of citation information. The Impact Factor is most suitable in evaluating journals for those hard disciplines where literature is cited soon after the publishing, and where literature gets out of date quickly too.

The citation age distribution of published articles from 2003 in different disciplines. When calculating the Impact Factor of journals, the citations taken into account are from 2001-2002. The citation material is from the Thomson Reuters Web of Science database.
Image source: Joint Committeeon Quantitative Assessment of Research Citation Statistics. http://www.mathunion.org/fileadmin/IMU/Report/CitationStatistics.pdf. 8.4.2009.

Using citation data as the tool of evaluation is only applicable to those disciplines in which international journal articles are the main publishing forums. The Web of Science and Scopus databases with the citation information concentrate on international peer reviewed journals, and do not cover systematically other forms of publishing. Their journal collection is also centred around journals from the United States and English language journals, and their coverage of Social Science and Arts is poor, even with regard to international journals.

More information

Puuska HM (2014) Scholarly Publishing Patterns in Finland - A comparison of disciplinary groups. Acta Electronica Universitatis Tamperensis.

Statistics of Finnish universities and higher education

The Ministry of Education and Culture collects data on publications on an annual basis. The data are used to monitor the structure and progress in research and as a basis for allocating basic funding for universities.

KOTA contains statistics about Finnish universities and higher education between 1981-2009.

Vipunen contains statistics and indicators from 2010 for education in a number of educational sectors, placement of students after completion, research conducted in higher education institutions, the population's educational structure and the socio-economic background of students. 

Juuli provides a tool for browsing information on publications by Finnish research organisations and for conducting searches.
http://libguides.oulu.fi/evaluatingpublications
libguides.oulu.fi
crawl=CC-MAIN-2016-40
61606
application/http; msgtype=response
sha1:JVUZUU56MQ5FT4SY6X6WIEDAOUZCV4NE
<urn:uuid:62a709fd-5df6-4382-98a3-d07720901661>
2019-11-19T00:57:18Z
52.48.240.72
text/html
sha1:QD63UD6AGIJIE225HR73W3LOXE6JV66Y
<urn:uuid:a8ee4e59-3c49-4c3c-a5dd-0f995cfd765c>
http://libguides.oulu.fi/evaluatingpublications
response
<urn:uuid:f124edf4-a669-4faa-bc81-e7c2ae5a9a7d>
isPartOf: CC-MAIN-2019-47
publisher: Common Crawl
description: Wide crawl of the web for November 2019
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-211.ec2.internal
software: Apache Nutch 1.16 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.1-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/