The Future of Research Communications and e-Scholarship

Search
Close this search box.

New models for evaluation for research and researchers

On this page:

Session:  New models for evaluation for research and researchers

Session organizers:  Carole Goble

When:  Wed, March 20th 13:00-14:30

 

Description:  Online research and publishing gives us opportunities to revise how we evaluate research and how we evaluate researchers.  For the evaluation of research, reproducibility underpins the scientific method. For an experimental finding to be reproducible, its materials must be available and its methods clear, accurate and transparent. Increasingly within the scholarly community (academics, funders, publishers, industry) there is a demand that published results are not just reported as reproducible but can be evaluated as such by independent third parties. This demand is being addressed by a range of models: from reproducibility certification (e.g. https://www.scienceexchange.com/reproducibility) through to computational platforms for re-running codes (e.g. http://runmycode.org) and executable papers.  For the evaluation of researchers, the cost entailed by supporting reproducibility must be balanced by rewards. New (alt)metrics are needed to discriminate between reproducible and merely reported research in order to impact reputation and credit, and drive cultural change. There must be rewards, too, for the provisioning of reproducibility platforms and for peer-evaluating in addition to peer-reviewing.

The session aims to produce a roadmap of the steps needed to realise these new models of evaluation. The session is structured around a role-playing panel who will discuss several scenarios. Each panelist represents (several) communities, will be sceptical as well as positive, and will stay in their roles. The audience will participate by role-playing and scenario proposing too. There will be no formal presentations.

The panelists are:
Carole Goble (Manchester) : chair, technical engineer, realist, tool provider
Scott Edmunds (GigaScience): publisher (old school and new school)
Christine Borgman (UCLA): digital librarian, scholarship gatekeeper
Jan Reichelt (Mendeley): scholarly comms vendor, commercial funder
Victoria Stodden (Columbia): policy maker, government, public funder
Phil Bourne (PLoS, UCSD): institutional dean, promotions chair, hard pressed editor in chief and reviewer, data service manager
Steve Pettifer (UTOPIA, Manchester): young academic, enthusiast

Presentation: Click Here to download the PPT
 

Archive: https://archive.force11.net/node/4267
https://force11.org/info/new-models-for-evaluation-for-research-and-researchers/
force11.org
CC-MAIN-2024-10
174564
application/http; msgtype=response
sha1:EHGPTRIYFBQLYWIPQNEGHJXAORPNFWUM
<urn:uuid:ce25686a-d059-40f5-bbaa-c92ac2bf1d6d>
2024-02-21T19:38:23Z
35.212.50.41
text/html
sha1:D4MKMIAIWLENTLMXGIIJMXGUC67QVBVI
<urn:uuid:7eb01b83-f9c4-404b-be06-233b1dfc99f8>
https://force11.org/info/new-models-for-evaluation-for-research-and-researchers/
response
<urn:uuid:57c20b36-b0da-4535-b7c8-9dcfc11a36b8>
isPartOf: CC-MAIN-2024-10
publisher: Common Crawl
description: Wide crawl of the web for February/March 2024
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-31
software: Apache Nutch 1.19 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.5-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/