  Context: Scientific software plays an important role in critical decision
making, for example making weather predictions based on climate models, and
computation of evidence for research publications. Recently, scientists have
had to retract publications due to errors caused by software faults. Systematic
testing can identify such faults in code.
  Objective: This study aims to identify specific challenges, proposed
solutions, and unsolved problems faced when testing scientific software.
  Method: We conducted a systematic literature survey to identify and analyze
relevant literature. We identified 62 studies that provided relevant
information about testing scientific software.
  Results: We found that challenges faced when testing scientific software fall
into two main categories: (1) testing challenges that occur due to
characteristics of scientific software such as oracle problems and (2) testing
challenges that occur due to cultural differences between scientists and the
software engineering community such as viewing the code and the model that it
implements as inseparable entities. In addition, we identified methods to
potentially overcome these challenges and their limitations. Finally we
describe unsolved challenges and how software engineering researchers and
practitioners can help to overcome them.
  Conclusions: Scientific software presents special challenges for testing.
Specifically, cultural differences between scientist developers and software
engineers, along with the characteristics of the scientific software make
testing more difficult. Existing techniques such as code clone detection can
help to improve the testing process. Software engineers should consider special
challenges posed by scientific software such as oracle problems when developing
testing techniques.
