  Mutation testing is used to evaluate the effectiveness of test suites. In
recent years, a promising variation called extreme mutation testing emerged
that is computationally less expensive. It identifies methods where their
functionality can be entirely removed, and the test suite would not notice it,
despite having coverage. These methods are called pseudo-tested. In this paper,
we compare the execution and analysis times for traditional and extreme
mutation testing and discuss what they mean in practice. We look at how extreme
mutation testing impacts current software development practices and discuss
open challenges that need to be addressed to foster industry adoption. For
that, we conducted an industrial case study consisting of running traditional
and extreme mutation testing in a large software project from the semiconductor
industry that is covered by a test suite of more than 11,000 unit tests. In
addition to that, we did a qualitative analysis of 25 pseudo-tested methods and
interviewed two experienced developers to see how they write unit tests and
gathered opinions on how useful the findings of extreme mutation testing are.
Our results include execution times, scores, numbers of executed tests and
mutators, reasons why methods are pseudo-tested, and an interview summary. We
conclude that the shorter execution and analysis times are well noticeable in
practice and show that extreme mutation testing supplements writing unit tests
in conjunction with code coverage tools. We propose that pseudo-tested code
should be highlighted in code coverage reports and that extreme mutation
testing should be performed when writing unit tests rather than in a decoupled
session. Future research should investigate how to perform extreme mutation
testing while writing unit tests such that the results are available fast
enough but still meaningful.
