  Background: Test smells indicate potential problems in the design and
implementation of automated software tests that may negatively impact test code
maintainability, coverage, and reliability. When poorly described, manual tests
written in natural language may suffer from related problems, which enable
their analysis from the point of view of test smells. Despite the possible
prejudice to manually tested software products, little is known about test
smells in manual tests, which results in many open questions regarding their
types, frequency, and harm to tests written in natural language. Aims:
Therefore, this study aims to contribute to a catalog of test smells for manual
tests. Method: We perform a two-fold empirical strategy. First, an exploratory
study in manual tests of three systems: the Ubuntu Operational System, the
Brazilian Electronic Voting Machine, and the User Interface of a large
smartphone manufacturer. We use our findings to propose a catalog of eight test
smells and identification rules based on syntactical and morphological text
analysis, validating our catalog with 24 in-company test engineers. Second,
using our proposals, we create a tool based on Natural Language Processing
(NLP) to analyze the subject systems' tests, validating the results. Results:
We observed the occurrence of eight test smells. A survey of 24 in-company test
professionals showed that 80.7% agreed with our catalog definitions and
examples. Our NLP-based tool achieved a precision of 92%, recall of 95%, and
f-measure of 93.5%, and its execution evidenced 13,169 occurrences of our
cataloged test smells in the analyzed systems. Conclusion: We contribute with a
catalog of natural language test smells and novel detection strategies that
better explore the capabilities of current NLP mechanisms with promising
results and reduced effort to analyze tests written in different idioms.
