  Effective unit tests can help guard and improve software quality but require
a substantial amount of time and effort to write and maintain. A unit test
consists of a test prefix and a test oracle. Synthesizing test oracles,
especially functional oracles, is a well-known challenging problem. Recent
studies proposed to leverage neural models to generate test oracles, i.e.,
neural test oracle generation (NTOG), and obtained promising results. However,
after a systematic inspection, we find there are some inappropriate settings in
existing evaluation methods for NTOG. These settings could mislead the
understanding of existing NTOG approaches' performance. We summarize them as 1)
generating test prefixes from bug-fixed program versions, 2) evaluating with an
unrealistic metric, and 3) lacking a straightforward baseline. In this paper,
we first investigate the impacts of these settings on evaluating and
understanding the performance of NTOG approaches. We find that 1)
unrealistically generating test prefixes from bug-fixed program versions
inflates the number of bugs found by the state-of-the-art NTOG approach TOGA by
61.8%, 2) FPR (False Positive Rate) is not a realistic evaluation metric and
the Precision of TOGA is only 0.38%, and 3) a straightforward baseline
NoException, which simply expects no exception should be raised, can find 61%
of the bugs found by TOGA with twice the Precision. Furthermore, we introduce
an additional ranking step to existing evaluation methods and propose an
evaluation metric named Found@K to better measure the cost-effectiveness of
NTOG approaches. We propose a novel unsupervised ranking method to instantiate
this ranking step, significantly improving the cost-effectiveness of TOGA.
Eventually, we propose a more realistic evaluation method TEval+ for NTOG and
summarize seven rules of thumb to boost NTOG approaches into their practical
usages.
