  Graphical User Interface (GUI) provides a visual bridge between a software
application and end users, through which they can interact with each other.
With the upgrading of mobile devices and the development of aesthetics, the
visual effects of the GUI are more and more attracting, and users pay more
attention to the accessibility and usability of applications. However, such GUI
complexity posts a great challenge to the GUI implementation. According to our
pilot study of crowdtesting bug reports, display issues such as text overlap,
component occlusion, missing image always occur during GUI rendering on
different devices due to the software or hardware compatibility. They
negatively influence the app usability, resulting in poor user experience. To
detect these issues, we propose a fully automated approach, Nighthawk, based on
deep learning for modelling visual information of the GUI screenshot. Nighthawk
can detect GUIs with display issues and also locate the detailed region of the
issue in the given GUI for guiding developers to fix the bug. At the same time,
training the model needs a large amount of labeled buggy screenshots, which
requires considerable manual effort to prepare them. We therefore propose a
heuristic-based training data auto-generation method to automatically generate
the labeled training data. The evaluation demonstrates that our Nighthawk can
achieve average 0.84 precision and 0.84 recall in detecting UI display issues,
average 0.59 AP and 0.60 AR in localizing these issues. We also evaluate
Nighthawk with popular Android apps on Google Play and F-Droid, and
successfully uncover 151 previously-undetected UI display issues with 75 of
them being confirmed or fixed so far.
