  Context: Visual GUI testing (VGT) is referred to as the latest generation
GUI-based testing. It is a tool-driven technique, which uses image recognition
for interacting with and asserting the behavior of the system under test.
Motivated by the industrial need of a large Turkish software and systems
company providing solutions in the areas of defense and IT sector, an
action-research project was recently initiated to implement VGT in several
teams and projects in the company.
  Objective: To address the above needs, we planned and carried out an
empirical investigation with the goal of assessing VGT using two tools (Sikuli
and JAutomate). The purpose was to determine a suitable approach and tool for
VGT of a given project (software product) in the company, increase the know-how
in the company's test teams.
  Method: Using an action-research case-study design, we investigated the use
of VGT in the studied organization. Specifically, using the two selected VGT
tools, we conducted a quantitative and a qualitative evaluation of VGT.
  Results: By assessing the list of Challenges, Problems and Limitations (CPL),
proposed in previous work, in the context of our empirical study, we found that
test-tool- and SUT-related CPLs were quite comparable to a previous empirical
study, e.g., the synchronization between SUT and test tools were not always
robust and there were failures in test tools' image recognition features. When
assessing the types of test maintenance activities, when executing the
automated test cases on next versions of the SUTs, for the case of the two test
tools, we found that about half of the test cases (59.1% and 47.8%) failed in
the next version.
  Conclusion: By our results, we confirm some of the previously-reported issues
when conducting VGT. Further, we highlight some additional challenges in test
maintenance when using VGT.
