  Performance is a key quality of modern software. Although recent years have
seen a spike in research on automated improvement of software's execution time,
energy, memory consumption, etc., there is a noticeable lack of standard
benchmarks for such work. It is also unclear how such benchmarks are
representative of current software. Furthermore, frequently non-functional
properties of software are targeted for improvement one-at-a-time, neglecting
potential negative impact on other properties.
  In order to facilitate more research on automated improvement of
non-functional properties of software, we conducted a survey gathering
benchmarks used in previous work. We considered 5 major online repositories of
software engineering work: ACM Digital Library, IEEE Xplore, Scopus, Google
Scholar, and ArXiV. We gathered 5000 publications (3749 unique), which were
systematically reviewed to identify work that empirically improves
non-functional properties of software. We identified 386 relevant papers.
  We find that execution time is the most frequently targeted property for
improvement (in 62% of relevant papers), while multi-objective improvement is
rarely considered (5%). Static approaches are prevalent (in 53% of papers),
with exploratory approaches (evolutionary in 18% and non-evolutionary in 14% of
papers) increasingly popular in the last 10 years. Only 40% of 386 papers
describe work that uses benchmark suites, rather than single software, of those
SPEC is most popular (covered in 33 papers). We also provide recommendations
for choice of benchmarks in future work, noting, e.g., lack of work that covers
Python or JavaScript. We provide all programs found in the 386 papers on our
dedicated webpage at https://bloa.github.io/nfunc_survey/
  We hope that this effort will facilitate more research on the topic of
automated improvement of software's non-functional properties.
