  Big Data is considered proprietary asset of companies, organizations, and
even nations. Turning big data into real treasure requires the support of big
data systems. A variety of commercial and open source products have been
unleashed for big data storage and processing. While big data users are facing
the choice of which system best suits their needs, big data system developers
are facing the question of how to evaluate their systems with regard to general
big data processing needs. System benchmarking is the classic way of meeting
the above demands. However, existent big data benchmarks either fail to
represent the variety of big data processing requirements, or target only one
specific platform, e.g. Hadoop.
  In this paper, with our industrial partners, we present BigOP, an end-to-end
system benchmarking framework, featuring the abstraction of representative
Operation sets, workload Patterns, and prescribed tests. BigOP is part of an
open-source big data benchmarking project, BigDataBench. BigOP's abstraction
model not only guides the development of BigDataBench, but also enables
automatic generation of tests with comprehensive workloads.
  We illustrate the feasibility of BigOP by implementing an automatic test
generation tool and benchmarking against three widely used big data processing
systems, i.e. Hadoop, Spark and MySQL Cluster. Three tests targeting three
different application scenarios are prescribed. The tests involve relational
data, text data and graph data, as well as all operations and workload
patterns. We report results following test specifications.
