  The amazing advances being made in the fields of machine and deep learning
are a highlight of the Big Data era for both enterprise and research
communities. Modern applications require resources beyond a single node's
ability to provide. However this is just a small part of the issues facing the
overall data processing environment, which must also support a raft of data
engineering for pre- and post-data processing, communication, and system
integration. An important requirement of data analytics tools is to be able to
easily integrate with existing frameworks in a multitude of languages, thereby
increasing user productivity and efficiency. All this demands an efficient and
highly distributed integrated approach for data processing, yet many of today's
popular data analytics tools are unable to satisfy all these requirements at
the same time.
  In this paper we present Cylon, an open-source high performance distributed
data processing library that can be seamlessly integrated with existing Big
Data and AI/ML frameworks. It is developed with a flexible C++ core on top of a
compact data structure and exposes language bindings to C++, Java, and Python.
We discuss Cylon's architecture in detail, and reveal how it can be imported as
a library to existing applications or operate as a standalone framework.
Initial experiments show that Cylon enhances popular tools such as Apache Spark
and Dask with major performance improvements for key operations and better
component linkages. Finally, we show how its design enables Cylon to be used
cross-platform with minimum overhead, which includes popular AI tools such as
PyTorch, Tensorflow, and Jupyter notebooks.
