  HPC environments have traditionally been designed to meet the compute demand
of scientific applications and data has only been a second order concern. With
science moving toward data-driven discoveries relying more on correlations in
data to form scientific hypotheses, the limitations of HPC approaches become
apparent: Architectural paradigms such as the separation of storage and compute
are not optimal for I/O intensive workloads (e.g. for data preparation,
transformation and SQL). While there are many powerful computational and
analytical libraries available on HPC (e.g. for scalable linear algebra), they
generally lack the usability and variety of analytical libraries found in other
environments (e.g. the Apache Hadoop ecosystem). Further, there is a lack of
abstractions that unify access to increasingly heterogeneous infrastructure
(HPC, Hadoop, clouds) and allow reasoning about performance trade-offs in this
complex environment. At the same time, the Hadoop ecosystem is evolving rapidly
and has established itself as de-facto standard for data-intensive workloads in
industry and is increasingly used to tackle scientific problems. In this paper,
we explore paths to interoperability between Hadoop and HPC, examine the
differences and challenges, such as the different architectural paradigms and
abstractions, and investigate ways to address them. We propose the extension of
the Pilot-Abstraction to Hadoop to serve as interoperability layer for
allocating and managing resources across different infrastructures. Further,
in-memory capabilities have been deployed to enhance the performance of
large-scale data analytics (e.g. iterative algorithms) for which the ability to
re-use data across iterations is critical. As memory naturally fits in with the
Pilot concept of retaining resources for a set of tasks, we propose the
extension of the Pilot-Abstraction to in-memory resources.
