  Shared high-performance computing (HPC) platforms, such as those provided by
XSEDE and Compute Canada, enable researchers to carry out large-scale
computational experiments at a fraction of the cost of the cloud. Most systems
require the use of distributed filesystems (e.g. Lustre) for providing a highly
multi-user, large capacity storage environment. These suffer performance
penalties as the number of files increases due to network contention and
metadata performance. We demonstrate how a combination of two technologies,
Singularity and SquashFS, can help developers, integrators, architects, and
scientists deploy large datasets (O(10M) files) on these shared systems with
minimal performance limitations. The proposed integration enables more
efficient access and indexing than normal file-based dataset installations,
while providing transparent file access to users and processes. Furthermore,
the approach does not require administrative privileges on the target system.
While the examples studied here have been taken from the field of neuroimaging,
the technologies adopted are not specific to that field. Currently, this
solution is limited to read-only datasets. We propose the adoption of this
technology for the consumption and dissemination of community datasets across
shared computing resources.
