  The trend towards highly parallel multi-processing is ubiquitous in all
modern computer architectures, ranging from handheld devices to large-scale HPC
systems; yet many applications are struggling to fully utilise the multiple
levels of parallelism exposed in modern high-performance platforms. In order to
realise the full potential of recent hardware advances, a mixed-mode between
shared-memory programming techniques and inter-node message passing can be
adopted which provides high-levels of parallelism with minimal overheads. For
scientific applications this entails that not only the simulation code itself,
but the whole software stack needs to evolve. In this paper, we evaluate the
mixed-mode performance of PETSc, a widely used scientific library for the
scalable solution of partial differential equations. We describe the addition
of OpenMP threaded functionality to the library, focusing on sparse
matrix-vector multiplication. We highlight key challenges in achieving good
parallel performance, such as explicit communication overlap using task-based
parallelism, and show how to further improve performance by explicitly load
balancing threads within MPI processes. Using a set of matrices extracted from
Fluidity, a CFD application code which uses the library as its linear solver
engine, we then benchmark the parallel performance of mixed-mode PETSc across
multiple nodes on several modern HPC architectures. We evaluate the parallel
scalability on Uniform Memory Access (UMA) systems, such as the Fujitsu
PRIMEHPC FX10 and IBM BlueGene/Q, as well as a Non-Uniform Memory Access (NUMA)
Cray XE6 platform. A detailed comparison is performed which highlights the
characteristics of each particular architecture, before demonstrating efficient
strong scalability of sparse matrix-vector multiplication with significant
speedups over the pure-MPI mode.
