  The Gyrokinetic Toroidal Code at Princeton (GTC-P) is a highly scalable and
portable particle-in-cell (PIC) code. It solves the 5D Vlasov-Poisson equation
featuring efficient utilization of modern parallel computer architectures at
the petascale and beyond. Motivated by the goal of developing a modern code
capable of dealing with the physics challenge of increasing problem size with
sufficient resolution, new thread-level optimizations have been introduced as
well as a key additional domain decomposition. GTC-P's multiple levels of
parallelism, including inter-node 2D domain decomposition and particle
decomposition, as well as intra-node shared memory partition and vectorization
have enabled pushing the scalability of the PIC method to extreme computational
scales. In this paper, we describe the methods developed to build a highly
parallelized PIC code across a broad range of supercomputer designs. This
particularly includes implementations on heterogeneous systems using NVIDIA GPU
accelerators and Intel Xeon Phi (MIC) co-processors and performance comparisons
with state-of-the-art homogeneous HPC systems such as Blue Gene/Q. New
discovery science capabilities in the magnetic fusion energy application domain
are enabled, including investigations of Ion-Temperature-Gradient (ITG) driven
turbulence simulations with unprecedented spatial resolution and long temporal
duration. Performance studies with realistic fusion experimental parameters are
carried out on multiple supercomputing systems spanning a wide range of cache
capacities, cache-sharing configurations, memory bandwidth, interconnects and
network topologies. These performance comparisons using a realistic
discovery-science-capable domain application code provide valuable insights on
optimization techniques across one of the broadest sets of current high-end
computing platforms worldwide.
