  Scientists increasingly rely on Python tools to perform scalable distributed
memory array operations using rich, NumPy-like expressions. However, many of
these tools rely on dynamic schedulers optimized for abstract task graphs,
which often encounter memory and network bandwidth-related bottlenecks due to
sub-optimal data and operator placement decisions. Tools built on the message
passing interface (MPI), such as ScaLAPACK and SLATE, have better scaling
properties, but these solutions require specialized knowledge to use. In this
work, we present NumS, an array programming library which optimizes NumPy-like
expressions on task-based distributed systems. This is achieved through a novel
scheduler called Load Simulated Hierarchical Scheduling (LSHS). LSHS is a local
search method which optimizes operator placement by minimizing maximum memory
and network load on any given node within a distributed system. Coupled with a
heuristic for load balanced data layouts, our approach is capable of attaining
communication lower bounds on some common numerical operations, and our
empirical study shows that LSHS enhances performance on Ray by decreasing
network load by a factor of 2x, requiring 4x less memory, and reducing
execution time by 10x on the logistic regression problem. On terabyte-scale
data, NumS achieves competitive performance to SLATE on DGEMM, up to 20x
speedup over Dask on a key operation for tensor factorization, and a 2x speedup
on logistic regression compared to Dask ML and Spark's MLlib.
