  This paper proposes two distributed random reshuffling methods, namely
Gradient Tracking with Random Reshuffling (GT-RR) and Exact Diffusion with
Random Reshuffling (ED-RR), to solve the distributed optimization problem over
a connected network, where a set of agents aim to minimize the average of their
local cost functions. Both algorithms invoke random reshuffling (RR) update for
each agent, inherit favorable characteristics of RR for minimizing smooth
nonconvex objective functions, and improve the performance of previous
distributed random reshuffling methods both theoretically and empirically.
Specifically, both GT-RR and ED-RR achieve the convergence rate of
$O(1/[(1-\lambda)^{1/3}m^{1/3}T^{2/3}])$ in driving the (minimum) expected
squared norm of the gradient to zero, where $T$ denotes the number of epochs,
$m$ is the sample size for each agent, and $1-\lambda$ represents the spectral
gap of the mixing matrix. When the objective functions further satisfy the
Polyak-{\L}ojasiewicz (PL) condition, we show GT-RR and ED-RR both achieve
$O(1/[(1-\lambda)mT^2])$ convergence rate in terms of the averaged expected
differences between the agents' function values and the global minimum value.
Notably, both results are comparable to the convergence rates of centralized RR
methods (up to constant factors depending on the network topology) and
outperform those of previous distributed random reshuffling algorithms.
Moreover, we support the theoretical findings with a set of numerical
experiments.
