  Fix primes $p$ and $\ell$ with $\ell\neq p$. If $(A,\lambda)$ is a
$g$-dimensional principally polarized abelian variety, an $(\ell)^g$-isogeny of
$(A,\lambda)$ has kernel a maximal isotropic subgroup of the $\ell$-torsion of
$A$; the image has a natural principal polarization. We define three isogeny
graphs associated to such $(\ell)^g$-isogenies -- the big isogeny graph
$\mathit{Gr}_{\!g}(\ell,p)$, the little isogeny graph
$\mathit{gr}_{\!g}(\ell,p)$, and the enhanced isogeny graph
$\widetilde{\mathit{gr}}_{\!g}(\ell, p)$. We prove that all three isogeny
graphs are connected. One ingredient of the proof is strong approximation for
the quaternionic unitary group, which has previously been applied to moduli of
abelian varieties in charateristic $p$ by Chai, Ekedahl/Oort, and Chai/Oort.
The adjacency matrices of the three isogeny graphs are given in terms of the
Brandt matrices defined by Hashimoto, Ibukiyama, Ihara, and Shimizu. We study
some basic properties of these Brandt matrices and recast the theory using the
notion of Brandt graphs. We show that the isogeny graphs
$\mathit{Gr}_{\!g}(\ell, p)$ and $\mathit{gr}_{\!g}(\ell, p)$ are in fact our
Brandt graphs. We give the $\ell$-adic uniformization of
$\mathit{gr}_{\!g}(\ell,p)$ and $\widetilde{\mathit{gr}}_{\!g}(\ell,p)$. The
$(\ell+1)$-regular isogeny graph $\mathit{Gr}_1(\ell,p)$ for supersingular
elliptic curves is well known to be Ramanujan. We calculate the Brandt matrices
for a range of $g>1$, $\ell$, and $p$. These calculations give four examples
with $g>1$ where the regular graph $\mathit{Gr}_{\!g}(\ell,p)$ has two vertices
and is Ramanujan, and all other examples we computed with $g>1$ and two or more
vertices were not Ramanujan. In particular, the $(\ell)^g$-isogeny graph is not
in general Ramanujan for $g>1$.
