  We study a system of particles in two dimensions interacting via a dipolar
long-range potential $D/r^3$ and subject to a square-lattice substrate
potential $V({\bf r})$ with amplitude $V$ and lattice constant $b$. The
isotropic interaction favors a hexagonal arrangement of the particles with
lattice constant $a$, which competes against the square symmetry of the
substrate lattice. We determine the minimal-energy states at fixed external
pressure $p$ generating the commensurate density $n = 1/b^2 = (4/3)^{1/2}/a^2$
in the absence of thermal and quantum fluctuations, using both analytical and
numerical techniques. At large substrate amplitude $V > 0.2\, e_D$, with $e_D =
D/b^3$ the dipolar energy scale, the particles reside in the substrate minima
and hence arrange in a square lattice. Upon decreasing $V$, the square lattice
turns unstable with respect to a zone-boundary shear-mode and deforms into a
period-doubled zig-zag lattice. Analytic and numerical results show that this
period-doubled phase in turn becomes unstable at $V \approx 0.074\, e_D$
towards a non-uniform phase developing an array of domain walls or solitons; as
the density of solitons increases, the particle arrangement approaches that of
a rhombic (or isosceles triangular) lattice. At a yet smaller substrate value
estimated as $V \approx 0.046\, e_D$, a further solitonic transition
establishes a second non-uniform phase which smoothly approaches the hexagonal
(or equilateral triangular) lattice phase with vanishing amplitude $V$. At
small but finite amplitude $V$, the hexagonal phase is distorted and
hexatically locked at an angle of $\varphi \approx 3.8^\circ$ with respect to
the substrate lattice. The square-to-hexagonal transformation in this
two-dimensional commensurate-incommensurate system thus involves a complex
pathway with various non-trivial lattice- and modulated phases.
