  We demonstrate the presence of an optical phase transition with
frustration-induced spontaneous symmetry breaking in a triangular planar atomic
array due to cooperative light-mediated interactions. We show how the array
geometry of triangle unit cells at low light intensities leads to degenerate
collective radiative excitations forming nearly flat bands. We drive degenerate
pairs of collective excitations to be equally populated in both cases of the
atomic polarization in the lattice plane and perpendicular to it. At higher
intensities, above specific threshold values, this symmetry in the populations
is spontaneously broken. We also develop an effective few-mode model that
provides semianalytic descriptions of the symmetry-breaking threshold and
infinite-lattice limit phase transition. Surprisingly, we find how excitations
due to dipolar interactions correspond to optical analogs of those found in
frustrated magnets and superfluids, with closely related symmetry-breaking
mechanisms despite the significant physical differences between these systems,
opening potential for simulating even quantum magnetism. Transmitted light
through the array conveys information about symmetry breaking in the hysteresis
behavior of the spectrum. Moreover, in a Mott-insulator state, the atomic
positions are subject to zero-point quantum fluctuations. Interpreting each
stochastic realization as a light-induced quantum measurement of the atomic
position configuration, we find how strong nonlinearities and even weak
position uncertainties lead to considerable measurement-induced symmetry
breaking, while ensemble-averaging over many realizations restores the original
symmetry and the unbroken state. Larger position uncertainty results in the
formation of domains of different broken symmetries.
