  Fluctuations and disorder effects are substantially enhanced in reduced
dimensionalities. While they are mostly considered as the foe for long-range
orders, fluctuations and disorders can also stimulate the emergence of novel
phases of matter, for example, vestigial orders. Taking 2D magnetism as a
platform, existing efforts have been focused on maintaining 2D long-range
magnetic orders by suppressing fluctuations, whereas the other side, exploiting
fluctuations for realizing new 2D magnetic phases, remains as an uncharted
territory. Here, using a combination of NV spin relaxometry, optical
spectroscopy, and Monte Carlo simulations, we report, in an XY-type honeycomb
magnet NiPS3, the phase transition from the zigzag AFM order in 3D bulk to a
new Z3 vestigial Potts-nematicity in 2D few layers. Spin fluctuations are shown
to significantly enhance over the GHz-THz range as the layer number of NiPS3
reduces, using the NV spin relaxometry and the optical Raman quasi-elastic
scattering. As a result, the Raman signatures of the zigzag AFM for bulk NiPS3,
a zone-folded phonon at ~30cm-1 from the broken translational symmetry (PBTS)
and a degeneracy lift of two phonons at ~180cm-1 for the broken 3-fold
rotational symmetry (PBRS), evolve into the disappearance of PBTS and the
survival of PBRS in few-layer NiPS3, with a critical thickness of ~10nm. The
optical linear dichroism microscopy images all three nematic domain states in a
single few-layer NiPS3 flake. The large-scale Monte Carlo simulations for
bilayer NiPS3 model confirms the absence of long-range zigzag AFM order but the
formation of the Z3 vestigial Potts-nematic phase, corroborating with the
experimental finding. Our results demonstrate the positivity of strong
fluctuations in creating new phases of matter after destroying more
conventional ones, and offer an unprecedented pathway for developing novel 2D
phases.
