  Tensor networks have a gauge degree of freedom on the virtual degrees of
freedom that are contracted. A canonical form is a choice of fixing this degree
of freedom. For matrix product states, choosing a canonical form is a powerful
tool, both for theoretical and numerical purposes. On the other hand, for
tensor networks in dimension two or greater there is only limited understanding
of the gauge symmetry. Here we introduce a new canonical form, the minimal
canonical form, which applies to projected entangled pair states (PEPS) in any
dimension, and prove a corresponding fundamental theorem. Already for matrix
product states this gives a new canonical form, while in higher dimensions it
is the first rigorous definition of a canonical form valid for any choice of
tensor. We show that two tensors have the same minimal canonical forms if and
only if they are gauge equivalent up to taking limits; moreover, this is the
case if and only if they give the same quantum state for any geometry. In
particular, this implies that the latter problem is decidable - in contrast to
the well-known undecidability for PEPS on grids. We also provide rigorous
algorithms for computing minimal canonical forms. To achieve this we draw on
geometric invariant theory and recent progress in theoretical computer science
in non-commutative group optimization.
