  The state sums defining the quantum hyperbolic invariants (QHI) of hyperbolic
oriented cusped $3$-manifolds can be split in a "symmetrization" factor and a
"reduced" state sum. We show that these factors are invariants on their own,
that we call "symmetry defects" and "reduced QHI", provided the manifolds are
endowed with an additional "non ambiguous structure", a new type of
combinatorial structure that we introduce in this paper. A suitably normalized
version of the symmetry defects applies to compact $3$-manifolds endowed with
$PSL_2(\mathbb{C})$-characters, beyond the case of cusped manifolds. Given a
manifold $M$ with non empty boundary, we provide a partial "holographic"
description of the non-ambiguous structures in terms of the intrinsic geometric
topology of $\partial M$. Special instances of non ambiguous structures can be
defined by means of taut triangulations, and the symmetry defects have a
particularly nice behaviour on such "taut structures". Natural examples of taut
structures are carried by any mapping torus with punctured fibre of negative
Euler characteristic, or by sutured manifold hierarchies. For a cusped
hyperbolic $3$-manifold $M$ which fibres over $S^1$, we address the question of
determining whether the fibrations over a same fibered face of the Thurston
ball define the same taut structure. We describe a few examples in detail. In
particular, they show that the symmetry defects or the reduced QHI can
distinguish taut structures associated to different fibrations of $M$. To
support the guess that all this is an instance of a general behaviour of state
sum invariants of 3-manifolds based on some theory of 6j-symbols, finally we
describe similar results about reduced Turaev-Viro invariants.
