  A modular invariant for a chiral conformal field theory is physical if there
is a full conformal field theory with the given chiral halves realising the
modular invariant. The easiest modular invariants are the charge conjugation
and the diagonal modular invariants. While the charge conjugation modular
invariant is always physical there are examples of chiral CFTs for which the
diagonal modular invariant is not physical. Here we give (in group theoretical
terms) a necessary and sufficient condition for diagonal modular invariants of
$G$-orbifolds of holomorphic conformal field theories to be physical.
  Mathematically a physical modular invariant is an invariant of a Lagrangian
algebra in the product of (chiral) modular categories. The chiral modular
category of a $G$-orbifold of a holomorphic conformal field theory is the
so-called (twisted) Drinfeld centre ${\cal Z}(G,\alpha)$ of the finite group
$G$. We show that the diagonal modular invariant for ${\cal Z}(G)$ is physical
if and only if the group $G$ has a {\em double class inverting} automorphism,
that is an automorphism $\phi:G\to G$ with the property that for any commuting
$x,y\in G$ there is $g\in G$ such that $\phi(x) = gx^{-1}g^{-1},\ \phi(y) =
gy^{-1}g^{-1}.$
  Groups without double class inverting automorphisms are abundant and provide
examples of chiral conformal field theories for which the diagonal modular
invariant is unphysical.
