  Topological edge states form at the edges of periodic materials with specific
degeneracies in their modal spectra, such as Dirac points, under the action of
effects breaking certain symmetries of the system. In particular, in Floquet
topological insulators unidirectional edge states appear upon breakup of the
effective time-reversal symmetry due to dynamical modulations of the underlying
lattice potential. However, such states are usually reported for certain simple
lattice terminations, for example, at zigzag or bearded edges in honeycomb
lattices. Here we show that unconventional topological edge states may exist in
Floquet insulators based on arrays of helical waveguides with hybrid edges
involving alternating zigzag and armchair segments, even if the latter are
long. Such edge states appear in the largest part of the first Brillouin zone
and show topological protection upon passage through the defects. Topological
states at hybrid edges persist in the presence of focusing nonlinearity of the
material. Our results can be extended to other lattice types and physical
systems, they lift some of the constraints connected with lattice terminations
that may not support edge states in the absence of effects breaking
time-reversal symmetry of the system and expand the variety of geometrical
shapes in which topological insulators can be constructed.
