  Groups of eukaryotic cilia and flagella are capable of coordinating their
beating over large scales, routinely exhibiting collective dynamics in the form
of metachronal waves. The origin of this behaviour -- possibly influenced by
both mechanical interactions and direct biological regulation -- is poorly
understood, in large part due to lack of quantitative experimental studies.
Here we characterise in detail flagellar coordination on the surface of the
multicellular alga $Volvox~carteri$, an emerging model organism for flagellar
dynamics. Our studies reveal for the first time that the average metachronal
coordination observed is punctuated by periodic phase defects during which
synchrony is partial and limited to specific groups of cells. A minimal model
of hydrodynamically coupled oscillators can reproduce semi-quantitatively the
characteristics of the average metachronal dynamics, and the emergence of
defects. We systematically study the model's behaviour by assessing the effect
of changing intrinsic rotor characteristics, including oscillator stiffness and
the nature of their internal driving force, as well as their geometric
properties and spatial arrangement. Our results suggest that metachronal
coordination follows from deformations in the oscillators' limit cycles induced
by hydrodynamic stresses, and that defects result from sufficiently steep local
biases in the oscillators' intrinsic frequencies. Additionally, we find that
random variations in the intrinsic rotor frequencies increase the robustness of
the average properties of the emergent metachronal waves.
