  Natural human interactions for Mixed Reality Applications are overwhelmingly
multimodal: humans communicate intent and instructions via a combination of
visual, aural and gestural cues. However, supporting low-latency and accurate
comprehension of such multimodal instructions (MMI), on resource-constrained
wearable devices, remains an open challenge, especially as the state-of-the-art
comprehension techniques for each individual modality increasingly utilize
complex Deep Neural Network models. We demonstrate the possibility of
overcoming the core limitation of latency--vs.--accuracy tradeoff by exploiting
cross-modal dependencies -- i.e., by compensating for the inferior performance
of one model with an increased accuracy of more complex model of a different
modality. We present a sensor fusion architecture that performs MMI
comprehension in a quasi-synchronous fashion, by fusing visual, speech and
gestural input. The architecture is reconfigurable and supports dynamic
modification of the complexity of the data processing pipeline for each
individual modality in response to contextual changes. Using a representative
"classroom" context and a set of four common interaction primitives, we then
demonstrate how the choices between low and high complexity models for each
individual modality are coupled. In particular, we show that (a) a judicious
combination of low and high complexity models across modalities can offer a
dramatic 3-fold decrease in comprehension latency together with an increase
10-15% in accuracy, and (b) the right collective choice of models is context
dependent, with the performance of some model combinations being significantly
more sensitive to changes in scene context or choice of interaction.
