  While VideoQA Transformer models demonstrate competitive performance on
standard benchmarks, the reasons behind their success are not fully understood.
Do these models jointly capture and leverage the rich multimodal structures and
dynamics from video and text? Or are they merely exploiting shortcuts to
achieve high scores? Hence, we design $\textit{QUAG}$ (QUadrant AveraGe), a
lightweight and non-parametric probe, to critically analyze multimodal
representations. QUAG facilitates combined dataset-model study by systematic
ablation of model's coupled multimodal understanding during inference.
Surprisingly, it demonstrates that the models manage to maintain high
performance even under multimodal impairment. We extend QUAG to design
"QUAG-attention", a simplistic and less-expressive replacement of
self-attention. We find that the models with QUAG-attention achieve similar
performance with significantly less mulops without any finetuning. These
findings indicate that the current VideoQA benchmarks and metrics do not
penalize models that find shortcuts and discount joint multimodal
understanding. Motivated by this, we propose the $\textit{CLAVI}$
(Counterfactual in LAnguage and VIdeo), a diagnostic dataset for coupled
multimodal understanding in VideoQA. CLAVI consists of temporal questions and
videos that are augmented to curate balanced counterfactuals in language and
video domains. We evaluate models on CLAVI and find that all models achieve
high performance on multimodal shortcut instances, but most of them have poor
performance on the counterfactual instances that necessitate joint multimodal
understanding. Overall, with the multimodal representation analysis using QUAG
and diagnostic analysis using CLAVI, we show that many VideoQA models are
incapable of learning multimodal representations and that their success on
standard datasets is an illusion of joint multimodal understanding.
