er-summary">Effectively handling temporal redundancy remains a key challenge in learning
video models. Prevailing approaches often treat each set of frames
independently, failing to effectively capture the temporal dependencies and
redundancies inherent in videos. To address this limitation, we introduce
RefTok, a novel reference-based tokenization method capable of capturing
complex temporal dynamics and contextual information. Our method encodes and
decodes sets of frames conditioned on an unquantized ref