  The Vision Transformer (ViT) architecture has established its place in
computer vision literature, however, training ViTs for RGB-D object recognition
remains an understudied topic, viewed in recent literature only through the
lens of multi-task pretraining in multiple vision modalities. Such approaches
are often computationally intensive, relying on the scale of multiple
pretraining datasets to align RGB with 3D information. In this work, we propose
a simple yet strong recipe for transferring pretrained ViTs in RGB-D domains
for 3D object recognition, focusing on fusing RGB and depth representations
encoded jointly by the ViT. Compared to previous works in multimodal
Transformers, the key challenge here is to use the attested flexibility of ViTs
to capture cross-modal interactions at the downstream and not the pretraining
stage. We explore which depth representation is better in terms of resulting
accuracy and compare early and late fusion techniques for aligning the RGB and
depth modalities within the ViT architecture. Experimental results in the
Washington RGB-D Objects dataset (ROD) demonstrate that in such RGB -> RGB-D
scenarios, late fusion techniques work better than most popularly employed
early fusion. With our transfer baseline, fusion ViTs score up to 95.4% top-1
accuracy in ROD, achieving new state-of-the-art results in this benchmark. We
further show the benefits of using our multimodal fusion baseline over unimodal
feature extractors in a synthetic-to-real visual adaptation as well as in an
open-ended lifelong learning scenario in the ROD benchmark, where our model
outperforms previous works by a margin of >8%. Finally, we integrate our method
with a robot framework and demonstrate how it can serve as a perception utility
in an interactive robot learning scenario, both in simulation and with a real
robot.
