Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: MutualFormer: Multi-Modality Representation Learning via Mutual
  Transformer   Aggregating multi-modality data to obtain accurate and reliable data
representation attracts more and more attention. The pristine researchers
generally adopt the CNN to extract features of independent modality and
aggregate them with a fusion module. However, the overall performance is
becoming saturated due to limited local convolutional features. Recent studies
demonstrate that Transformer models usually work comparable or even better than
CNN for multi-modality task, but they simply adopt concatenation or
cross-attention for feature fusion which may just obtain sub-optimal results.
In this work, we re-thinking the self-attention based Transformer and propose a
novel MutualFormer for multi-modality data fusion and representation. The core
of MutualFormer is the design of both token mixer and modality mixer to conduct
the communication among both tokens and modalities. Specifically, it contains
three main modules, i.e., i) Self-attention (SA) for intra-modality token
mixer, ii) Cross-diffusion attention (CDA) for inter-modality mixer and iii)
Aggregation module. The main advantage of the proposed CDA is that it is
defined based on individual domain similarities in the metric space which thus
can naturally avoid the issue of domain/modality gap in cross-modality
similarities computation. We successfully apply the MutualFormer to the
saliency detection problem and propose a novel approach to obtain the
reinforced features of RGB and Depth images. Extensive experiments on six
popular datasets demonstrate that our model achieves comparable results with 16
SOTA models.

14
The categories of this paper are: math-ph
11
The categories of this paper are: hep-lat
26
The categories of this paper are: bayes-an
3
The categories of this paper are: physics
19
The categories of this paper are: eess
29
The categories of this paper are: funct-an
13
The categories of this paper are: quant-ph
35
The categories of this paper are: mtrl-th
33
The categories of this paper are: plasm-ph
25
The categories of this paper are: patt-sol
15
The categories of this paper are: nucl-ex
16
The categories of this paper are: stat
21
The categories of this paper are: adap-org
6
The categories of this paper are: astro-ph
31
The categories of this paper are: ao-sci
0
The categories of this paper are: hep-ph