Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Efficient Attention Network: Accelerate Attention by Searching Where to
  Plug   Recently, many plug-and-play self-attention modules are proposed to enhance
the model generalization by exploiting the internal information of deep
convolutional neural networks (CNNs). Previous works lay an emphasis on the
design of attention module for specific functionality, e.g., light-weighted or
task-oriented attention. However, they ignore the importance of where to plug
in the attention module since they connect the modules individually with each
block of the entire CNN backbone for granted, leading to incremental
computational cost and number of parameters with the growth of network depth.
Thus, we propose a framework called Efficient Attention Network (EAN) to
improve the efficiency for the existing attention modules. In EAN, we leverage
the sharing mechanism (Huang et al. 2020) to share the attention module within
the backbone and search where to connect the shared attention module via
reinforcement learning. Finally, we obtain the attention network with sparse
connections between the backbone and modules, while (1) maintaining accuracy
(2) reducing extra parameter increment and (3) accelerating inference.
Extensive experiments on widely-used benchmarks and popular attention networks
show the effectiveness of EAN. Furthermore, we empirically illustrate that our
EAN has the capacity of transferring to other tasks and capturing the
informative features. The code is available at
https://github.com/gbup-group/EAN-efficient-attention-network.

33
The categories of this paper are: plasm-ph
9
The categories of this paper are: nlin
16
The categories of this paper are: stat
35
The categories of this paper are: mtrl-th
3
The categories of this paper are: physics
12
The categories of this paper are: nucl-th
14
The categories of this paper are: math-ph
27
The categories of this paper are: comp-gas
11
The categories of this paper are: hep-lat
37
The categories of this paper are: supr-con
2
The categories of this paper are: math
5
The categories of this paper are: gr-qc
7
The categories of this paper are: hep-th
22
The categories of this paper are: chao-dyn
20
The categories of this paper are: acc-phys
8
The categories of this paper are: hep-ex