Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Activate or Not: Learning Customized Activation   We present a simple, effective, and general activation function we term ACON
which learns to activate the neurons or not. Interestingly, we find Swish, the
recent popular NAS-searched activation, can be interpreted as a smooth
approximation to ReLU. Intuitively, in the same way, we approximate the more
general Maxout family to our novel ACON family, which remarkably improves the
performance and makes Swish a special case of ACON. Next, we present meta-ACON,
which explicitly learns to optimize the parameter switching between non-linear
(activate) and linear (inactivate) and provides a new design space. By simply
changing the activation function, we show its effectiveness on both small
models and highly optimized large models (e.g. it improves the ImageNet top-1
accuracy rate by 6.7% and 1.8% on MobileNet-0.25 and ResNet-152, respectively).
Moreover, our novel ACON can be naturally transferred to object detection and
semantic segmentation, showing that ACON is an effective alternative in a
variety of tasks. Code is available at https://github.com/nmaac/acon.

25
The categories of this paper are: patt-sol
24
The categories of this paper are: dg-ga
30
The categories of this paper are: q-alg
26
The categories of this paper are: bayes-an
17
The categories of this paper are: q-fin
15
The categories of this paper are: nucl-ex
14
The categories of this paper are: math-ph
5
The categories of this paper are: gr-qc
13
The categories of this paper are: quant-ph
34
The categories of this paper are: chem-ph
3
The categories of this paper are: physics
20
The categories of this paper are: acc-phys
6
The categories of this paper are: astro-ph
7
The categories of this paper are: hep-th
35
The categories of this paper are: mtrl-th
2
The categories of this paper are: math