Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: A Comprehensive Survey and Performance Analysis of Activation Functions
  in Deep Learning   Neural networks have shown tremendous growth in recent years to solve
numerous problems. Various types of neural networks have been introduced to
deal with different types of problems. However, the main goal of any neural
network is to transform the non-linearly separable input data into more
linearly separable abstract features using a hierarchy of layers. These layers
are combinations of linear and nonlinear functions. The most popular and common
non-linearity layers are activation functions (AFs), such as Logistic Sigmoid,
Tanh, ReLU, ELU, Swish and Mish. In this paper, a comprehensive overview and
survey is presented for AFs in neural networks for deep learning. Different
classes of AFs such as Logistic Sigmoid and Tanh based, ReLU based, ELU based,
and Learning based are covered. Several characteristics of AFs such as output
range, monotonicity, and smoothness are also pointed out. A performance
comparison is also performed among 18 state-of-the-art AFs with different
networks on different types of data. The insights of AFs are presented to
benefit the researchers for doing further research and practitioners to select
among different choices. The code used for experimental comparison is released
at: \url{https://github.com/shivram1987/ActivationFunctions}.

31
The categories of this paper are: ao-sci
15
The categories of this paper are: nucl-ex
26
The categories of this paper are: bayes-an
10
The categories of this paper are: q-bio
9
The categories of this paper are: nlin
4
The categories of this paper are: cond-mat
13
The categories of this paper are: quant-ph
17
The categories of this paper are: q-fin
25
The categories of this paper are: patt-sol
30
The categories of this paper are: q-alg
28
The categories of this paper are: alg-geom
37
The categories of this paper are: supr-con
32
The categories of this paper are: atom-ph
5
The categories of this paper are: gr-qc
33
The categories of this paper are: plasm-ph
36
The categories of this paper are: cmp-lg