  The simulation of human neurons and neurotransmission mechanisms has been
realized in deep neural networks based on the theoretical implementations of
activation functions. However, recent studies have reported that the threshold
potential of neurons exhibits different values according to the locations and
types of individual neurons, and that the activation functions have limitations
in terms of representing this variability. Therefore, this study proposes a
simple yet effective activation function that facilitates different thresholds
and adaptive activations according to the positions of units and the contexts
of inputs. Furthermore, the proposed activation function mathematically
exhibits a more generalized form of Swish activation function, and thus we
denoted it as Adaptive SwisH (ASH). ASH highlights informative features that
exhibit large values in the top percentiles in an input, whereas it rectifies
low values. Most importantly, ASH exhibits trainable, adaptive, and
context-aware properties compared to other activation functions. Furthermore,
ASH represents general formula of the previously studied activation function
and provides a reasonable mathematical background for the superior performance.
To validate the effectiveness and robustness of ASH, we implemented ASH into
many deep learning models for various tasks, including classification,
detection, segmentation, and image generation. Experimental analysis
demonstrates that our activation function can provide the benefits of more
accurate prediction and earlier convergence in many deep learning applications.
