Instruct: Identify the main category of Biorxiv papers based on the titles and abstracts
Query: The role of capacity constraints in Convolutional Neural Networks for learning random versus natural data Convolutional neural networks (CNNs) are often described as promising models of human vision, yet they show many differences from human abilities. We focus on a superhuman capacity of top-performing CNNs, namely, their ability to learn very large datasets of random patterns. We verify that human learning on such tasks is extremely limited, even with few stimuli. We argue that the performance difference is due to CNNs overcapacity and introduce biologically inspired mechanisms to constrain it, while retaining the good test set generalisation to structured images as characteristic of CNNs. We investigate the efficacy of adding noise to hidden units activations, restricting early convolutional layers with an information bottleneck, and using a bounded activation function. Internal noise was the most potent intervention and the only one which, by itself, could reduce random data performance in the tested models to chance levels. We also investigated whether networks with biologically inspired capacity constraints show improved generalisation to out-of-distribution stimuli, however little benefit was observed. Our results suggest that constraining networks with biologically motivated mechanisms paves the way for closer correspondence between network and human performance, but the few manipulations we have tested are only a small step towards that goal.
15
The categories of this paper are: developmental
7
The categories of this paper are: ecology
27
The categories of this paper are: pathology
3
The categories of this paper are: evolutionary
5
The categories of this paper are: cell
8
The categories of this paper are: genomics
22
The categories of this paper are: physiology
17
The categories of this paper are: molecular
19
The categories of this paper are: epidemiology
20
The categories of this paper are: pharmacology
6
The categories of this paper are: plant
25
The categories of this paper are: bioengineering
14
The categories of this paper are: biophysics
12
The categories of this paper are: behavior
11
The categories of this paper are: cognition
32
The categories of this paper are: zoology