  Recognizing the actions of others from visual stimuli is a crucial aspect of
human visual perception that allows individuals to respond to social cues.
Humans are able to identify similar behaviors and discriminate between distinct
actions despite transformations, like changes in viewpoint or actor, that
substantially alter the visual appearance of a scene. This ability to
generalize across complex transformations is a hallmark of human visual
intelligence. Advances in understanding motion perception at the neural level
have not always translated in precise accounts of the computational principles
underlying what representation our visual cortex evolved or learned to compute.
Here we test the hypothesis that invariant action discrimination might fill
this gap. Recently, the study of artificial systems for static object
perception has produced models, CNNs, that achieve human level performance in
complex discriminative tasks. Within this class of models, architectures that
better support invariant object recognition also produce image representations
that match those implied by human and primate neural data. However, whether
these models produce representations of action sequences that support
recognition across complex transformations and closely follow neural
representations remains unknown. Here we show that spatiotemporal CNNs
appropriately categorize video stimuli into actions, and that deliberate model
modifications that improve performance on an invariant action recognition task
lead to data representations that better match human neural recordings. Our
results support our hypothesis that performance on invariant discrimination
dictates the neural representations of actions computed by human visual cortex.
