  Background: Building visual encoding models to accurately predict visual
responses is a central challenge for current vision-based brain-machine
interface techniques. To achieve high prediction accuracy on neural signals,
visual encoding models should include precise visual features and appropriate
prediction algorithms. Most existing visual encoding models employ hand-craft
visual features (e.g., Gabor wavelets or semantic labels) or data-driven
features (e.g., features extracted from deep neural networks (DNN)). They also
assume a linear mapping between feature representation to brain activity.
However, it remains unknown whether such linear mapping is sufficient for
maximizing prediction accuracy. New Method: We construct a new visual encoding
framework to predict cortical responses in a benchmark functional magnetic
resonance imaging (fMRI) dataset. In this framework, we employ the transfer
learning technique to incorporate a pre-trained DNN (i.e., AlexNet) and train a
nonlinear mapping from visual features to brain activity. This nonlinear
mapping replaces the conventional linear mapping and is supposed to improve
prediction accuracy on brain activity. Results: The proposed framework can
significantly predict responses of over 20% voxels in early visual areas (i.e.,
V1-lateral occipital region, LO) and achieve unprecedented prediction accuracy.
Comparison with Existing Methods: Comparing to two conventional visual encoding
models, we find that the proposed encoding model shows consistent higher
prediction accuracy in all early visual areas, especially in relatively
anterior visual areas (i.e., V4 and LO). Conclusions: Our work proposes a new
framework to utilize pre-trained visual features and train non-linear mappings
from visual features to brain activity.
