  Feature visualizations such as synthetic maximally activating images are a
widely used explanation method to better understand the information processing
of convolutional neural networks (CNNs). At the same time, there are concerns
that these visualizations might not accurately represent CNNs' inner workings.
Here, we measure how much extremely activating images help humans to predict
CNN activations. Using a well-controlled psychophysical paradigm, we compare
the informativeness of synthetic images by Olah et al. (2017) with a simple
baseline visualization, namely exemplary natural images that also strongly
activate a specific feature map. Given either synthetic or natural reference
images, human participants choose which of two query images leads to strong
positive activation. The experiments are designed to maximize participants'
performance, and are the first to probe intermediate instead of final layer
representations. We find that synthetic images indeed provide helpful
information about feature map activations ($82\pm4\%$ accuracy; chance would be
$50\%$). However, natural images - originally intended as a baseline -
outperform synthetic images by a wide margin ($92\pm2\%$). Additionally,
participants are faster and more confident for natural images, whereas
subjective impressions about the interpretability of the feature visualizations
are mixed. The higher informativeness of natural images holds across most
layers, for both expert and lay participants as well as for hand- and
randomly-picked feature visualizations. Even if only a single reference image
is given, synthetic images provide less information than natural images
($65\pm5\%$ vs. $73\pm4\%$). In summary, synthetic images from a popular
feature visualization method are significantly less informative for assessing
CNN activations than natural images. We argue that visualization methods should
improve over this baseline.
