  Representational distinctions within categories are important in all
perceptual modalities and also in cognitive and motor representations. Recent
pattern-information studies of brain activity have used condition-rich designs
to sample the stimulus space more densely. To test whether brain response
patterns discriminate among a set of stimuli (e.g. exemplars within a category)
with good sensitivity, we can pool statistical evidence over all pairwise
comparisons. Here we describe a wide range of statistical tests of exemplar
discriminability and assess the validity (specificity) and power (sensitivity)
of each test. The tests include previously used and novel, parametric and
nonparametric tests, which treat subject as a random or fixed effect, and are
based on different dissimilarity measures, different test statistics, and
different inference procedures. We use simulated and real data to determine
which tests are valid and which are most sensitive. A popular test statistic
reflecting exemplar information is the exemplar discriminability index (EDI),
which is defined as the average of the pattern dissimilarity estimates between
different exemplars minus the average of the pattern dissimilarity estimates
between repetitions of identical exemplars. The popular across-subject t test
of the EDI (typically using correlation distance as the pattern dissimilarity
measure) requires the assumption that the EDI is 0-mean normal under H0.
Although this assumption is not strictly true, our simulations suggest that the
test controls the false-positives rate at the nominal level, and is thus valid,
in practice. However, test statistics based on average Mahalanobis distances or
average linear-discriminant t values (both accounting for the multivariate
error covariance among responses) are substantially more powerful for both
random- and fixed-effects inference.
