  Vision transformer (ViT) and its variants have achieved remarkable successes
in various visual tasks. The key characteristic of these ViT models is to adopt
different aggregation strategies of spatial patch information within the
artificial neural networks (ANNs). However, there is still a key lack of
unified representation of different ViT architectures for systematic
understanding and assessment of model representation performance. Moreover, how
those well-performing ViT ANNs are similar to real biological neural networks
(BNNs) is largely unexplored. To answer these fundamental questions, we, for
the first time, propose a unified and biologically-plausible relational graph
representation of ViT models. Specifically, the proposed relational graph
representation consists of two key sub-graphs: aggregation graph and affine
graph. The former one considers ViT tokens as nodes and describes their spatial
interaction, while the latter one regards network channels as nodes and
reflects the information communication between channels. Using this unified
relational graph representation, we found that: a) a sweet spot of the
aggregation graph leads to ViTs with significantly improved predictive
performance; b) the graph measures of clustering coefficient and average path
length are two effective indicators of model prediction performance, especially
when applying on the datasets with small samples; c) our findings are
consistent across various ViT architectures and multiple datasets; d) the
proposed relational graph representation of ViT has high similarity with real
BNNs derived from brain science data. Overall, our work provides a novel
unified and biologically-plausible paradigm for more interpretable and
effective representation of ViT ANNs.
