  MetaFormer, the abstracted architecture of Transformer, has been found to
play a significant role in achieving competitive performance. In this paper, we
further explore the capacity of MetaFormer, again, without focusing on token
mixer design: we introduce several baseline models under MetaFormer using the
most basic or common mixers, and summarize our observations as follows: (1)
MetaFormer ensures solid lower bound of performance. By merely adopting
identity mapping as the token mixer, the MetaFormer model, termed
IdentityFormer, achieves >80% accuracy on ImageNet-1K. (2) MetaFormer works
well with arbitrary token mixers. When specifying the token mixer as even a
random matrix to mix tokens, the resulting model RandFormer yields an accuracy
of >81%, outperforming IdentityFormer. Rest assured of MetaFormer's results
when new token mixers are adopted. (3) MetaFormer effortlessly offers
state-of-the-art results. With just conventional token mixers dated back five
years ago, the models instantiated from MetaFormer already beat state of the
art. (a) ConvFormer outperforms ConvNeXt. Taking the common depthwise separable
convolutions as the token mixer, the model termed ConvFormer, which can be
regarded as pure CNNs, outperforms the strong CNN model ConvNeXt. (b) CAFormer
sets new record on ImageNet-1K. By simply applying depthwise separable
convolutions as token mixer in the bottom stages and vanilla self-attention in
the top stages, the resulting model CAFormer sets a new record on ImageNet-1K:
it achieves an accuracy of 85.5% at 224x224 resolution, under normal supervised
training without external data or distillation. In our expedition to probe
MetaFormer, we also find that a new activation, StarReLU, reduces 71% FLOPs of
activation compared with GELU yet achieves better performance. We expect
StarReLU to find great potential in MetaFormer-like models alongside other
neural networks.
