  Pretrained Foundation Models (PFMs) are regarded as the foundation for
various downstream tasks with different data modalities. A PFM (e.g., BERT,
ChatGPT, and GPT-4) is trained on large-scale data which provides a reasonable
parameter initialization for a wide range of downstream applications. BERT
learns bidirectional encoder representations from Transformers, which are
trained on large datasets as contextual language models. Similarly, the
generative pretrained transformer (GPT) method employs Transformers as the
feature extractor and is trained using an autoregressive paradigm on large
datasets. Recently, ChatGPT shows promising success on large language models,
which applies an autoregressive language model with zero shot or few shot
prompting. The remarkable achievements of PFM have brought significant
breakthroughs to various fields of AI. Numerous studies have proposed different
methods, raising the demand for an updated survey. This study provides a
comprehensive review of recent research advancements, challenges, and
opportunities for PFMs in text, image, graph, as well as other data modalities.
The review covers the basic components and existing pretraining methods used in
natural language processing, computer vision, and graph learning. Additionally,
it explores advanced PFMs used for different data modalities and unified PFMs
that consider data quality and quantity. The review also discusses research
related to the fundamentals of PFMs, such as model efficiency and compression,
security, and privacy. Finally, the study provides key implications, future
research directions, challenges, and open problems in the field of PFMs.
Overall, this survey aims to shed light on the research of the PFMs on
scalability, security, logical reasoning ability, cross-domain learning
ability, and the user-friendly interactive ability for artificial general
intelligence.
