FinGPT: Open-Source Financial Large Language Models
Hongyang (Bruce) Yang1, Xiao-Yang Liu1, Christina Dan Wang2
1Columbia University;2New York University (Shanghai)
{HY2500, XL2427 }@columbia.edu; christina.wang@nyu.edu
Abstract
Large language models (LLMs) have shown the
potential of revolutionizing natural language pro-
cessing tasks in diverse domains, sparking great
interest in finance. Accessing high-quality fi-
nancial data is the first challenge for financial
LLMs (FinLLMs). While proprietary models
like BloombergGPT have taken advantage of their
unique data accumulation, such privileged access
calls for an open-source alternative to democratize
Internet-scale financial data.
In this paper, we present an open-source large
language model, FinGPT, for the finance sec-
tor. Unlike proprietary models, FinGPT takes a
data-centric approach, providing researchers and
practitioners with accessible and transparent re-
sources to develop their FinLLMs. We high-

light the importance of an automatic data cura-
tion pipeline and the lightweight low-rank adap-
tation technique in building FinGPT. Further-
more, we showcase several potential applica-
tions as stepping stones for users, such as robo-
advising, algorithmic trading, and low-code devel-
opment. Through collaborative efforts within the
open-source AI4Finance community, FinGPT aims
to stimulate innovation, democratize FinLLMs,
and unlock new opportunities in open finance.
Two associated code repos are https://github.
com/AI4Finance-Foundation/FinGPT and https://
github.com/AI4Finance-Foundation/FinNLP
1 Introduction
The continual expansion and evolution of artificial intel-
ligence have provided a fertile ground for the prolifera-
tion of large language models [Vaswani et al. , 2017; Rad-

ford et al. , 2018; Devlin et al. , 2018; Ethayarajh, 2019;
Lewis et al. , 2019; Lewis et al. , 2020; Brown et al. , 2020;
Thoppilan et al. , 2022 ], thereby effecting a transformative
shift in the landscape of natural language processing across
diverse domains. This sweeping change has engendered keen
interest in the potential application of these models in the fi-
nancial realm. It is, however, evident that the acquisition ofhigh-quality, relevant, and up-to-date data stands as a criti-
cal factor in the development of an efficacious and efficient
open-source financial language model.
Utilizing language models in the financial arena reveals
intricate hurdles. These range from difficulties in obtaining
data, dealing with diverse data formats and types, and man-

aging data quality inconsistencies, to the essential require-
ment of up-to-date information. Especially, historical or spe-
cialized financial data extraction proves to be complex due
to varying data mediums such as web platforms, APIs, PDF
documents, and images.
In the proprietary sphere, models like BloombergGPT [Wu
et al. , 2023 ]have capitalized on their exclusive access to spe-
cialized data to train finance-specific language models. How-
ever, the restricted accessibility and transparency of their data
collections and training protocols have accentuated the de-
mand for a more open and inclusive alternative. In response
to this demand, we are witnessing a shifting trend towards de-
mocratizing Internet-scale financial data in the open-source
domain.

domain.
In this paper, we address these aforementioned challenges
associated with financial data and introduce FinGPT, an end-
to-end open-source framework for financial large language
models (FinLLMs). Adopting a data-centric approach, Fin-
GPT underscores the crucial role of data acquisition, clean-
ing, and preprocessing in developing open-source FinLLMs.
By championing data accessibility, FinGPT aspires to en-
hance research, collaboration, and innovation in finance,
paving the way for open finance practices.
Our contributions are summarized as follows:
•Democratization : FinGPT, as an open-source framework,
aims to democratize financial data and FinLLMs, uncover-
ing untapped potentials in open finance.
•Data-centric approach : Recognizing the significance of

data curation, FinGPT adopts a data-centric approach and
implements rigorous cleaning and preprocessing methods
for handling varied data formats and types, thereby ensur-
ing high-quality data.
•End-to-end framework : FinGPT embraces a full-stack
framework for FinLLMs with four layers:
–Data source layer : This layer assures comprehensive
market coverage, addressing the temporal sensitivityarXiv:2306.06031v1  [q-fin.ST]  9 Jun 2023of financial data through real-time information cap-
ture.
–Data engineering layer : Primed for real-time NLP
data processing, this layer tackles the inherent chal-
lenges of high temporal sensitivity and low signal-to-
noise ratio in financial data.
–LLMs layer : Focusing on a range of fine-tuning
methodologies, this layer mitigates the highly dy-
namic nature of financial data, ensuring the model’s

relevance and accuracy.
–Application layer : Showcasing practical applications
and demos, this layer highlights the potential capabil-
ity of FinGPT in the financial sector.
Our vision for FinGPT is to serve as a catalyst for stimulat-
ing innovation within the finance domain. FinGPT is not lim-
ited to providing technical contributions, but it also cultivates
an open-source ecosystem for FinLLMs, promoting real-time
processing and customized adaptation for users. By nurtur-
ing a robust collaboration ecosystem within the open-source
AI4Finance community, FinGPT is positioned to reshape our
understanding and application of FinLLMs.
2 Related Work
2.1 LLMs and ChatGPT
Large Language Models (LLMs) have been recognized as
a technological breakthrough in natural language process-

ing, such as GPT-3 and GPT-4 [Brown et al. , 2020 ]. They
take transformer-based architectures, demonstrating impres-
sive performance across various generative tasks.
As an offshoot of the GPT family developed by OpenAI,
ChatGPT was designed to produce human-like text based on
input prompts. It has shown significant utility in diverse ap-
plications, from drafting emails to writing code and even in
creating written content.
2.2 LLMs in Finance
LLMs have been applied to various tasks within the financial
sector [Dredze et al. , 2016; Araci, 2019; Bao et al. , 2021;
DeLucia et al. , 2022 ], from predictive modeling to generating
insightful narratives from raw financial data. Recent literature
has focused on using these models for financial text analysis,

given the abundance of text data in this field, such as news
articles, earnings call transcripts, and social media posts.
The first example of financial LLMs is BloombergGPT
[Wuet al. , 2023 ], which was trained on a mixed dataset of
financial and general sources. Despite its impressive capabil-
ities, access limitations exist, and the prohibitive training cost
has motivated the need for low-cost domain adaptation.
Our FinGPT responds to these challenges, presenting
an open-source financial LLM. It employs Reinforcement
Learning from Human Feedback (RLHF) to understand and
adapt to individual preferences, paving the way for person-
alized financial assistants. We aim to combine the strengths
of general LLMs like ChatGPT with financial adaptation, ex-

ploiting LLM’s capability in finance.2.3 Why Open-Source FinLLMs?
AI4Finance Foundation is a non-profit, open-source organi-
zation that integrates Artificial Intelligence (AI) and finan-
cial applications, including financial Large Language Models
(FinLLMs). With a proven track record of nurturing an in-
novative ecosystem of FinTech tools, such as FinRL [Liuet
al., 2021 ]and FinRL-Meta [Liuet al. , 2022 ], the foundation
is poised to accelerate the evolution of FinLLMs further. It
is steadfast commitment and cutting-edge contributions pave
the way for AI’s transformative application in finance.
• Advancing equal opportunities via democratizing Fin-
LLMs: Adopting an open-source methodology promotes
universal access to state-of-the-art technology, adhering to
the ethos of democratizing FinLLMs.

• Cultivating transparency and trust: Open-source FinLLMs
offer a comprehensive overview of their foundational code-
base, bolstering transparency and trust.
• Accelerating research and innovation: The open-source
model fuels progress in research and development within
the AI domain. It allows researchers to leverage existing
models, thus nurturing a faster progression of innovation
and scientific discovery.
• Enhancing education: Open-source FinLLMs serve as ro-
bust educational tools, presenting students and researchers
with the prospect of exploring the complexities of Fin-
LLMs through direct engagement with fully operational
models.
• Promoting community development and collaborative en-
gagement: Open-source promotes a global community of
contributors. This collaborative participation bolsters the

models.
model’s long-term durability and effectiveness.
3 Data-Centric Approach for FinLLMs
For financial large language models (FinLLMs), a success-
ful strategy is not solely based on the capability of the model
architecture but is equally reliant on the training data. Our
data-centric approach prioritizes collecting, preparing, and
processing high-quality data.
3.1 Financial Data and Unique Characteristics
Financial data comes from a variety of sources, with unique
characteristics. We delve into the specifics of different finan-
cial data sources, such as Financial News, Company Fillings,
Social Media Discussions, and Company Announcements.
Financial news carries vital information about the world
economy, specific industries, and individual companies. This
data source typically features:

•Timeliness: Financial news reports are timely and up-to-
date, often capturing the most recent developments in the
financial world.
•Dynamism: The information contained in financial news
is dynamic, changing rapidly in response to evolving eco-
nomic conditions and market sentiment.
•Influence: Financial news has a significant impact on fi-
nancial markets, influencing traders’ decisions and poten-
tially leading to dramatic market movements.Company filings and announcements are official docu-
ments that corporations submit to regulatory bodies, provid-
ing insight into a company’s financial health and strategic di-
rection. They feature:
•Granularity : These documents offer granular information
about a company’s financial status, including assets, liabil-
ities, revenue, and profitability.

•Reliability : Company fillings contain reliable and verified
data vetted by regulatory bodies.
•Periodicity : Company fillings are periodic, usually sub-
mitted on a quarterly or annual basis, offering regular snap-
shots of a company’s financial situation.
•Impactfulness : Company announcements often have sub-
stantial impacts on the market, influencing stock prices and
investor sentiment.
Social media discussions related to finance can reflect
public sentiment towards specific stocks, sectors, or the over-
all market. These discussions tend to exhibit:
•Variability: Social media discussions vary widely in tone,
content, and quality, making them rich, albeit complex,
sources of information.
•Real-time sentiment: These platforms often capture real-
time market sentiment, enabling the detection of trends and

shifts in public opinion.
•Volatility: Sentiments expressed on social media can be
highly volatile, changing rapidly in response to news events
or market movements.
Trends , often observable through websites like Seeking
Alpha, Google Trends, and other finance-oriented blogs and
forums, offer critical insights into market movements and in-
vestment strategies. They feature:
•Analyst perspectives: These platforms provide access to
market predictions and investment advice from seasoned
financial analysts and experts.
•Market sentiment: The discourse on these platforms can
reflect the collective sentiment about specific securities,
sectors, or the overall market, providing valuable insights
into the prevailing market mood.
•Broad coverage: Trends data spans diverse securities and

market segments, offering comprehensive market coverage.
Each of these data sources provides unique insights into
the financial world. By integrating these diverse data types,
financial language models like FinGPT can facilitate a com-
prehensive understanding of financial markets and enable ef-
fective financial decision-making.
3.2 Challenges in Handling Financial Data
We summarize three major challenges for handling financial
data as follows:
•High temporal sensitivity : Financial data are character-
ized by their time-sensitive nature. Market-moving news or
updates, once released, provide a narrow window of oppor-
tunity for investors to maximize their alpha (the measure of
an investment’s relative return).•High dynamism : The financial landscape is perpetually
evolving, with a daily influx of news, social media posts,

and other market-related information. It’s impractical and
cost-prohibitive to retrain models frequently to cope with
these changes.
•Low signal-to-noise ratio (SNR) : Financial data often ex-
hibit a low signal-to-noise ratio [Liuet al. , 2022 ], meaning
that the useful information is usually dwarfed by a substan-
tial amount of irrelevant or noisy data. Extracting valuable
insights from this sea of information necessitates sophisti-
cated techniques.
Addressing these challenges is critical for the effective uti-
lization of financial data and maximizing the potential of Fin-
LLMs. As we navigate these challenges, we propose an open-
source framework FinGPT.
4 Overview of FinGPT: An Open-Source
Framework for FinLLMs
FinGPT represents an innovative open-source framework
designed specifically for applying large language models

(LLMs) within the financial domain. As delineated in Fig.
1, FinGPT consists of four fundamental components: Data
Source, Data Engineering, LLMs, and Applications. Each of
these components plays a crucial role in maintaining the func-
tionality and adaptability of FinGPT in addressing dynamic
financial data and market conditions.
•Data source layer : The starting point of the FinGPT
pipeline is the Data Source Layer, which orchestrates the
acquisition of extensive financial data from a wide array
of online sources. This layer ensures comprehensive mar-
ket coverage by integrating data from news websites, social
media platforms, financial statements, market trends, and
more. The goal is to capture every nuance of the market,
thereby addressing the inherent temporal sensitivity of fi-

nancial data.
•Data engineering layer : This layer focuses on the real-
time processing of NLP data to tackle the challenges of
high temporal sensitivity andlow signal-to-noise ratio in-
herent in financial data. It incorporates state-of-the-art NLP
techniques to filter noise and highlight the most salient
pieces of information.
•LLMs layer : Lying at the heart, it encompasses various
fine-tuning methodologies, with a priority on lightweight
adaptation, to keep the model updated and pertinent. By
maintaining an updated model, FinGPT can deal with the
highly dynamic nature of financial data, ensuring its re-
sponses are in sync with the current financial climate.
•Application layer : The final component of FinGPT is
the Applications Layer, designed to demonstrate the prac-

tical applicability of FinGPT. It offers hands-on tutorials
and demo applications for financial tasks, including robo-
advisory services, quantitative trading, and low-code devel-
opment. These practical demonstrations not only serve as a
guide to potential users but also underscore the transforma-
tive potential of LLMs in finance.Figure 1: FinGPT Framework.
4.1 Data Sources
The first stage of the FinGPT pipeline involves the collec-
tion of extensive financial data from a wide array of online
sources. These include, but are not limited to:
•Financial news: Websites such as Reuters, CNBC, Yahoo
Finance, among others, are rich sources of financial news
and market updates. These sites provide valuable informa-
tion on market trends, company earnings, macroeconomic
indicators, and other financial events.

•Social media : Platforms such as Twitter, Facebook, Red-
dit, Weibo, and others, offer a wealth of information in
terms of public sentiment, trending topics, and immediate
reactions to financial news and events.
•Filings : Websites of financial regulatory authorities, such
as the SEC in the United States, offer access to company
filings. These filings include annual reports, quarterly earn-
ings, insider trading reports, and other important company-
specific information. Official websites of stock exchanges
(NYSE, NASDAQ, Shanghai Stock Exchange, etc.) pro-
vide crucial data on stock prices, trading volumes, company
listings, historical data, and other related information.
•Trends : Websites like Seeking Alpha, Google Trends, and
other finance-focused blogs and forums provide access to

analysts’ opinions, market predictions, the movement of
specific securities or market segments and investment ad-
vice.•Academic datasets : Research-based datasets that offer cu-
rated and verified information for sophisticated financial
analysis.
To harness the wealth of information from these diverse
sources, FinGPT incorporates data acquisition tools capable
of scraping structured and unstructured data, including APIs,
web scraping tools, and direct database access where avail-
able. Moreover, the system is designed to respect the terms
of service of these platforms, ensuring data collection is ethi-
cal and legal.
Data APIs : In the FinGPT framework, APIs are used not
only for initial data collection but also for real-time data up-
dates, ensuring the model is trained on the most current data.

Additionally, error handling and rate-limiting strategies are
implemented to respect API usage limits and avoid disrup-
tions in the data flow.
4.2 Real-Time Data Engineering Pipeline for
Financial NLP
Financial markets operate in real-time and are highly sensi-
tive to news and sentiment. Prices of securities can change
rapidly in response to new information, and delays in pro-
cessing that information can result in missed opportunities or
increased risk. As a result, real-time processing is essential in
financial NLP.
The primary challenge with a real-time NLP pipeline is
managing and processing the continuous inflow of data ef-
ficiently. The first step in the pipeline is to set up a system toingest data in real-time. This data could be streaming from

our data source APIs. Below are the steps to design a real-
time NLP pipeline for data ingestion.
Data cleaning : Real-time data can be noisy and inconsis-
tent. Therefore, real-time data cleaning involves removing
irrelevant data, handling missing values, text normalization
(like lowercasing), and error corrections.
Tokenization : In real-time applications, tokenization has
to be performed on the fly. This involves breaking down the
stream of text into smaller units or tokens.
Stop word removal and stemming/lemmatization : For
real-time processing, a predefined list of stop words can be
used to filter out these common words from the stream of
tokens. Likewise, stemming and lemmatization techniques
can be applied to reduce words to their root form.
Feature extraction and sentiment analysis : Feature ex-

traction involves transforming raw data into an input that can
be understood by machine learning models. In real-time sys-
tems, this often needs to be a fast and efficient process. Tech-
niques such as TF-IDF, Bag of Words, or embedding vectors
like Word2Vec can be used. Sentiment analysis can also be
performed on the cleaned data. This is where we categorize a
span of text as positive, negative, or neutral.
Prompt engineering : The creation of effective prompts
that can guide the language model’s generation process to-
ward desirable outputs.
Alerts/Decision making : Once the prompt is entered, the
results need to be communicated or acted upon. This might
involve triggering alerts based on certain conditions, inform-
ing real-time decision-making processes, or feeding the out-
put into another system.

Continuous learning : In real-time systems, the models
should adapt to changes in the data. Continuous learning sys-
tems can be implemented, where models are periodically re-
trained on new data or online learning algorithms are used
that can update the model with each new data point.
Monitoring : Real-time systems require continuous mon-
itoring to ensure they are functioning correctly. Any delays
or issues in the pipeline can have immediate impacts, so it’s
important to have robust monitoring and alerting in place.
4.3 Large Language Models (LLMs)
Once the data has been properly prepared, it is used with
LLMs to generate insightful financial analyses. The LLM
layer includes:
•LLM APIs : APIs from established LLMs provide baseline
language capability.
•Trainable models : FinGPT provides trainable models that

users can fine-tune on their private data, customizing for
financial applications.
•Fine-tuning methods : Various fine-tuning methods allow
FinGPT to be adapted to personalized robo-advisor.
Why fine-tune LLMs instead of retraining from scratch?
Leveraging pre-existing Large Language Models (LLMs)
and fine-tuning them for finance provides an efficient, cost-
effective alternative to expensive and lengthy model retrain-
ing from scratch.BloombergGPT, though remarkable in its finance-specific
capabilities, comes with an intensive computational require-
ment. It used approximately 1.3 million GPU hours for train-
ing, which, when calculated using AWS cloud’s $2.3 rate,
translates to a staggering cost of around $3 million per train-
ing. In contrast to the high computational cost of models like

BloombergGPT, FinGPT presents a more accessible solution
by focusing on the lightweight adaptation of top open-source
LLMs. The cost of adaptation falls significantly, estimated at
less than $300 per training.
This approach ensures timely updates and adaptability, es-
sential in the dynamic financial domain. Being open-source,
FinGPT not only promotes transparency but also allows
user customization, catering to the rising trend of personal-
ized financial advisory services. Ultimately, FinGPT’s cost-
effective, flexible framework holds the potential to democra-
tize financial language modeling and foster user-focused fi-
nancial services.
Fine-tuning via Low-rank Adaptation (LoRA)
In FinGPT, we fine-tune to a pre-trained LLM utilizing a
novel financial dataset. It’s well recognized that high-quality

labeled data is a pivotal determinant for many successful
LLMs, including ChatGPT. However, acquiring such top-
notch labeled data often proves costly in terms of time and
resources and generally requires the expertise of finance pro-
fessionals.
If our objective is to employ LLMs for analyzing financial-
related text data and assisting in quantitative trading, it seems
sensible to leverage the market’s inherent labeling capac-
ity. Consequently, we use the relative stock price change
percentage for each news item as the output label. We
establish thresholds to divide these labels into three cate-
gories—positive, negative, and neutral—based on the senti-
ment of the news item.
In a corresponding step, during the prompt engineering
process, we also prompt the model to select one from the pos-

itive, negative, and neutral outputs. This strategy ensures op-
timal utilization of the pre-trained information. By deploying
the Low-Rank Adaptation (LoRA) of LLMs [Huet al. , 2021;
Dettmers et al. , 2023 ], we manage to reduce the number of
trainable parameters from 6.17 billion to a mere 3.67 million.
Fine-tuning via Reinforcement Learning on Stock Prices
(RLSP)
Similarly, we can substitute Reinforcement Learning on
Stock Prices (RLSP) for Reinforcement Learning on Human
feedback, as utilized by ChatGPT. The reasoning behind this
substitution is that stock prices offer a quantifiable, objective
metric that reflects market sentiment in response to news and
events. This makes it a robust, real-time feedback mechanism
for training our model.

Reinforcement Learning (RL) allows the model to learn
through interaction with the environment and receiving feed-
back. In the case of RLSP, the environment is the stock
market, and the feedback comes in the form of stock price
changes. This approach permits FinGPT to refine its under-
standing and interpretation of financial texts, improving its
ability to predict market responses to various financial events.By associating news sentiment with the subsequent perfor-
mance of the related stocks, RLSP provides an effective way
to fine-tune FinGPT. In essence, RLSP allows the model to
infer the market’s response to different news events and ad-
just its understanding and predictions accordingly.
Therefore, the integration of RLSP into the fine-tuning pro-
cess of FinGPT provides a powerful tool for improving the

model’s financial market understanding and predictive accu-
racy. By using actual stock price movements as feedback, we
are directly harnessing the wisdom of the market to make our
model more effective.
4.4 Applications
FinGPT may find wide applications in financial services, aid-
ing professionals and individuals in making informed finan-
cial decisions. The potential applications include:
•Robo-advisor : Offering personalized financial advice, re-
ducing the need for regular in-person consultations.
•Quantitative trading : Producing trading signals for in-
formed trading decisions.
•Portfolio optimization : Utilizing numerous economic in-
dicators and investor profiles for optimal investment port-
folio construction.
•Financial sentiment analysis : Evaluating sentiments

across different financial platforms for insightful invest-
ment guidance.
•Risk management: Formulating effective risk strategies
by analyzing various risk factors.
•Financial Fraud detection : Identifying potential fraudu-
lent transaction patterns for enhanced financial security.
•Credit scoring : Predicting creditworthiness from financial
data to aid lending decisions.
•Insolvency prediction : Predicting potential insolvency or
bankruptcy of companies based on financial and market
data.
•Mergers and acquisitions (M&A) forecasting : Predict-
ing potential M&A activities by analyzing financial data
and company profiles, helping investors anticipate market
movements.
•ESG (Environmental, Social, Governance) scoring :
Evaluating companies’ ESG scores by analyzing public re-

ports and news articles.
•Low-code development : Facilitating software creation
through user-friendly interfaces, reducing reliance on tra-
ditional programming.
•Financial education : Serving as an AI tutor simplifying
complex financial concepts for better financial literacy.
By linking these distinct yet interconnected components,
FinGPT provides a holistic and accessible solution for lever-
aging AI in finance, facilitating research, innovation, and
practical applications in the financial industry.5 Conclusion
In conclusion, the transformative integration of large lan-
guage models (LLMs) into the financial sector brings unique
complexities and vast opportunities. Navigating challenges
such as high temporal sensitivity, dynamic financial land-
scape, and a low signal-to-noise ratio in financial data calls

for efficient solutions. FinGPT responds innovatively by
leveraging pre-existing LLMs and fine-tuning them to spe-
cific financial applications. This approach significantly re-
duces adaptation costs and computational requirements com-
pared to models like BloombergGPT, offering a more acces-
sible, flexible, and cost-effective solution for financial lan-
guage modeling. Thus, it enables consistent updates to en-
sure model accuracy and relevance, a critical aspect in the
dynamic and time-sensitive world of finance.
6 Future Work
FinLLMs, or Financial Large Language Models, present a
vision of the future where personalized robo-advisors or as-
sistants are within everyone’s reach. It aims to democratize
access to high-quality financial advice, leveraging advanced

language modeling techniques to make sense of vast amounts
of financial data and transform it into actionable insights. The
following blueprint outlines the future direction of FinLLM.
•Individualization : At the heart of FinLLM’s strategy is
the concept of individualized fine-tuning. Using techniques
such as LoRA and QLoRA, FinLLM enables users to tailor
models to their specific needs, thereby creating a personal
robo-advisor or assistant. This aligns with a broader trend
towards customization in financial services, as consumers
increasingly demand personalized advice that aligns with
their unique risk profiles and financial goals.
•Open-source and low-cost adaptation : FinLLM cham-
pions open-source values, providing users with the tools
they need to adapt Large Language Models (LLMs) to their

own requirements at a low cost, typically between $100 to
$300. This not only democratizes access to advanced finan-
cial modeling techniques but also fosters a vibrant commu-
nity of developers and researchers, collectively pushing the
boundaries of what’s possible in the field of financial AI.
•Access to high-quality financial data : FinLLM goes be-
yond just providing modeling techniques, also offering ac-
cess to high-quality financial data. This ensures that users
have the data they need to train their models effectively,
while also simplifying the data curation process. This ac-
cess is further enhanced by the provision of a data curation
pipeline with demos, empowering users to harness the full
potential of their financial data.
Disclaimer: We are sharing codes for academic pur-

poses under the MIT education license. Nothing herein is
financial advice, and NOT a recommendation to trade real
money. Please use common sense and always first consult
a professional before trading or investing.References
[Araci, 2019 ]Dogu Araci. Finbert: Financial sentiment
analysis with pre-trained language models. arXiv preprint
arXiv:1908.10063 , 2019.
[Baoet al. , 2021 ]Siqi Bao, Huang He, Fan Wang, Hua Wu,
Haifeng Wang, Wenquan Wu, Zhihua Wu, Zhen Guo, Hua
Lu, Xinxian Huang, et al. Plato-xl: Exploring the large-
scale pre-training of dialogue generation. arXiv preprint
arXiv:2109.09519 , 2021.
[Brown et al. , 2020 ]Tom Brown, Benjamin Mann, Nick Ry-
der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari-
wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry,
Amanda Askell, et al. Language models are few-shot

learners. Advances in Neural Information Processing Sys-
tems, 33:1877–1901, 2020.
[DeLucia et al. , 2022 ]Alexandra DeLucia, Shijie Wu,
Aaron Mueller, Carlos Aguirre, Philip Resnik, and Mark
Dredze. Bernice: a multilingual pre-trained encoder
for twitter. In Proceedings of the 2022 conference on
empirical methods in natural language processing , pages
6191–6205, 2022.
[Dettmers et al. , 2023 ]Tim Dettmers, Artidoro Pagnoni,
Ari Holtzman, and Luke Zettlemoyer. QLoRA: Ef-
ficient finetuning of quantized llms. arXiv preprint
arXiv:2305.14314 , 2023.
[Devlin et al. , 2018 ]Jacob Devlin, Ming-Wei Chang, Ken-
ton Lee, and Kristina Toutanova. Bert: Pre-training of
deep bidirectional transformers for language understand-
ing.arXiv preprint arXiv:1810.04805 , 2018.
[Dredze et al. , 2016 ]Mark Dredze, Prabhanjan Kambadur,

Gary Kazantsev, Gideon Mann, and Miles Osborne. How
twitter is changing the nature of financial news discovery.
Inproceedings of the second international workshop on
data science for macro-modeling , pages 1–5, 2016.
[Ethayarajh, 2019 ]Kawin Ethayarajh. How contextual are
contextualized word representations? comparing the ge-
ometry of bert, elmo, and gpt-2 embeddings. arXiv
preprint arXiv:1909.00512 , 2019.
[Huet al. , 2021 ]Edward J Hu, Yelong Shen, Phillip Wallis,
Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang,
and Weizhu Chen. LoRA: Low-rank adaptation of large
language models. International Conference on Learning
Representations , 2021.
[Lewis et al. , 2019 ]Mike Lewis, Yinhan Liu, Naman Goyal,
Marjan Ghazvininejad, Abdelrahman Mohamed, Omer
Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: De-

noising sequence-to-sequence pre-training for natural lan-
guage generation, translation, and comprehension. arXiv
preprint arXiv:1910.13461 , 2019.
[Lewis et al. , 2020 ]Patrick Lewis, Myle Ott, Jingfei Du, and
Veselin Stoyanov. Pretrained language models for biomed-
ical and clinical tasks: understanding and extending the
state-of-the-art. In Proceedings of the 3rd Clinical Natural
Language Processing Workshop , pages 146–157, 2020.[Liuet al. , 2021 ]Xiao-Yang Liu, Hongyang Yang, Jiechao
Gao, and Christina Dan Wang. FinRL: Deep reinforce-
ment learning framework to automate trading in quantita-
tive finance. ACM International Conference on AI in Fi-
nance (ICAIF) , 2021.
[Liuet al. , 2022 ]Xiao-Yang Liu, Ziyi Xia, Jingyang Rui,
Jiechao Gao, Hongyang Yang, Ming Zhu, Christina Dan

Wang, Zhaoran Wang, and Jian Guo. FinRL-Meta: Mar-
ket environments and benchmarks for data-driven financial
reinforcement learning. NeurIPS , 2022.
[Radford et al. , 2018 ]Alec Radford, Karthik Narasimhan,
Tim Salimans, Ilya Sutskever, et al. Improving language
understanding by generative pre-training. OpenAI , 2018.
[Thoppilan et al. , 2022 ]Romal Thoppilan, Daniel De Fre-
itas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha,
Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker,
Yu Du, et al. Lamda: Language models for dialog ap-
plications. arXiv preprint arXiv:2201.08239 , 2022.
[Vaswani et al. , 2017 ]Ashish Vaswani, Noam Shazeer, Niki
Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,
Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you

need. In I. Guyon, U. V on Luxburg, S. Bengio, H. Wal-
lach, R. Fergus, S. Vishwanathan, and R. Garnett, editors,
Advances in Neural Information Processing Systems , vol-
ume 30. Curran Associates, Inc., 2017.
[Wuet al. , 2023 ]Shijie Wu, Ozan Irsoy, Steven Lu, Vadim
Dabravolski, Mark Dredze, Sebastian Gehrmann, Prab-
hanjan Kambadur, David Rosenberg, and Gideon Mann.
BloombergGPT: A large language model for finance.
arXiv preprint arXiv:2303.17564 , 2023.Fast Segment Anything
Xu Zhao1,3Wenchao Ding1,2Yongqi An1,2Yinglong Du1,2
Tao Yu1,2Min Li1,2Ming Tang1,2Jinqiao Wang1,2,3,4
Institute of Automation, Chinese Academy of Sciences, Beijing, China1
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China2
Objecteye Inc., Beijing, China3
Wuhan AI Research, Wuhan, China4

{xu.zhao,yongqi.an,tangm,jqwang }@nlpr.ia.ac.cn
{dingwenchao2021,duyinglong2022,yutao2022,limin2021 }@ia.ac.cn
Abstract
The recently proposed segment anything model (SAM)
has made a significant influence in many computer vision
tasks. It is becoming a foundation step for many high-level
tasks, like image segmentation, image caption, and image
editing. However, its huge computation costs prevent it from
wider applications in industry scenarios. The computation
mainly comes from the Transformer architecture at high-
resolution inputs. In this paper, we propose a speed-up al-
ternative method for this fundamental task with compara-
ble performance. By reformulating the task as segments-
generation and prompting, we find that a regular CNN de-
tector with an instance segmentation branch can also ac-

complish this task well. Specifically, we convert this task
to the well-studied instance segmentation task and directly
train the existing instance segmentation method using only
1/50 of the SA-1B dataset published by SAM authors. With
our method, we achieve a comparable performance with
the SAM method at 50×higher run-time speed. We give
sufficient experimental results to demonstrate its effective-
ness. The codes and demos will be released at https:
//github.com/CASIA-IVA-Lab/FastSAM .
1. Introduction
Recently, the Segment Anything Model (SAM) [19] is
proposed. It is regarded as a milestone vision foundation
model. It can segment any object within the image guided
by various possible user interaction prompts. SAM lever-
ages a Transformer model trained on the extensive SA-1B

dataset, which gives it the ability to deftly handle a wide
range of scenes and objects. SAM opens the door to an ex-
citing new task known as Segment Anything . This task, due
to its generalizability and potentiality, has all the makings
of becoming a cornerstone for a broad spectrum of future
vision tasks.
FastSAM
SAM
40ms/img
2099ms/img
50×Faster
(a)
(b) (c)Figure 1. Comparative analysis of FastSAM and SAM. (a) Speed
comparison between FastSAM and SAM on a single NVIDIA
GeForce RTX 3090. (b) Comparison on the BSDS500 dataset
[1, 28] for edge detection. (c) Box AR@1000 evaluation of Fast-
SAM and SAM on the COCO dataset [25] for the object proposal.
Both SAM and FastSAM are tested using PyTorch for inference,
except FastSAM(TRT) uses TensorRT for inference.
However, despite these advancements and the promising

results shown by SAM and subsequent models in handling
the segment anything task, its practical applications are still
challenging. The glaring issue is the substantial computa-
tional resource requirements associated with Transformer
(ViT) models, the main part of SAM’s architecture. When
compared with their convolutional counterparts, ViTs stand
out for their heavy computation resources demands, which
presents a hurdle to their practical deployment, especially in
real-time applications. This limitation consequently hinders
the progress and potential of the segment anything task.
Motivated by the high demand from the industrial appli-arXiv:2306.12156v1  [cs.CV]  21 Jun 2023cations for the segment anything model, in this paper we
design a real-time solution for the segment anything task,

FastSAM. We decouple the segment anything task into two
sequential stages which are all-instance segmentation and
prompt-guided selection. The first stage hinges on the im-
plementation of a Convolutional Neural Network (CNN)-
based detector. It produces the segmentation masks of all in-
stances in the image. Then in the second stage, it output the
the region-of-interest corresponding the prompt. By lever-
aging the computational efficiency of CNNs, we demon-
strate that a real-time segment of anything model is achiev-
able without much compromising on performance quality.
We hope the proposed method would facilitate the indus-
trial applications of the foundational task of segmenting
anything.
Our proposed FastSAM is based on YOLOv8-seg [16],
an object detector equipped with the instance segmenta-
tion branch, which utilizes the YOLACT [4] method. We
also adopt the extensive SA-1B dataset published by SAM.

By directly training this CNN detector on only 2% (1/50)
of the SA-1B dataset, it achieves comparable performance
to SAM, but with drastically reduced computational and
resource demands, thereby enabling real-time application.
We also apply it to multiple downstream segmentation tasks
to show its generalization performance. On the object pro-
posal task on MS COCO [13], we achieve 63.7 at AR1000,
which is 1.2 points higher than SAM with 32 ×32 point-
prompt inputs, but running 50 times faster on a single
NVIDIA RTX 3090.
The real-time segment anything model is valuable for in-
dustrial applications. It can be applied to many scenarios.
The proposed approach not only provides a new, practical
solution for a large number of vision tasks but also does so
at a really high speed, tens or hundreds of times faster than

current methods.
It also offers new views for the large model architecture
for the general vision tasks. We think for specific tasks, spe-
cific models still take advantage to get a better efficiency-
accuracy trade-off.
Then, in the sense of model compression, our approach
demonstrates the feasibility of a path that can significantly
reduce the computational effort by introducing an artificial
prior to the structure.
Our contributions can be summarized as follow:
• A novel, real-time CNN-based solution for the Seg-
ment Anything task is introduced, which significantly
reduces computational demands while maintaining
competitive performance.
• This work presents the first study of applying a CNN
detector to the segment anything task, offering insights
into the potential of lightweight CNN models in com-

plex vision tasks.• A comparative evaluation between the proposed
method and SAM on multiple benchmarks provides
insights into the strengths and weaknesses of the ap-
proach in the segment anything domain.
2. Preliminary
In this section, we give a review of the segment anything
model and a clear definition of the segment anything task.
Segment Anything Model. In the evolving field of image
segmentation, the Segment Anything Model (SAM) [19] is
a significant innovation due to its proposed training method-
ology and performance on large-scale visual datasets. SAM
provides a high-precision, class-agnostic segmentation per-
formance, exhibiting distinct capabilities in zero-shot tasks.
As a foundation model, it expands the horizons of computer
vision by showing not just powerful interactive segmenta-

tion methods, but also exceptional adaptability across a va-
riety of segmentation tasks. SAM is a striking example of
the potential of foundation models for open-world image
understanding. However, while the model’s performance
is satisfying, it is worth noting that SAM faces a significant
limitation – the lack of real-time processing capability. This
restricts its wide application in scenarios where immediate
segmentation results are critical.
Segment Anything Task. TheSegment Anything task is de-
fined as a process whereby an effective segmentation mask
is produced given any form of the prompt. These prompts
range from foreground/background point sets, rough boxes
or masks, free-form text, or any information that indicates
the content to be segmented within an image. We have dis-

covered that the segment anything task can be effectively
broken down into two stages in the majority of practical
applications. The first stage involves detecting and seg-
menting all objects in the image, like a panoptic segmen-
tation [18] process. The second stage depends on the pro-
vided prompts to separate the specific object(s) of interest
from the segmented panorama. The decoupling of this task
significantly reduces its complexity, thus providing the pos-
sibility to propose a real-time segment of anything model.
3. Methodology
3.1. Overview
Fig. 2 gives the overview of the proposed method, Fast-
SAM. The method consists of two stages, i.e. the All-
instance Segmentation and the Prompt-guided Selection.
The former stage is a basis and the second stage is essen-

tially the task-oriented post-processing. Different from the
end-to-end transformers [7,8,19], the overall method intro-
duces many human priors which match the vision segmen-
tation tasks, like the local connections of convolutions andProtoNetDetectDetectDetect
Mask Coeff.Mask Coeff.Mask Coeff.NMS
Crop
Pred
ThresholdDetect Branch
·
TextPoint-prompt Box-prompt Text-promptP5
P4
P3
T1I1
I2
···
INI1·T1
···I2·T1
IN·T1 CLIP
“The black dog”Mask Branch
··· ···
- + =···+0.0137 -0.0342 +0.6846
-1+1
--
Mask Coeff.FPN CNN
Backbone
Image
EncoderText
Encoder
···
...Figure 2. The framework of FastSAM. It contains two stages: All-instance Segmentation (AIS) and Prompt-guided Selection (PGS). We
use YOLOv8-seg [16] to segment all objects or regions in an image. Then we use various prompts to identify the specific object(s) of

interest. It mainly involves the utilization of point prompts, box prompts, and text prompt. The text prompt is based on CLIP [31].
the receptive-field-relevant object assigning strategies. This
makes it tailored for the vision segmentation task and can
converge faster on a smaller number of parameters.
3.2. All-instance Segmentation
Model Architecture. The architecture of YOLOv8 [16] de-
velops from its predecessor, YOLOv5 [15], integrating key
design aspects from recent algorithms such as YOLOX [10],
YOLOv6 [22], and YOLOv7 [35]. YOLOv8’s backbone
network and neck module substitute YOLOv5’s C3 module
with the C2f module. The updated Head module embraces a
decoupled structure, separating classification and detection
heads, and shifts from Anchor-Based to Anchor-Free.
Instance Segmentation. YOLOv8-seg applies

YOLACT [4] principles for instance segmentation. It
begins with feature extraction from an image via a
backbone network and the Feature Pyramid Network
(FPN) [24], integrating diverse size features. The output
consists of the detection and segmentation branches.The detection branch outputs category and bounding
box, while the segmentation branch outputs kprototypes
(defaulted to 32 in FastSAM) along with k mask coef-
ficients. The segmentation and detection tasks are com-
puted in parallel. The segmentation branch inputs a high-
resolution feature map, preserves spatial details, and also
contains semantic information. This map is processed
through a convolution layer, upscaled, and then passed
through two more convolution layers to output the masks.
The mask coefficients, similar to the detection head’s clas-

sification branch, range between -1 and 1. The instance seg-
mentation result is obtained by multiplying the mask coef-
ficients with the prototypes and then summing them up.
YOLOv8 can be used in a variety of object detection
tasks. With the instance segmentation branch, YOLOv8-
Seg is born suitable for the segment anything task, which
aims to accurately detect and segment every object or region
in an image, regardless of the object category. The proto-
types and mask coefficients provide a lot of extensibility for
prompt guidance. As a simple example, a simple promptencoder and decoder structure is additionally trained, with
various prompts and image feature embeddings as input and
mask coefficients as output. In FastSAM, we directly use
the YOLOv8-seg method for the all-instance segmentation

stage. The more artificial design might bring additional im-
provements, but we regard it as out of the scope of this work
and leave it for future study.
3.3. Prompt-guided Selection
Following the successful segmentation of all objects or
regions in an image using YOLOv8, the second stage of the
segment anything task is to use various prompts to identify
the specific object(s) of interest. It mainly involves the uti-
lization of point prompts, box prompts, and text prompts.
Point prompt. The point prompt consists of matching
the selected points to the various masks obtained from the
first phase. The goal is to determine the mask in which
the point is located. Similar to SAM, we employ fore-
ground/background points as the prompt in our approach.
In cases where a foreground point is located in multiple

masks, background points can be utilized to filter out masks
that are irrelevant to the task at hand. By employing a set of
foreground/background points, we are able to select multi-
ple masks within the region of interest. These masks will be
merged into a single mask to completely mark the object of
interest. In addition, we utilize morphological operations to
improve the performance of mask merging.
Box prompt. The box prompt involves performing Inter-
section over Union (IoU) matching between the selected
box and the bounding boxes corresponding to the various
masks from the first phase. The aim is to identify the mask
with the highest IoU score with the selected box and thus
select the object of interest.
Text prompt. In the case of text prompt, the correspond-
ing text embeddings of the text are extracted using the

CLIP [31] model. The respective image embeddings are
then determined and matched to the intrinsic features of
each mask using a similarity metric. The mask with the
highest similarity score to the image embeddings of the text
prompt is then selected.
By carefully implementing these prompt-guided selec-
tion techniques, the FastSAM can reliably select specific
objects of interest from a segmented image. The above ap-
proach provides an efficient way to accomplish the segment
anything task in real-time, thus greatly enhancing the util-
ity of the YOLOv8 model for complex image segmentation
tasks. A more effective prompt-guided selection technique
is left for future exploration.
4. Experiments
In this section, we first analysis the run-time efficiency

of FastSAM. Then we experiment with four zero-shot tasks,along with applications in real-world scenarios, efficiency,
and deployment. In the first part of the experiments, our
goal is to test the similarity in capabilities between Fast-
SAM and SAM. Following SAM, we also experiment with
four tasks with different levels: (1) low-level: edge detec-
tion, (2) mid-level: object proposal generation, (3) high-
level: instance segmentation, and finally, (4) high-level:
segmenting objects with free-form text input. Our exper-
iments also further validate FastSAM’s capabilities with
real-world applications and speed.
Implementation Details. Unless stated otherwise, the
following conditions apply: (1) FastSAM employs a
YOLOv8-x [16] model as the main part of its architecture,

with the input size of 1024; (2) FastSAM’s training was car-
ried out on 2 %of the SA-1B dataset [19]; (3) We train the
model for 100 epochs using the default hyper-parameter set-
tings except that the regmax in the bounding box regres-
sion module is changed from 16 to 26 for predicting large
instances.
4.1. Run-time Efficiency Evaluation
SAM uses the Transformer architecture to construct an
end-to-end algorithm. The Transformer is a universal ar-
chitecture that can represent many forms of mapping func-
tions of various tasks. To segment anything, SAM learns
the vision-oriented inductive bias through the learning pro-
cess on large-scale data. On the contrary, with the human
priori knowledge in structure designing, FastSAM obtains
a relatively compact model. From Figure 3, The FastSAM

generates relatively satisfying results.
In Table 1, we report the running speed of SAM and Fast-
SAM on a single NVIDIA GeForce RTX 3090 GPU. It can
be seen that FastSAM surpasses SAM at all prompt num-
bers. Moreover, the running speed of FastSAM does not
change with the prompts, making it a better choice for the
Everything mode.
4.2. Zero-Shot Edge Detection
Approach. FastSAM is assessed on the basic low-level task
of edge detection using BSDS500 [1, 28]. Specifically, we
select the mask probability map from the results of Fast-
SAM’s all-instance segmentation stage. After that, Sobel
filtering [33] is applied to all mask probability maps to
generate edge maps. Finally, we conclude with the edge
NMS [6] step.
Results. The representative edge maps are illustrated in

Figure 4. Upon qualitative observation, it becomes evi-
dent that despite FastSAM’s significantly fewer parameters
(only 68M), it produces a generally good edge map. In com-
parison to the ground truth, both FastSAM and SAM tend
to predict a larger number of edges, including some logi-
cal ones that aren’t annotated in the BSDS500. This biasRunning Speed under Different Point Prompt Numbers (ms)
method params 1 10 100 E(16 ×16) E(32 ×32*) E(64 ×64)
SAM-H [20] 0.6G 446 464 627 852 2099 6972
SAM-B [20] 136M 110 125 230 432 1383 5417
FastSAM (Ours) 68M 40
Table 1. Running Speed (ms/image) of SAM and FastSAM under different point prompt numbers. E: Everything Mode of SAM. *: 32 ×32
is the default setting of SAM for many tasks1.
Figure 3. Segmentation Results of FastSAM
method year ODS OIS AP R50

HED [37] 2015 .788 .808 .840 .923
EDETR [30] 2022 .840 .858 .896 .930
zero-shot transfer methods:
Sobel filter 1968 .539 - - -
Canny [6] 1986 .600 .640 .580 -
Felz-Hutt [9] 2004 .610 .640 .560 -
SAM [19] 2023 .768 .786 .794 .928
FastSAM 2023 .750 .790 .793 .903
Table 2. Zero-shot transfer to edge detection on BSDS500. Eval-
uation Data of other methods is from [20].
is quantitatively reflected in Table 2. Table 2 shows that
we achieve similar performance with SAM, specifically a
higher R50 and a lower AP.
4.3. Zero-Shot Object Proposal Generation
Background. Object proposal generation has long been a
basic pre-processing step for many computer vision tasks,
including general object detection, few-shot object detec-
tion, and image understanding. Many famous proposal gen-
eration methods witness the evolution of visual recognition

in the past decades, as a role of the basic step of visual
recognition methods. These proposal generation methods
includes EdgeBox [38], Geodesic [21], Selective Search
1https://github.com/facebookresearch/segment-
anything[34], MCG [2]. These years, many deep learning-based
methods are proposed like DeepMask [29], OLN [17]. For
example, RCNN-series object detection methods [11, 12]
adopts the Seletive Search method, and the recently pro-
posed open world detector, UniDetector [36], adopts the
OLN method. Though RPN [32] is used by most exist-
ing object detectors, it can only generate object proposals
of the learned categories, limiting its application in open-
vocabulary recognition tasks. Therefore, zero-shot object
proposal generation is rather important. A good proposed

method is important for the good performance of these vi-
sual recognition tasks.
We directly use the generated bounding boxes of the first
stage of FastSAM as the object proposals. To evaluate the
performance, we test on LVIS [13] and COCO [25] dataset,
following the existing evaluating strategies. Besides this,
following the experimental settings of SAM, we also test
the mask proposal accuracy by using the all-instance masks
of the first stage.
Details. We report the results of SAM, ViTDet [23], and our
FastSAM on the LVIS dataset. As SAM does not publicize
its detailed evaluation codes, we reproduced the category-
agnostic mask and box recall using the official LVIS eval-
uation code [13]. However, we fail to reproduce the Mask
AR results of ViTDet and SAM presented in the SAM’s pa-

per [20]. Nevertheless, we think our evaluation results still
reflect several features of FastSAM compared with SAM.image ground truth SAM FastSAM
Figure 4. Zero-shot edge prediction on BSDS500. FastSAM achieves comparable results to SAM.
Results. The results is shown in Table 3, 4, and 5. The re-
sults show that our method has a significant advantage on
the box proposal generation tasks. Table 3 presents the Av-
erage Recall (AR) of various methods on the COCO vali-
dation set. Among these, EdgeBoxes [38], Geodesic [21],
Sel.Search [34], and MCG [2] are methods that do not
require training, whereas DeepMask [29] and OLN [17]
are supervised methods that are trained on VOC categories
within the COCO training set, and then tested across all cat-
egories. In contrast, our approach and SAM [20] implement

a fully zero-shot transfer. From the table, it can be seen
that our method and SAM [20] do not perform as well in
AR@10 precision compared to previous supervised meth-
ods such as OLN [17]. However, in AR@1000, our method
significantly outperforms OLN [17]. The reason for this is
that previous methods were trained on certain categories in
COCO, leading to a higher confidence level in these cat-
egories during inference. However, since our method and
SAM are zero-shot, this results in balanced confidence lev-
els across different categories, thereby recalling more cate-
gories not present in COCO. More comparisons can be seen
in Figure 5.
In Table 4, we report the bbox AR@1000 results of
VitDet-H [23], SAM [20], and our method on the LVIS
v1 dataset. Our method substantially surpasses the most

computationally intensive model of SAM, SAM-H E64, by
over 5%. However, it falls short compared to VitDet-H [23],
which was trained on the LVIS dataset. The reason for
these results is that during our training process, we used
the ground truth (gt) bounding box (bbox) information as a
supervisory signal. SAM [20], on the other hand, only used
the mask as a supervisory signal, and its bbox at inference
is generated by extracting the outer box from the mask.
From Table 5, our mask proposal generation is relatively
lower on Recall. We find this mainly results from that
our segmentation mask on small-sized objects is not fine-AR10 AR100 AR1000 AUC
EdgeBoxes [38] 7.4 17.8 33.8 13.9
Geodesic [21] 4.0 18.0 35.9 12.6
Sel.Search [34] 5.2 16.3 35.7 12.6
MCG [2] 10.1 24.6 39.8 18.0
DeepMask [29] 13.9 28.6 43.1 21.7

OLN-Box [17] 27.7 46.1 55.7 34.3
SAM-H E64 15.5 45.6 67.7 32.1
SAM-H E32 18.5 49.5 62.5 33.7
SAM-B E32 11.4 39.6 59.1 27.3
FastSAM (Ours) 15.7 47.3 63.7 32.2
Table 3. Comparison with learning-free methods on All Categories
of COCO. We report average recall (AR) and AUC of learning
free methods, deep learning methods (trained on VOC), and ours
vs SAM on All generalization. The scores of competing methods
are taken from [17], which test object proposal methods on all 80
COCO classes.
Figure 5. Comparison with OLN [17] and SAM-H [20]. We test
object proposal methods on all 80 COCO classes
grained enough. We give more detailed discussions in Sec-
tion 6.bbox AR@1000
method all small med. large
ViTDet-H [23] 65.0 53.2 83.3 91.2
SAM-H E64 52.1 36.6 75.1 88.2
SAM-H E32 50.3 33.1 76.2 89.8

SAM-B E32 45.0 29.3 68.7 80.6
FastSAM (Ours) 57.1 44.3 77.1 85.3
Table 4. Object proposal generation on LVIS v1. FastSAM and
SAM is applied zero-shot, i.e. it was not trained for object proposal
generation nor did it access LVIS images or annotations.
mask AR@1000
method all small med. large freq. com. rare
results reported in the SAM paper:
ViTDet-H [23] 63.0 51.7 80.8 87.0 63.1 63.3 58.3
SAM [20] – single out. 54.9 42.8 76.7 74.4 54.7 59.8 62.0
SAM [20] 59.3 45.5 81.6 86.9 59.1 63.9 65.8
results after our replication:
ViTDet-H [23] 59.9 48.3 78.1 84.8 - - -
SAM-H E64 54.2 39.6 77.9 83.9
SAM-H E32 51.8 35.2 78.7 85.2 - - -
SAM-B E32 45.8 31.1 70.5 73.6 - - -
FastSAM (Ours) 49.7 35.6 72.7 77.6 - - -
Table 5. Object proposal generation on LVIS v1. FastSAM and

SAM are applied zero-shot, i.e. it was not trained for object pro-
posal generation nor did it access LVIS images or annotations.
Figure 6. Segmentation results with text prompts
4.4. Zero-Shot Instance Segmentation
Approach. Similarly to the SAM method, we accomplish
the instance segmentation task by utilizing the bounding
box (bbox) generated by ViTDet [23] as the prompt. As de-
scribed by Section 3.3, we choose the mask with the highest
Intersection over Union (IoU) with the bbox as the predicted
mask.
Results. Table 6 gives the evaluation results. On this task,
we fail to achieve a high AP. We infer that this mainly be-
cause of the segmentation mask accuracy or the box-based
mask selection strategy. Section 6 gives several examples.
4.5. Zero-Shot Object Localization with Text
Prompts

Prompts
Approach. Finally, we consider an even high-level task, i.e.
segmenting objects by free-form texts. This experiment is
to show the FastSAM ability of processing text prompts like
SAM. Different wit SAM, FastSAM doesn’t need to modifyCOCO [26] LVIS v1 [13]
method AP APSAPMAPLAP APSAPMAPL
ViTDet-H [23] 51.0 32.0 54.3 68.9 46.6 35.0 58.0 66.3
zero-shot transfer methods (segmentation module only):
SAM 46.5 30.8 51.0 61.7 44.7 32.5 57.6 65.5
FastSAM 37.9 23.9 43.4 50.0 34.5 24.6 46.2 50.8
Table 6. Instance segmentation results. Fastsam is prompted with
ViTDet boxes to do zero-shot segmentation. The fully-supervised
ViTDet outperforms SAM, but the gap shrinks on the higher-
quality LVIS masks.
the training procedure. It directly runs text through CLIP’s
text encoder and then uses the resulting text embedding to

find the most similar mask at inference time.
Results. We show qualitative results in Fig. 6. FastSAM
can segment objects well based on the text prompts. Never-
theless, the running speed of the text-to-mask segmentation
is not satisfying, since each mask region is required to be
fed into the CLIP feature extractor. How to combine the
CLIP embedding extractor into the FastSAM’s backbone
network remains an interesting problem with respect to the
model compression.
5. Real-world Applications
In this section, we evaluate the performance of FastSAM
across different application scenarios and analyze its advan-
tages and limitations. We showcase visualizations of Fast-
SAM’s segmentation using point-prompt, box-prompt, and
everything modes, and compare it with SAM and ground
truths.

truths.
Anomaly Detection.
As detailed in [3], anomaly detection is a task that aims
to distinguish between defective and normal samples in the
manufacture. FastSAM is evaluated using the MVTec AD
dataset [3], with results displayed in Fig. 7. Under ev-
erything mode, FastSAM can segment nearly all regions
similar to SAM, but with a lower level of precision com-
pared to SAM. In addition, the mask for the background
does not completely cover the entire background, which
is an inherent characteristic of YOLACT [4]. By fore-
ground/background points (yellow and magenta points in
FastSAM-point respectively) or box-guided selection, Fast-
SAM can segment on the exact defective regions.
Salient Object Segmentation.
The aim of salient object segmentation [5] is to segment
the most attention-grabbing objects from an image. This

task is class-agnostic, setting it apart from semantic seg-
mentation. We apply FastSAM to the well-known saliency
dataset, ReDWeb-S [27]. As presented in Fig. 8, FastSAM
exhibited only a minor difference from SAM under every-
thing mode, as it segment fewer background objects which
are irrelevant to the task. By points-guided selection, suchoriginal image
ground truth FastSAM-point FastSAM-box FastSAM-everythingSAM-point SAM-box SAM-everything
Figure 7. Application on anomaly detection , where SAM-point/box/everything means using point-prompt, box-prompt, and everything
modes respectively.
original image

Figure 8. Application on salient object segmentation , where SAM-point/box/everything mean using point-prompt, box-prompt, and
everything modes respectively.
as yellow points in FastSAM-point, we can obtain masks of
all objects of interest. The segmentation result of FastSAM-
point is nearly identical to that of the SAM-point and the
ground truth, with only minor details lost at the edges. The
object of interest can also be selected by box prompt, such
as the green box in FastSAM-box. However, it is impossi-
ble to select multiple objects with a single box, which even
SAM-box cannot realize.
Building Extracting.
Building Extracting from optical remote sensing im-
agery has a wide range of applications, such as urbanplanning. We evaluate FastSAM on the dataset proposed
by [14]. As demonstrated in Fig.9, FastSAM performs

well in segmenting regularly shaped objects, but segments
fewer regions related to shadows compared to SAM. We can
also select regions of interest with point-prompt and box-
prompt, as presented in FastSAM-point and FastSAM-box.
It is worth noting that we position a point in a shadow region
in FastSAM-point. However, the correct mask for the build-
ing can still be obtained by merging based on this point.
This indicates that our method can resist the interference of
noise to some extent.original image
Figure 9. Application on building extracting , where SAM-point/box/everything means using point-prompt, box-prompt, and everything

Figure 10. FastSAM generates finer segmentation masks on the narrow region of the large objects
6. Discussion
Generally, the proposed FastSAM achieves a compara-
ble performance with SAM and runs 50x faster than SAM
(32×32) and 170x faster than SAM (64 ×64). The running
speed makes it a good choice for industrial applications,
such as road obstacle detection, video instance tracking, and
image manipulation. On some images, FastSAM even gen-
erates better masks for large objects, as shown in Figure 10.
Weakness. However, as presented in the experiments, our
box generation has a significant advantage, but our mask
generation performance is below SAM. We visualize these
examples in Figure 11. We find that FastSAM has the fol-
lowing features.
• The low-quality small-sized segmentation masks have

large confidence scores. We think this is because
the confidence score is defined as the bbox score of
YOLOv8, which is not strongly related to the maskquality. Modifying the network to predict the mask
IoU or other quality indicators is a way to improve that.
• The masks of some of the tiny-sized objects tend to
be near the square. Besides, the mask of large objects
may have some artifacts on the border of the bounding
boxes. This is the weakness of the YOLACT method.
By enhancing the capacity of the mask prototypes or
reformulating the mask generator, the problem is ex-
pected to solve.
Moreover, since we only use 1/50 of all SA-1B datasets,
the model’s performance can also be further enhanced by
utilizing more training data.
7. Conclusion
In this paper, we rethink the segment of anything task

and the model architecture choosing, and propose an al-
ternative solution with 50 times faster running speed thanFigure 11. Some examples for the bad case of FastSAM.
SAM-ViT-H (32 ×32). The experiments show that Fast-
SAM can solve multiple downstream tasks well. Still,
there are several weaknesses that can be improved for Fast-
SAM, like the scoring mechanism and the instance mask-
generating paradigm. These problems are left for future
study.
References
[1] Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Ji-
tendra Malik. Contour detection and hierarchical image seg-
mentation. IEEE transactions on pattern analysis and ma-
chine intelligence , 33(5):898–916, 2010. 1, 4
[2] Pablo Arbel ´aez, Jordi Pont-Tuset, Jonathan T Barron, Fer-
ran Marques, and Jitendra Malik. Multiscale combinatorial

grouping. In Proceedings of the IEEE conference on com-
puter vision and pattern recognition , pages 328–335, 2014.
5, 6
[3] Paul Bergmann, Michael Fauser, David Sattlegger, and
Carsten Steger. Mvtec ad–a comprehensive real-world
dataset for unsupervised anomaly detection. In CVPR , pages
9592–9600, 2019. 7
[4] Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee.
Yolact: Real-time instance segmentation. In Proceedings of
the IEEE/CVF international conference on computer vision ,
pages 9157–9166, 2019. 2, 3, 7
[5] Ali Borji, Ming-Ming Cheng, Qibin Hou, Huaizu Jiang, and
Jia Li. Salient object detection: A survey. Computational
Visual Media , 5:117–150, 2019. 7
[6] John Canny. A computational approach to edge detection.
IEEE Transactions on pattern analysis and machine intelli-
gence , (6):679–698, 1986. 4, 5

[7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas
Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-
end object detection with transformers. In Computer Vision–
ECCV 2020: 16th European Conference, Glasgow, UK, Au-gust 23–28, 2020, Proceedings, Part I 16 , pages 213–229.
Springer, 2020. 2
[8] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-
pixel classification is not all you need for semantic segmen-
tation. Advances in Neural Information Processing Systems ,
34:17864–17875, 2021. 2
[9] Pedro F Felzenszwalb and Daniel P Huttenlocher. Efficient
graph-based image segmentation. International journal of
computer vision , 59:167–181, 2004. 5
[10] Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian
Sun. Yolox: Exceeding yolo series in 2021. arXiv preprint

arXiv:2107.08430 , 2021. 3
[11] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE inter-
national conference on computer vision , pages 1440–1448,
2015. 5
[12] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra
Malik. Region-based convolutional networks for accurate
object detection and segmentation. IEEE transactions on
pattern analysis and machine intelligence , 38(1):142–158,
2015. 5
[13] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A
dataset for large vocabulary instance segmentation. In Pro-
ceedings of the IEEE/CVF conference on computer vision
and pattern recognition , pages 5356–5364, 2019. 2, 5, 7
[14] Shunping Ji, Shiqing Wei, and Meng Lu. Fully convolutional
networks for multisource building extraction from an open
aerial and satellite imagery data set. IEEE Transactions on

Geoscience and Remote Sensing , 57(1):574–586, 2018. 8
[15] Glenn Jocher. Yolov5 by ultralytics, 2020. https://
github.com/ultralytics/yolov5 . 3
[16] Glenn Jocher, Ayush Chaurasia, and Jing Qiu. Yolo by ultra-
lytics, 2023. https://github.com/ultralytics/
ultralytics . 2, 3, 4
[17] Dahun Kim, Tsung-Yi Lin, Anelia Angelova, In So Kweon,
and Weicheng Kuo. Learning open-world object proposals
without learning to classify. IEEE Robotics and Automation
Letters , 7(2):5453–5460, 2022. 5, 6[18] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten
Rother, and Piotr Doll ´ar. Panoptic segmentation. In Pro-
ceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition , pages 9404–9413, 2019. 2
[19] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao,

Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White-
head, Alexander C Berg, Wan-Yen Lo, et al. Segment any-
thing. arXiv preprint arXiv:2304.02643 , 2023. 1, 2, 4, 5
[20] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao,
thing. arXiv preprint arXiv:2304.02643 , 2023. 5, 6, 7
[21] Philipp Kr ¨ahenb ¨uhl and Vladlen Koltun. Geodesic object
proposals. In Computer Vision–ECCV 2014: 13th European
Conference, Zurich, Switzerland, September 6-12, 2014,
Proceedings, Part V 13 , pages 725–739. Springer, 2014. 5, 6
[22] Chuyi Li, Lulu Li, Yifei Geng, Hongliang Jiang, Meng
Cheng, Bo Zhang, Zaidan Ke, Xiaoming Xu, and Xiangx-
iang Chu. Yolov6 v3.0: A full-scale reloading, 2023. 3

[23] Jingjing Li, Tianyu Yang, Wei Ji, Jue Wang, and Li
Cheng. Exploring denoised cross-video contrast for weakly-
supervised temporal action localization. In CVPR , pages
19914–19924, 2022. 5, 6, 7
[24] Tsung-Yi Lin, Piotr Doll ´ar, Ross Girshick, Kaiming He,
Bharath Hariharan, and Serge Belongie. Feature pyra-
mid networks for object detection. In Proceedings of the
IEEE conference on computer vision and pattern recogni-
tion, pages 2117–2125, 2017. 3
[25] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,
Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C Lawrence
Zitnick. Microsoft coco: Common objects in context. In
Computer Vision–ECCV 2014: 13th European Conference,
Zurich, Switzerland, September 6-12, 2014, Proceedings,
Part V 13 , pages 740–755. Springer, 2014. 1, 5

[26] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,
Part V 13 , pages 740–755. Springer, 2014. 7
[27] Nian Liu, Ni Zhang, Ling Shao, and Junwei Han. Learning
selective mutual attention and contrast for rgb-d saliency de-
tection. IEEE Transactions on Pattern Analysis and Machine
Intelligence , 44(12):9026–9042, 2021. 7
[28] David Martin, Charless Fowlkes, Doron Tal, and Jitendra
Malik. A database of human segmented natural images
and its application to evaluating segmentation algorithms and
measuring ecological statistics. In Proceedings Eighth IEEE

International Conference on Computer Vision. ICCV 2001 ,
volume 2, pages 416–423. IEEE, 2001. 1, 4
[29] Pedro O O Pinheiro, Ronan Collobert, and Piotr Doll ´ar.
Learning to segment object candidates. Advances in neural
information processing systems , 28, 2015. 5, 6
[30] Mengyang Pu, Yaping Huang, Yuming Liu, Qingji Guan,
and Haibin Ling. Edter: Edge detection with transformer.
InProceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition , pages 1402–1412, 2022. 5[31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya
Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry,
Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning
transferable visual models from natural language supervi-
sion. In International conference on machine learning , pages
8748–8763. PMLR, 2021. 3, 4

[32] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.
Faster r-cnn: Towards real-time object detection with region
proposal networks. Advances in neural information process-
ing systems , 28, 2015. 5
[33] Irwin Sobel, Gary Feldman, et al. A 3x3 isotropic gradient
operator for image processing. a talk at the Stanford Artifi-
cial Project in , pages 271–272, 1968. 4
[34] Jasper RR Uijlings, Koen EA Van De Sande, Theo Gev-
ers, and Arnold WM Smeulders. Selective search for ob-
ject recognition. International journal of computer vision ,
104:154–171, 2013. 5, 6
[35] Chien-Yao Wang, Alexey Bochkovskiy, and Hong-
Yuan Mark Liao. YOLOv7: Trainable bag-of-freebies sets
new state-of-the-art for real-time object detectors. arXiv
preprint arXiv:2207.02696 , 2022. 3
[36] Zhenyu Wang, Yali Li, Xi Chen, Ser-Nam Lim, Antonio Tor-

ralba, Hengshuang Zhao, and Shengjin Wang. Detecting ev-
erything in the open world: Towards universal object detec-
tion. In Proceedings of the IEEE/CVF Conference on Com-
puter Vision and Pattern Recognition , pages 11433–11443,
2023. 5
[37] Saining Xie and Zhuowen Tu. Holistically-nested edge de-
tection. In Proceedings of the IEEE international conference
on computer vision , pages 1395–1403, 2015. 5
[38] C Lawrence Zitnick and Piotr Doll ´ar. Edge boxes: Lo-
cating object proposals from edges. In Computer Vision–
ECCV 2014: 13th European Conference, Zurich, Switzer-
land, September 6-12, 2014, Proceedings, Part V 13 , pages
391–405. Springer, 2014. 5, 6FASTER SEGMENT ANYTHING :
TOWARDS LIGHTWEIGHT SAM FOR MOBILE APPLICATIONS
A P REPRINT
Chaoning Zhang∗
Kyung Hee UniversityDongshen Han
Kyung Hee UniversityYu Qiao

A P REPRINT
Kyung Hee UniversityJung Uk Kim
Kyung Hee University
Sung-Ho Bae
Kyung Hee UniversitySeungkyu Lee
Kyung Hee UniversityChoong Seon Hong
June 27, 2023
ABSTRACT
Segment anything model (SAM) is a prompt-guided vision foundation model for cutting out the
object of interest from its background. Since Meta research team released the SA project, SAM has
attracted significant attention due to its impressive zero-shot transfer performance and high versatility
of being compatible with other models for advanced vision applications like image editing with
fine-grained control. Many of such use cases need to be run on resource-constraint edge devices,
like mobile Apps. In this work, we aim to make SAM mobile-friendly by replacing the heavyweight

image encoder with a lightweight one. A naive way to train such a new SAM as in the original SAM
paper leads to unsatisfactory performance, especially when limited training sources are available.
We find that this is mainly caused by the coupled optimization of the image encoder and mask
decoder, motivated by which we propose decoupled distillation. Concretely, we distill the knowledge
from the image encoder ViT-H in the original SAM to a lightweight image encoder, which can be
automatically compatible with the mask decoder in the original SAM. The training can be completed
on a single GPU within less than one day, and the resulting lightweight SAM is termed MobileSAM
which is more than 60 times smaller yet performs on par with the original SAM. For inference speed,

MobileSAM runs around 10ms per image: 8ms on the image encoder and 2ms on the mask decoder.
With superior performance and a higher versatility, our MobileSAM is 7 times smaller and 4 times
faster than the concurrent FastSAM, making it more suitable for mobile applications. The code for
MobileSAM project is provided at https://github.com/ChaoningZhang/MobileSAM.
ChatGPT Zhang et al. [2023a] has revolutionized the NLP field, marking a breakthrough in generative AI (AIGC,
a.k.a Artificial intelligence generated content) Zhang et al. [2023b]. What has made this possible lies in GPT-series
models Brown et al. [2020], Radford et al. [2018, 2019], which are foundation models Bommasani et al. [2021] trained

on web-scale text datasets. Following the success of foundation models in NLP, multiple works He et al. [2020],
Qiao et al. [2023a], Zhang et al. [2022a] have learned an image encoder together with a text encoder via contrastive
learning He et al. [2020], Zhang et al. [2022b]. Very recently, Meta Research team has released the "Segment Anything"
project Kirillov et al. [2023], where a prompt-guided vision foundation termed SAM has been proposed and is believed
to be a GPT moment for vision. SAM consists of two components: ViT-based image encoder and prompt-guided mask
decoder, which work in sequence (see Figure 1).
Since its advent, SAM has attracted significant attention for multiple reasons. First, it is the first to show that vision
can follow NLP to pursue a path that combines foundation model with prompt engineering. Second, it is the first to

perform label-free segmentation, a fundamental vision task that is in parallel to label prediction Zhang et al. [2023c].
∗You are welcome to contact the authors through chaoningzhang1990@gmail.comarXiv:2306.14289v1  [cs.CV]  25 Jun 2023MobileSAM -1
image 
encoder
(632M)ViT-based image encoder 
mask decoder
(3.87M) prompt -guided mask decoder
prompt encoder
(0.006M) 
imagePromptimage 
embedding
HeavyweightLightweight
Figure 1: The overview of Segment Anything Model.
Moreover, this fundamental task makes SAM compatible with other models to realize advanced vision applications,
like text-guided segmentation and image editing with fine-grained control. Many of such use cases, however, need
to be run on resource-constrained edge-devices, like mobile apps. As shown in the official demo, with a processed

image embedding, the SAM can work on resource-constrained devices because the mask decoder is lightweight. What
makes the SAM pipeline computation heavy lies in the huge image encoder. In this work, we investigate how to obtain
a lightweight SAM suitable for resource-constrained mobile devices, which is therefore termed MobileSAM.
Table 1: Parameters SAM with different image encoders.
Parameters SAM (ViT-H) SAM (ViT-L) SAM (ViT-B)
ViT-based encoder 632M 307M 86M
prompt-guided encoder 0.006M 0.006M 0.006MGiven that the default image encoder
in the SAM is based on ViT-H, a
straightforward way to obtain Mobile-
SAM is to follow the official pipeline
in Kirillov et al. [2023] to retrain a
new SAM with a smaller image en-
coder like replacing the ViT-H with a
smaller ViT-L or even smaller ViT-B.

The parameters of SAM with different variants of image encoder are summarized in Table 3. As stated in Kirillov
et al. [2023], training a new SAM with ViT-L or ViT-B as the image encoder requires 128 GPUs for multiple days.
Such resource-intensive retraining can be a non-trivial burden to reproduce or improve their results. This optimization
difficulty mainly comes from the coupled optimization of the image encoder and mask decoder. Motivated by this
understanding, we propose to decouple the optimization of the image encoder and mask decoder. Concretely, we
first distill the knowledge from the default image encoder ViT-H to a tiny ViT. After that, we can finetune the mask
decoder in the original SAM to better align with the distilled image encoder. It is worth highlighting that the alignment

optimization is optional because the fact that the lightweight image encoder is distilled from the default image encoder
guarantees its inherent alignment with the default mask decoder.
By turning the problem of seeking a new SAM pipeline into a decoupled distillation, our approach has the advantage of
being simple, and effective, while being reproducible at a low cost (on a single GPU with less than a day). The resulting
MobileSAM reduces the encoder parameters by 100 times and total parameters by 60 times yet. Surprisingly, such a
lightweight MobileSAM performs on par with the original heavyweight SAMs, which constitutes a significant step
for pushing SAM for mobile applications. For the inference with MobileSAM, a single image takes runs only around
10ms: 8ms on the image encoder and 2ms on the mask decoder. It is worth highlighting that our MobileSAM is 7 times

smaller and 4 times faster than the concurrent FastSAM Zhao et al. [2023], while achieving superior performance.
2 Related work
SAM: generalization and versatility. Since its advent in early April of this year, numerous projects and papers have
emerged to investigate SAM from different perspectives. Given that SAM claims to segment anything, a line of works
has reported its performance in real-world situations, including medical images Ma and Wang [2023], Zhang et al.
[2023d], camouflaged objects Tang et al. [2023], and transparent objects Han et al. [2023]. The findings consistently
show that SAM works well in general setups, but not in the above challenging setups. Another significant research
direction has focused on enhancing SAM to improve its practicality. Attack-SAM Zhang et al. [2023e] has shown that

the output masks of SAM can be easily manipulated by adversarial attacks through maliciously generated adversarial
perturbations. Another work Qiao et al. [2023b] further conducts a comprehensive robustness evaluation of SAM,
ranging from style transfer and common corruptions to local occlusion and adversarial perturbation. It is found in Qiao
et al. [2023b] SAM has high robustness but not for adversarial perturbation, which aligns well with the finding in Zhang
2et al. [2023e]. Another line of work focuses on demonstrating the versatility of SAM. Grounded SAM IDEA-Research
[2023] is the pioneering work to combine Grounding DINO Liu et al. [2023a] with SAM for segmenting anything with
text inputs. Specifically, it relies on Grounding DINO to generate a bounding box from text and then the generated

box can be used as a prompt to segment the mask. SAM predicts masks without labels and multiple works Chen
et al. [2023], Park [2023] combine SAM with other models such as CLIP Radford et al. [2021] to semantically
segment anything. Beyond object segmentation, multiple works have also shown its versatility in other fields, including
image editing Rombach et al. [2022], as well as inpainting tasks Yu et al. [2023], object tracking within videos Yang
et al. [2023], Zxyang [2023]. Beyond 2D vision, the investigation of SAM has also been extended to 3D object
reconstruction Shen et al. [2023], Kang et al. [2022], demonstrating its capabilities in assisting 3D model generation
from a single image. For a complete survey of SAM, the readers are suggested to refer to Zhang et al. [2023c].

ViT: lightweight and efficient. Early mobile vision applications have been mainly powered by lightweight CNNs,
such as MobileNet Howard et al. [2017] and its improved varinats Sandler et al. [2018], Howard et al. [2019]. The core
idea of MobileNet lies in separating a normal convolution block into depth-wise convolution and point-wise convolution,
which significantly reduces the mode parameters and computation time. Since the advent of ViT Dosovitskiy et al.
[2021], numerous works have attempted to make it lightweight and efficient. Complementary to ViT-Huge (ViT-H),
ViT-Large (ViT-L), ViT-Base (ViT-B) in the original ViT paper Dosovitskiy et al. [2021], smaller ViTs are introduced
in Touvron et al. [2020] and are denoted as Deit-Small (Deit-S) and Deit-Tiny (Deit-T) ViT-Small and ViT-Tiny.

MobileViT Mehta and Rastegari [2021] is a pioneering work to combine ViT with standard convolutions for improving
its performance, which outperforms MobileNet v2 Sandler et al. [2018]. The main motivation is to exploit the local
representation capability of CNN, and this practice is followed by multiple follow-up works which aim to enhance the
model speed, including EfficientFormer Li et al. [2022a], EfficientViT Liu et al. [2023b], Next-ViT Li et al. [2022b]
and Tiny-ViT Wu et al. [2022]. The recent progress in lightweight and faster ViT is complementary to our proposed
decoupled distillation towards making the next-generation SAM suitable for resource-constrained mobile devices.
3 Mobile-Friendly SAM
3.1 Background and Project Goal
Background on SAM. Here, we first summarize the structure of SAM and how it works. SAM consists of a ViT-based

image encoder and a prompt-guided mask decoder. The image encoder takes the image as the input and generates an
embedding, which is then fed to the mask decoder. The mask decoder generates a mask to cut out any object from the
background based on a prompt like a point (or box). Moreover, SAM allows generating multiple masks for the same
prompt for addressing the ambiguity issue, which provides valuable flexibility. Considering this, this work maintains
the pipeline of SAM to first adopt a ViT-based encoder to generate image embedding and then to adopt a prompt-guided
decoder to generate the desired mask. This pipeline is optimally designed for the “segment anything", which can be
used for the downstream task of “segment everything" (see Sec. 4.3 for more discussion).

Project goal. The goal of this project is to generate a mobile-friendly SAM (MobileSAM) that achieves satisfactory
performance in a lightweight manner and is much faster than the original SAM. The prompt-guided mask decoder
in the original SAM has less than 4M parameters and is thus considered lightweight. Given an image embedding
processed by the encoder, as shown in their public demo, SAM can work in resource-constrained devices since the
mask decoder is lightweight. However, the default image encoder in the original SAM is based on ViT-H with more
than 600M parameters, which is very heavyweight and makes the whole SAM pipeline incompatible with mobile
devices. Therefore, the key to obtaining a mobile-friendly SAM lies in replacing the heavyweight image encoder with
a lightweight one, which also automatically keeps all its functions and characteristics of the original SAM. In the

following, we elaborate on our proposed method for achieving this project goal.
3.2 Proposed Method
Coupled distillation. A straightforward way to realize our project goal is to follow the official pipeline in Kirillov
et al. [2023] to retrain a new SAM with a smaller image encoder. As stated in Kirillov et al. [2023], training a SAM with
ViT-H image encoder requires takes 68 hours on 256 A100 GPUs. Replacing the ViT-H with ViT-L or ViT-B reduces
the required GPUs to 128, which is still a non-trivial burden for many researchers in the community to reproduce or
improve their results. Following their approach, we can further adopt an even smaller image encoder and retrain a new
SAM with their provided segmentation dataset which is 11-T. Note that the masks in the provided dataset are given by

the pretrained SAM (with the ViT image encoder). In essence, this retraining process is knowledge distillation Hinton
et al. [2015], which transfers the knowledge from a ViT-H-based SAM to a SAM with a smaller image encoder (see
Figure 2 left).
3distillationMobileSAM -2
ViT-based ( large )
image encoder prompt -guided 
mask decoderTeacher SAM
ViT-based ( small ) 
mask decodermask
mask
trainable trainableimag e
distillationMobileSAM -3
mask
trainable frozencopy imageFigure 2: Coupled knowledge distillation of SAM. The left subfigure denotes the fully-coupled distillation, while the
right one represents the semi-coupled distillation.

From semi-coupled to decoupled distillation. When performing a KD from the original SAM to that with a smaller
image encoder, the difficulty mainly lies in a coupled optimization of the image encoder and combined decoder.
Intuitively, the optimization of the image encoder depends on the quality of the image decoder, and vice versa. When the
two modules in the SAM are both in a bad state, it is more challenging to train them both to a good state. Inspired by the
divide-and-conquer algorithm Zhang et al. [2022c], we propose to divide the KD task into two sub-tasks: image encoder
distillation and mask decoder finetuning. Concretely, we first perform the KD on the image encoder by transferring the

knowledge from ViT-H to a smaller encoder. Since the mask decoder in the original SAM is already lightweight, we
plan to keep its architecture. This brings a benefit of a readily used combined decoder for finetuning instead of training
it from scratch. To alleviate the optimization issue of coupled distillation, a straightforward way is to optimize the
image encoder with a copied and frozen mask decoder (see Figure 2 right). The freezing operation can help prevent the
quality of the mask decoder from being deteriorated by a poor image encoder. We call this distillation semi-coupled
because the optimization of the image encoder is still not fully decoupled from the mask decoder. Empirically, we find
that this optimization is still challenging because the choice of a prompt is random, which makes the mask decoder

variable and thus increases the optimization difficulty. Therefore, we propose to distill the small image encoder directly
from the ViT-H in the original SAM without resorting to the combined decoder, which is termed decoupled distillation
(see Figure 3). Another advantage of performing distillation on the image embedding is that we can adopt a simple
MSE loss instead of using a combination of focal loss Lin et al. [2017] and dice loss Milletari et al. [2016] for making
the mask prediction as in Kirillov et al. [2023].
distillationMobileSAM -4
ViT-based ( small )
mask
image
image 
embedding
image 
embeddingcopyFinetuning (optional)
Figure 3: Decoupled distillation for SAM.

mask
image
image 
embedding
image 
On the necessity of mask decoder finetuning. Unlike the semi-coupled distillation, the above decoupled distillation
yields a lightweight image encoder that might not align well with the original frozen mask decoder. Empirically, we
find that this is not true because the generated image encoding from the student image encoder can be sufficiently close
to that of the original teacher encoder, which renders finetuning on the combined decoder in the second stage optional.
4It is expected that finetuning the mask decoder on the frozen lightweight image encoder or jointly finetuning them
together might further improve the performance.
Preliminary evaluation. Here, we conduct a preliminary investigation to compare coupled distillation and decoupled

distillation. Here, for performance evaluation, we compute the mIoU between the two masks generated by the teacher
SAM and student SAM on the same prompt point. Intuitively, a higher mIoU indicates a higher mask prediction
performance by assuming that the mask generated by ViT-H is ground-truth. For the coupled distillation, we adopt the
SAM with ViT-B provided in the original SAM Kirillov et al. [2023]. It was trained on SA-1B (11M images) on 128
GPUs (1 sample per GPU) for 180k iterations. By contrast, in our decoupled distillation setup, we train the model on 2
GPUs (two samples per GPU to save computation resources) on 0.1% samples of SA-1B dataset (11k) images for 55k
iterations. Overall, decoupled distillation takes less than 1% of the computation resources than coupled distillation,

while achieving a superior performance of mIoU of 0.75 vs 0.72 for the coupled sit (averaged on 200 samples). Since
ViT-B is still a non-trivial burden for mobile devices, therefore in the following we experiment with a TinyViT (with
5M parameters) Wu et al. [2022] based on our proposed decoupled distillation.
Table 2: Comparison of coupled distillation and decoupled distillation fro SAM with ViT-B as the image encoder.
Decoupled distillation performs better and requires less than 1% computation resources than coupled distillation.
/ SAM (coupled distillation) SAM (decoupled distillation)
MIoU 0.72 0.75
Training GPUs 128 2
Batch size 128 4
Iterations 180k 55k
Training Data 11M 11K
4 Experiments
4.1 Experimental Setup
Table 3: Parameters of MobileSAM with TinyViT(5M) as the image en-
coder. The original SAM is also reported for comparison. The inference

speed on a single GPU is also reported.
Parameters Original SAM MobileSAM
Image encoder 632M 5.78M
prompt-guided decoder 3.876M 3.876M
Speed 0.452s 0.008sLightweight Image Encoder. The goal
of our project is to obtain an efficient
SAM by replacing the default ViT-H with
a lightweight image encoder for mobile de-
vices. As a ViT-based backbone, ViT-Tiny
has similar parameters as Deit-Tiny but per-
forms better. For example, on ImageNet-1K,
Deit-Yiny achieves an accuracy of 72.2%,
while ViT-Tiny achieves 79.1%. Therefore,
we adopt ViT-Tiny for the proof of concept
to demonstrate the effectiveness of our pro-
posed decoupled distillation for training a lightweight MobileSAM that can be much faster than the original SAM.

The adopted lightweight image encoder consists of four stages which gradually reduce the resolution. The first stage
is constructed by convolution blocks with inverted residuals Sandler et al. [2018], while the remaining three stages
consist of transformer blocks. At the beginning of the model, there are 2 convolutions blocks with a stride of 2 for
downsampling the resolution. The downsampling operation between different stages is processed by convolution blocks
with the stride of 2. Different from Wu et al. [2022], we set the stride of 2 in the last downsampling convolution to 1 for
making the final resolution match that of the ViT-H image encoder of the original SAM. Note that other efficient image
encoders discussed in Section 2 can also be adopted as the image encoder.

Training and evaluation details. For the decoupled KD on the image encoder, we train the lightweight encoder
with 1% of the SA-1B dataset Kirillov et al. [2023] for 8 epochs on a single GPU (RTX3090). We observe that more
computation is spent on the forward process on the teacher image encoder considering that it is significantly more heavy
than our adopted student image encoder (see above). To make the distillation faster, we follow the practice in Wu et al.
[2022] to save the image embeddings beforehand so that we only need to run the forward process once. With a single
GPU, we can obtain our MobileSAM in less than a day. Training our MobileSAM with more GPUs for a longer time is

expected to yield better performance. The initial investigation of performing mask decoder finetuning further improves
the performance of MobileSAM, however, we omit this step in this version of our paper for simplicity. For quantitative
5evaluation of the distilled SAM, we compute the mIoU between the masks predicted by the original SAM and our
MobileSAM.
Figure 4: Mask prediction with a single point as the prompt.
4.2 MobileSAM performs on par with the orignal SAM
For the main results, we report the predicted masks with two types of prompts: point and box. We do not report the
results with text prompt because the official github project of SAM does not provide pretrained models for text-guided
mask decoder. The results with point as the prompt are shown in Figure 4, and those with box as the prompt are shown

in Figure 5. We observe that MobileSAM makes a satisfactory mask prediction similar to that of the original SAM.
Figure 5: Mask prediction with a box as the prompt.
6Table 4: Ablation study on the influence of
training computation on the MobileSAM
performance.
batch size epochs Iterations mIoU
4 2 50k 0.7057
8 4 50k 0.7286
8 8 100k 0.7447Ablation study. Here, we conduct an ablation study on the influence
of the training computation on the performance of SAM. The results
in Table 7 show that, under the same number of iterations, increasing
the batch size increases the model performance. Moreover, under the
batch size, the performance also benefits from more update iterations
by increasing the training epochs. Note that all the experiments are
conducted on a single GPU. We expect that increasing the number of

GPUs for allowing a larger batch size or further increasing the iterations
can further improve the performance.
4.3 MobileSAM outperforms FastSAM in All Aspects
Table 5: Comparison between segment any-
thing and segment everything.
anything everything
# of objects 1 N
prompt-aware yes noSegment anything v.s.segment everything . Note that the title of the
original SAM paper Kirillov et al. [2023] is “segment anything" instead
of “segment everything". As highlighted in Kirillov et al. [2023], SAM
performs the task of promptable segmentation which “returns a valid
segmentation mask given any segmentation prompt" (quote from Kirillov
et al. [2023]). The role of the prompt is to specify what to segment in the
image. In theory, any object can be segmented as long as the prompt is set

properly, therefore, it is called “segment anything". By contrast, “segment
everything" is in essence object proposal generation Kirillov et al. [2023],
for which the prompt is not necessary. In Kirillov et al. [2023], “segment everything" (object proposal generation) is
chosen as one of the downstream tasks for demonstrating its zero-shot transfer performance. To summarize, “segment
anything" solves the foundation task of promptable segmentation for any object, while “segment everything" solves the
downstream task of mask proposal generation for all objects. Since “segment everything" does not necessarily require a
prompt, FastSAM directly generates the mask proposal with YOLO v8 in a prompt-free manner. To enable promptable

segmentation, a mapping algorithm is designed to select the mask from the proposal mask sets. It is worth highlighting
that the follow-up works that evaluate its generalization/robustness or investigate its versatility mainly focus on the
anything instead of everything mode because the former addresses the foundation task. Therefore, the comparison with
FastSAM mainly focuses on “segment anything", but we also provide a comparison regarding “segment everything" for
completeness.
Table 6: Comparison between FastSAM and Mo-
bileSAM.
FastSAM MobileSAM Ratio
Size 68M 9.66M 7
Speed 40ms 10ms 4MobileSAM is faster and smaller. FastSAM consists of a
YOLOv8-based detection branch and a YOLACT-based segmenta-
tion branch to perform a prompt-free mask proposal generation. It
has 68M parameters and takes 40ms to process an image. By con-

trast, MobileSAM has less 10M parameters, which is significantly
smaller. For the inference speed, on a single GPU, it takes 40ms
to process an image while ours only takes 10ms, which is 4 times
faster than FastSAM.
Table 7: mIoU comparison. With the assumption
that the predicted mask from the original SAM
is ground-truth, a higher mIoU indicates a better
100 200 300 400 500
FastSAM 0.27 0.33 0.37 0.41 0.41
MobileSAM 0.73 0.71 0.74 0.73 0.73mIoU comparison under segment anything mode. We further
compare the mIoU between the predicted masks with that of the
original SAM. Note that FastSAM cannot predict the mask with
a single point as the original SAM. Instead, it requires at least
two prompt points: one for the foreground and the other for the

background. The results in Table 6 show the mIoU for FastSAM is
much smaller than that for MobileSAM, suggesting that the mask
prediction of FastSAM is very different from that of the original
SAM. Moreover, the mIoU for the FastSAM decreases very fast
when the distance between the two prompt points. This is mainly
caused by the fact that FastSAM often fails to predict the object when the foreground prompt point is set too close to
the background prompt point.
Results for segment everything. The results for “segment everything" are shown in Figure 6. For completeness, we
also report the results of the original SAM, which generates a pleasing object proposal. We have two major observations.

First, the results of our MobileSAM align surprisingly well with that of the original SAM. By contrast, the results of
FastSAM are often less satisfactory. For example, FastSAM often fails to predict some objects, like the roof in the first
image. Moreover, the mask proposal is sometimes difficult to interpret (see the mask for the stage in the first image and
7Figure 6: Comparison of segment everything results.
that for the sky in the second image). Second, FastSAM often generates masks that have non-smooth boundaries, for
which we suggest the reader zoom in to check the details in Figure 6. For example, the pillars in the third image have
non-smooth boundaries, while the original SAM and our MobileSAM do not have this issue.
5 Conclusion
In this work, we aim to make SAM mobile-friendly by replacing the heavyweight image encoder with a lightweight one.

We find that the naive way to train such a new SAM as in the original SAM paper leads to unsatisfactory performance,
especially under a setup of limited training sources. The coupled optimization of the image encoder and mask decoder
is the reason, and thus we propose decoupled distillation, whhere the knowledge is distilled from the image encoder
ViT-H in the original SAM to a lightweight image encoder. We show that the resulting lightweight image encoder
can be automatically compatible with the mask decoder in the original SAM. Our MobileSAM is more than 60 times
smaller yet performs on par with the original SAM. Moreover, we conduct a comparison with the concurrent FastSAM
and show that MobileSAM achieve superior performance. Our MobileSAM is also 4 times faster and 7 times smaller

than the concurrent FastSAM, making it more suitable for mobile applications. Since our MobileSAM keeps all the
pipeline of the original SAM and just replaces the image encoder, it can be plug-and-play for the existing SAM-based
projects to move from a heavyweight SAM to a lightweight one with zero effort.
Chaoning Zhang, Chenshuang Zhang, Chenghao Li, Yu Qiao, Sheng Zheng, Sumit Kumar Dam, Mengchun Zhang,
Jung Uk Kim, Seong Tae Kim, Jinwoo Choi, et al. One small step for generative ai, one giant leap for agi: A complete
survey on chatgpt in aigc era. arXiv preprint arXiv:2304.06488 , 2023a.
8Chaoning Zhang, Chenshuang Zhang, Sheng Zheng, Yu Qiao, Chenghao Li, Mengchun Zhang, Sumit Kumar Dam,
Chu Myaet Thwal, Ye Lin Tun, Le Luang Huy, et al. A complete survey on generative ai (aigc): Is chatgpt from

gpt-4 to gpt-5 all you need? arXiv preprint arXiv:2303.11717 , 2023b.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan,
Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural
information processing systems , 2020.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by
generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are
unsupervised multitask learners. OpenAI blog , 2019.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein,
Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv
preprint arXiv:2108.07258 , 2021.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual
representation learning. In CVPR , 2020.
Yu Qiao, Md Munir, Apurba Adhikary, Huy Q Le, Avi Deb Raha, Chaoning Zhang, Choong Seon Hong, et al. Mp-fedcl:
Multi-prototype federated contrastive learning for edge intelligence. arXiv preprint arXiv:2304.01950 , 2023a.
Chaoning Zhang, Kang Zhang, Chenshuang Zhang, Trung X Pham, Chang D Yoo, and In So Kweon. How does
simsiam avoid collapse without negative samples? a unified understanding with self-supervised contrastive learning.
InICLR , 2022a.
Chaoning Zhang, Kang Zhang, Trung X. Pham, Changdong Yoo, and In-So Kweon. Dual temperature helps contrastive
learning without many negative samples: Towards understanding and simplifying moco. In CVPR , 2022b.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer
Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643 , 2023.
Chaoning Zhang, Sheng Zheng, Chenghao Li, Yu Qiao, Taegoo Kang, Xinru Shan, Chenshuang Zhang, Caiyan Qin,
Francois Rameau, Sung-Ho Bae, et al. A survey on segment anything model (sam): Vision foundation model meets
prompt engineering. 2023c.
Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment
anything. arXiv preprint arXiv:2306.12156 , 2023.
Jun Ma and Bo Wang. Segment anything in medical images. arXiv preprint arXiv:2304.12306 , 2023.
Yizhe Zhang, Tao Zhou, Peixian Liang, and Danny Z Chen. Input augmentation with sam: Boosting medical image

segmentation with segmentation foundation model. arXiv preprint arXiv:2304.11332 , 2023d.
Lv Tang, Haoke Xiao, and Bo Li. Can sam segment anything? when sam meets camouflaged object detection. arXiv
preprint arXiv:2304.04709 , 2023.
Dongsheng Han, Chaoning Zhang, Yu Qiao, Maryam Qamar, Yuna Jung, SeungKyu Lee, Sung-Ho Bae, and
Choong Seon Hong. Segment anything model (sam) meets glass: Mirror and transparent objects cannot be easily
detected. arXiv preprint , 2023.
Chenshuang Zhang, Chaoning Zhang, Taegoo Kang, Donghun Kim, Sung-Ho Bae, and In So Kweon. Attack-sam:
Towards evaluating adversarial robustness of segment anything model. arXiv preprint , 2023e.
Yu Qiao, Chaoning Zhang, Taegoo Kang, Donghun Kim, Shehbaz Tariq, Chenshuang Zhang, and Choong Seon Hong.
Robustness of sam: Segment anything under corruptions and beyond. arXiv preprint arXiv:2306.07713 , 2023b.

IDEA-Research. Grounded segment anything, 2023. URL https://github.com/IDEA-Research/
Grounded-Segment-Anything . GitHub repository.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun
Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint
arXiv:2303.05499 , 2023a.
Jiaqi Chen, Zeyu Yang, and Li Zhang. Semantic-segment-anything, 2023. URL https://github.com/
fudan-zvg/Semantic-Segment-Anything . GitHub repository.
Curt Park. segment anything with clip, 2023. URL https://github.com/Curt-Park/
segment-anything-with-clip . GitHub repository.
9Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda
Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In

ICML , 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image
synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition , pages 10684–10695, 2022.
Tao Yu, Runseng Feng, Ruoyu Feng, Jinming Liu, Xin Jin, Wenjun Zeng, and Zhibo Chen. Inpaint anything: Segment
anything meets image inpainting. arXiv preprint arXiv:2304.06790 , 2023.
Jinyu Yang, Mingqi Gao, Zhe Li, Shang Gao, Fangjing Wang, and Feng Zheng. Track anything: Segment anything
meets videos. arXiv preprint arXiv:2304.11968 , 2023.
Zxyang. Segment and track anything, 2023. URL https://github.com/z-x-yang/
Segment-and-Track-Anything . GitHub repository.
Qiuhong Shen, Xingyi Yang, and Xinchao Wang. Anything-3d: Towards single-view anything reconstruction in the

wild. arXiv preprint arXiv:2304.10261 , 2023.
Minki Kang, Dongchan Min, and Sung Ju Hwang. Any-speaker adaptive text-to-speech synthesis with diffusion models.
arXiv preprint arXiv:2211.09383 , 2022.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto,
and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv
preprint arXiv:1704.04861 , 2017.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted
residuals and linear bottlenecks. In CVPR , 2018.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu,
Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international

conference on computer vision , pages 1314–1324, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner,
Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image
is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training
data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012.12877 , 2020.
Sachin Mehta and Mohammad Rastegari. Mobilevit: light-weight, general-purpose, and mobile-friendly vision
transformer. arXiv preprint arXiv:2110.02178 , 2021.
Yanyu Li, Geng Yuan, Yang Wen, Ju Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, and Jian Ren.

Efficientformer: Vision transformers at mobilenet speed. Advances in Neural Information Processing Systems , 35:
12934–12949, 2022a.
Xinyu Liu, Houwen Peng, Ningxin Zheng, Yuqing Yang, Han Hu, and Yixuan Yuan. Efficientvit: Memory efficient
vision transformer with cascaded group attention. In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition , pages 14420–14430, 2023b.
Jiashi Li, Xin Xia, Wei Li, Huixia Li, Xing Wang, Xuefeng Xiao, Rui Wang, Min Zheng, and Xin Pan. Next-
vit: Next generation vision transformer for efficient deployment in realistic industrial scenarios. arXiv preprint
arXiv:2207.05501 , 2022b.
Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining

distillation for small vision transformers. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv,
Israel, October 23–27, 2022, Proceedings, Part XXI , pages 68–85. Springer, 2022.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint
arXiv:1503.02531 , 2015.
Chaoning Zhang, Kang Zhang, Chenshuang Zhang, Axi Niu, Jiu Feng, Chang D Yoo, and In So Kweon. Decoupled
adversarial contrastive learning for self-supervised adversarial robustness. In ECCV , pages 725–742. Springer, 2022c.
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In
Proceedings of the IEEE international conference on computer vision , pages 2980–2988, 2017.

Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric
medical image segmentation. In 2016 fourth international conference on 3D vision (3DV) , pages 565–571. Ieee,
2016.
10