  Stakeholders constantly make assumptions in the development of deep learning
(DL) frameworks. These assumptions are related to various types of software
artifacts (e.g., requirements, design decisions, and technical debt) and can
turn out to be invalid, leading to system failures. Existing approaches and
tools for assumption management usually depend on manual identification of
assumptions. However, assumptions are scattered in various sources (e.g., code
comments, commits, pull requests, and issues) of DL framework development, and
manually identifying assumptions has high costs (e.g., time and resources). To
overcome the issues of manually identifying assumptions in DL framework
development, we constructed a new and largest dataset (i.e., AssuEval) of
assumptions collected from the TensorFlow and Keras repositories on GitHub;
explored the performance of seven traditional machine learning models (e.g.,
Support Vector Machine, Classification and Regression Trees), a popular DL
model (i.e., ALBERT), and a large language model (i.e., ChatGPT) of identifying
assumptions on the AssuEval dataset. The experiment results show that: ALBERT
achieves the best performance (f1-score: 0.9584) of identifying assumptions on
the AssuEval dataset, which is much better than the other models (the 2nd best
f1-score is 0.6211, achieved by ChatGPT). Though ChatGPT is the most popular
large language model, we do not recommend using it to identify assumptions in
DL framework development because of its low performance on the task.
Fine-tuning ChatGPT specifically for assumption identification could improve
the performance. This study provides researchers with the largest dataset of
assumptions for further research (e.g., assumption classification, evaluation,
and reasoning) and helps practitioners better understand assumptions and how to
manage them in their projects.
