Papers
arxiv:2506.10521

Scientists' First Exam: Probing Cognitive Abilities of MLLM via Perception, Understanding, and Reasoning

Published on Jun 12
ยท Submitted by schrodingers-tiger on Jun 17
#2 Paper of the day
Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,

Abstract

Scientists' First Exam (SFE) benchmark assesses scientific cognitive capacities of Multimodal Large Language Models through perception, understanding, and comparative reasoning.

AI-generated summary

Scientific discoveries increasingly rely on complex multimodal reasoning based on information-intensive scientific data and domain-specific expertise. Empowered by expert-level scientific benchmarks, scientific Multimodal Large Language Models (MLLMs) hold the potential to significantly enhance this discovery process in realistic workflows. However, current scientific benchmarks mostly focus on evaluating the knowledge understanding capabilities of MLLMs, leading to an inadequate assessment of their perception and reasoning abilities. To address this gap, we present the Scientists' First Exam (SFE) benchmark, designed to evaluate the scientific cognitive capacities of MLLMs through three interconnected levels: scientific signal perception, scientific attribute understanding, scientific comparative reasoning. Specifically, SFE comprises 830 expert-verified VQA pairs across three question types, spanning 66 multimodal tasks across five high-value disciplines. Extensive experiments reveal that current state-of-the-art GPT-o3 and InternVL-3 achieve only 34.08% and 26.52% on SFE, highlighting significant room for MLLMs to improve in scientific realms. We hope the insights obtained in SFE will facilitate further developments in AI-enhanced scientific discoveries.

Community

Paper submitter

๐Ÿ”ฌ Can MLLMs master complex scientific cognition?

Introducing Scientists' First Exam (SFE) โ€“ the pioneering benchmark that evaluates multimodal large language models across 5 scientific domains and 66 high-value tasks. Unlike traditional tests focused on knowledge understanding, SFEโ€™s 3-layer framework (signal perception โ†’ attribute understanding โ†’ comparative reasoning) challenges models on real scientific data and cross-disciplinary reasoning.

๐Ÿš€ Key Findings:
ยท SOTA models score only ~30% on SFEโ€™s advanced scientific tasks ๐Ÿ“‰ , lagging far behind human expertise ๐Ÿ”
ยท Closed-source models outperform open-source by 6-8%
ยท Models show 10%+ improvement in high-order reasoning (L3) but stagnate in knowledge understanding (L2) ๐Ÿง 
ยท Model scale doesn't always correlate with scientific ability โ€” Qwen2.5-VL-72B even underperforms its smaller version, suggesting scientific data expansion is needed for scaling.

๐Ÿ“Š SFE paves the way for benchmarking AI to drive real scientific discovery. Dive into the future of AI-powered research!

๐Ÿ“ฆ Dataset & Benchmark open-source now.

๐Ÿ“š Read the paper: https://arxiv.org/abs/2506.10521
๐Ÿ‘‰ Explore SFE: https://prismax.opencompass.org.cn/
๐Ÿ” Dataset: https://huggingface.co/datasets/PrismaX/SFE

This comment has been hidden (marked as Off-Topic)

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2506.10521 in a model README.md to link it from this page.

Datasets citing this paper 1

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2506.10521 in a Space README.md to link it from this page.

Collections including this paper 6