NucBench: The First Multimodal Large Language Model Evaluation Benchmark for Nuclear Engineering

NucBench is the first open-source multimodal large language model evaluation benchmark for the nuclear engineering field, developed by the team from the University of Sharjah. It includes approximately 4292 multiple-choice questions from the Reactor Operator License Exam (GFE), over 100 mixed-type questions from undergraduate nuclear engineering exams, and a two-phase flow regime image recognition dataset, aiming to provide a standardized test for evaluating LLMs' knowledge mastery and reasoning abilities in the nuclear engineering field.

Nuclear engineering is a highly specialized field with extremely high safety requirements, involving complex knowledge systems such as reactor physics and thermal-hydraulics. Existing general evaluation benchmarks (e.g., MMLU, GSM8K) lack in-depth coverage of professional engineering fields. Nuclear engineering requires models to have abilities like solving quantitative calculations and understanding visual information, hence NucBench came into being.

It includes three types of tasks: 1. GFE Exam: Approximately 4292 multiple-choice questions from the U.S. NRC, covering PWR/BWR reactor types; 2. Undergraduate Nuclear Engineering Exams: Over 100 mixed-type questions covering 6 core subfields such as reactor thermal-hydraulics and physics; 3. Two-phase Flow Regime Image Recognition: From the Texas A&M University dataset, including 4 flow regime categories like bubbly flow and slug flow.

The objective is to comprehensively evaluate the abilities of multimodal LLMs in the nuclear engineering field, such as knowledge breadth, reasoning depth, multimodal understanding, professional context adaptation, and numerical accuracy, covering comprehensive assessment from basic physics to engineering practice.

It fills the gap in LLM evaluation for professional engineering fields. It is valuable for model developers (standardized testing platform), practitioners (reliability evaluation of AI tools), educational institutions (AI-assisted teaching benchmark), and safety assessment (preliminary screening mechanism), providing a reference for benchmark development in other engineering fields.

Currently, there are issues such as small question scale, limited question types (mainly multiple-choice), and insufficient field coverage (focusing on reactor engineering). In the future, it can expand the question scale, add open-ended questions/auto-scoring question types, cover fields like nuclear fuel cycle, and update regularly.

The dataset has a clear structure. The code repository includes directories such as exams, images, and docs. It uses the CC BY 4.0 license, allowing free use, modification, and redistribution, promoting collaboration and reproducibility in nuclear engineering AI research.