LithoBench: Evaluation of Multimodal Large Models' Capabilities in Remote Sensing Petrology Interpretation

This article introduces the LithoBench benchmark, which assesses the geological semantic understanding capabilities of large vision-language models in remote sensing petrology interpretation tasks. The benchmark includes 10,000 expert-annotated samples covering five cognitive levels, and experiments reveal that existing models have significant limitations in high-order reasoning tasks.

Remote sensing petrology interpretation is a fundamental task in geological surveys, mineral exploration, and regional geological mapping. It is a highly knowledge-intensive task where experts need to integrate multiple clues such as visual and spectral information to infer rock types. Traditional methods can only handle simple classification and face challenges like large intra-class differences and small inter-class differences. In recent years, multimodal large models have brought opportunities, but there is a lack of benchmarks to comprehensively evaluate their geological semantic understanding capabilities. Existing general benchmarks are too simple and lack expert-level evaluation standards.

LithoBench is a multi-level benchmark specifically designed to evaluate geological semantic understanding in remote sensing petrology interpretation. Its features include: Scale and diversity (10,000 expert-annotated instances, 12 representative rock categories); Comprehensive task types (4,000 multiple-choice questions + 6,000 open-ended questions); Five-level cognitive architecture (Identification and description, Comparative analysis, Mechanism explanation, Practical application, Comprehensive reasoning), which can accurately locate the boundary of model capabilities.

To ensure data validity, an expert-participated, knowledge-driven semi-automated construction process is adopted: Structured geological image description (professional geologists annotate key information such as rock type and texture); Multi-round quality review (at least two experts review independently, and a third arbitrator is introduced for disagreements); Difficulty grading (assign cognitive levels based on expert evaluation).

Evaluation results of mainstream large vision-language models on LithoBench show: High-order reasoning is still a shortcoming (basic identification is acceptable, but mechanism explanation, application, and comprehensive reasoning are insufficient); Insufficient knowledge integration capability (poor performance in handling complex problems with multi-source information); Open-ended questions are more challenging (require correct answers + reasonable explanations).

LithoBench provides a standardized evaluation platform to help objectively assess the capabilities of existing models and point out development directions. It reveals the significant limitations of current multimodal models in handling deep domain knowledge tasks, suggesting that attention should be paid to deep knowledge integration and reasoning capability improvement. In the future, it is expected to become an important tool to promote geological semantic understanding research and help develop multimodal AI systems with expert-level geological knowledge.