PlantXpert: Benchmarking and Breakthroughs of Multimodal Large Models in Plant Phenotyping Analysis

PlantXpert has built the first multimodal reasoning benchmark for soybean and cotton phenotyping analysis, covering key areas such as pest and disease management, weed control, and yield prediction. Evaluations show that domain fine-tuning can bring significant performance improvements, but quantitative reasoning and cross-crop generalization remain unsolved challenges. This benchmark provides a standardized evaluation framework and research starting point for agricultural AI, promoting the application of multimodal large models in precision agriculture.

Core Value of Phenotyping Analysis

Phenotyping analysis is the bridge connecting genotype and phenotype, requiring systematic measurement of observable crop characteristics (such as plant height, pest and disease severity). Traditional manual methods are time-consuming, labor-intensive, and subjective; with the popularization of high-throughput imaging technology, the demand for automation has become urgent.

Unique Challenges in Plant Science

General multimodal models are difficult to directly apply to the plant field:

1. Deep Domain Knowledge Requirement: Need to understand professional knowledge such as pathogen life cycles and symptom development rules;
2. Fine-grained Visual Recognition: Need to identify subtle spots, discoloration, and other early disease signs on soybean/cotton leaves;
3. Complex Multi-step Reasoning: Need to integrate multi-dimensional information (such as plant density, pest and disease pressure) for causal reasoning.

Dataset Composition

PlantXpert contains 385 digital images and over 3000 test samples, covering four core tasks:

• Disease Diagnosis: Identify and classify soybean/cotton diseases and their severity;
• Pest Monitoring: Detect signs of pest infestation and their damage level;
• Weed Management: Distinguish between crops and weeds, and evaluate competition pressure;
• Yield Prediction: Predict final yield based on growth images. Each sample is equipped with a detailed reasoning chain and evidence annotation to ensure interpretability.

Evaluation Dimensions

Three core dimensions are designed:

1. Visual Professional Ability: Identify key phenotypic features and understand their significance;
2. Quantitative Reasoning Ability: Estimate quantitative indicators such as plant density and lesion coverage;
3. Multi-step Agronomic Reasoning: Integrate visual observations and domain knowledge for multi-step decision-making (e.g., disease type → transmission risk → yield impact → prevention and control recommendations).

The research team evaluated 11 state-of-the-art (SOTA) models and drew the following conclusions:

1. Significant Value of Domain Fine-tuning: General models perform mediocrely in zero-shot/few-shot scenarios; after fine-tuning with soybean/cotton data, their accuracy improved significantly (Qwen3-VL series reached approximately 78% after fine-tuning);
2. Diminishing Marginal Returns of Model Scale: 30B parameter models have limited advantages over 4B parameter models, and the bottleneck is speculated to be insufficient training data in the agricultural field;
3. Unbalanced Cross-crop Generalization: Models trained on a single crop show a significant performance drop when transferred to another crop;
4. Challenges in Quantitative and Biological Reasoning: Pure visual recognition tasks perform well, but quantitative calculations (e.g., lesion area estimation) and deep biological reasoning (e.g., disease transmission dynamics) have high error rates.

Methodological Insights

1. Data Priority Over Scale: Investing in domain-specific training data yields higher returns than expanding model scale;
2. Multi-stage Training Strategy: The three-stage strategy of general pre-training → domain fine-tuning → task optimization is effective;
3. Evaluation-driven Development: A structured evaluation framework can quantitatively identify model shortcomings and guide iterative optimization.

Core Conclusions

PlantXpert demonstrates that multimodal large models can be competent for professional plant phenotyping tasks after adaptation, but still need breakthroughs in quantitative reasoning and cross-domain generalization.

Application Prospects

Agricultural technology companies can use PlantXpert to:

• Evaluate and select suitable models;
• Quickly launch domain adaptation;
• Track model iteration progress. In the long run, it is expected to spawn a new generation of agricultural decision support systems (such as mobile phone photo diagnosis, yield prediction, and management recommendations).

Limitations and Outlook

Limitations: Only covers soybeans and cotton; sample size needs to be expanded; does not involve complex decisions such as irrigation scheduling and fertilization optimization. Future Directions: Expand crop coverage, introduce time-series data to monitor growth dynamics, and integrate multi-data sources such as meteorological/soil sensors.