3DAeroVLM: The First 3D Vision-Language Benchmark Dataset for Post-Disaster Assessment

3DAeroVLM is the first 3D vision-language benchmark dataset for post-disaster assessment, built on real post-disaster point cloud data from Hurricane Ian (Florida, 2022). It supports seven task types including damage assessment, spatial reasoning, and report generation, providing a standardized evaluation framework for multimodal AI models in UAV post-disaster assessment scenarios, and filling the gap in the evaluation of specialized applications of 3D vision-language models in disaster scenarios.

Rapid and accurate assessment of disaster situations after natural disasters is crucial for rescue decision-making. Traditional manual on-site surveys are inefficient and have safety risks; UAV aerial surveys have become standard, but the need for professionals to interpret massive 3D point cloud data remains a bottleneck. While 2D vision-language models (VLMs) have made significant progress, there is a lack of standardized evaluation benchmarks for 3D spatial understanding, especially for specialized applications in disaster scenarios—thus the birth of the 3DAeroVLM project.

3DAeroVLM is built on the 3DAeroRelief dataset, collected from post-Hurricane Ian scenes in Florida in 2022, and is the first post-disaster assessment benchmark combining 3D point clouds with vision-language instruction pairs. Key statistics: 64 scenes (8 regions), 5 semantic categories (damaged/undamaged buildings, etc.), 924 instruction pairs (809 for training/115 for testing). Building counts were manually annotated using CloudCompare, replacing the earlier DBSCAN clustering (automatic clustering had a bias of 1-2 buildings, with a global overestimation of 456 vs. 297 manual annotations).

Seven task types are designed:

1. Simple counting (128 pairs, e.g., "How many buildings are there in the scene?")
2. Complex counting (198 pairs, e.g., "How many damaged buildings are there?")
3. Existence judgment (256 pairs, e.g., "Is there a road in the scene?")
4. Condition recognition (128 pairs, e.g., "What is the overall damage level?")
5. Comparative reasoning (128 pairs, e.g., "Are there more damaged buildings than undamaged ones?")
6. 3D spatial reasoning (22 pairs, e.g., "How are damaged buildings distributed?", only triggered in scenes with recorded spatial patterns)
7. Proportion analysis (64 pairs, e.g., "What is the dominant land cover type?")

Annotations follow a structured format (fields like scene_id, area, split, etc.). The damage distribution varies significantly across regions: Region 4 has an 89.7% damage rate, Region 5 100% (all 5 buildings damaged), Region 8 23.3%, ensuring that models need to generalize rather than memorize specific scene features.

1. For the first time, VLMs are introduced into high-value post-disaster assessment scenarios, providing an evaluation benchmark for UAV autonomous disaster situation analysis, which is expected to improve response speed;
2. Covers multi-level tasks from simple counting to complex spatial reasoning, comprehensively evaluating 3D spatial understanding capabilities;
3. Real disaster data is representative, and trained models can be transferred to other similar scenarios.

Current limitations:

• Task 8 (structural details) and Task 9 (facade/visibility) are postponed to version 2 (requiring building structural attributes and 2D images);
• The sample size for spatial reasoning tasks is small (22 pairs);
• The multiple-choice format cannot fully reflect the needs of open-ended Q&A. In the future, more diverse tasks, larger samples, and an open-ended answer evaluation framework will be introduced.

As the first 3D vision-language benchmark for post-disaster assessment, 3DAeroVLM fills the gap in standardized evaluation in this field. By combining real post-disaster point clouds with structured instructions, it provides researchers with a platform to test the capabilities of 3D VLMs in disaster response scenarios, promoting academic research and the intelligent upgrade of practical disaster response systems.