Multimodal Housing Price Prediction: A Regression Model Fusing CNN Visual Features and Structured Data

This article introduces a multimodal machine learning project whose core is to build a housing price prediction model by fusing house image features extracted via convolutional neural networks (CNN) with traditional structured data (area, location, building age, etc.), demonstrating the application value of multimodal learning in real estate valuation. Through simulating the image generation and feature extraction process, the project verifies that the fusion model is more accurate than unimodal models, providing new ideas for housing price prediction.

Housing price prediction is a classic regression problem. Traditional methods rely on structured data (area, building age, etc.) but ignore the visual information of houses—such as hard-to-quantify factors like decoration level and maintenance status. Multimodal learning can construct a more comprehensive house profile by utilizing both visual and structured information, addressing the limitations of single-modal approaches.

Data Preparation and Preprocessing

• Structured Data: Use 8 features from the California Housing Dataset (median income, building age, etc.), standardized via StandardScaler.
• Visual Feature Extraction: Extract image feature vectors using a simplified CNN or pre-trained models (e.g., ResNet).
• Feature Fusion: Adopt an early fusion strategy to concatenate visual features with structured features.

Model Architecture

• The fused features are input into a multi-layer regression network, including fully connected layers, batch normalization, and Dropout layers, outputting the predicted housing price.
• The loss function is Mean Squared Error (MSE), and the optimizer is Adam.

Compare three models:

Model	MAE	RMSE	R²	Description
Tabular Data Baseline	High	High	Medium	Uses only structured features
CNN Visual Features	Medium	Medium	Medium	Uses only image features
Multimodal Fusion	Lowest	Lowest	Highest	Fuses both modalities

The results show that the fusion model has smaller errors and better fitting. Visual features effectively complement the shortcomings of structured data, such as distinguishing differences in decoration levels.

Development Environment

Dependencies include PyTorch, Scikit-learn, Pandas, and other tools.

Code Structure

Organized in Jupyter Notebook: Data Loading → Preprocessing → CNN Feature Extraction → Fusion → Training → Evaluation.

Extensibility

Supports replacing datasets (e.g., Zillow), upgrading CNN models (e.g., EfficientNet), and trying other fusion strategies (e.g., attention fusion).

• Real Estate Valuation: Automatic valuation reduces labor costs; anomaly detection identifies properties with deviated pricing.
• Investment Decision-Making: Evaluate renovation potential, quantify decoration ROI, and optimize investment portfolios.
• Academic Research: Provides a foundation for cross-modal learning, explainable AI, and other directions.

Current Limitations

• Uses simulated images instead of real photos, lacking large-scale annotated data;
• CNN extracts global features, missing local details;
• The model only identifies correlations, not causal relationships.

Future Improvements

• Introduce more modalities (panoramic images, floor plans, street views);
• Use Vision Transformer or attention mechanisms to improve visual feature capture capabilities;
• Develop real-time valuation APIs and interactive tools.

This project verifies the advantages of multimodal fusion in housing price prediction. Its technology can be transferred to scenarios such as medical diagnosis (images + medical records) and product recommendation (images + attributes). With the development of large multimodal models, pre-trained models may be used directly for end-to-end prediction in the future, but understanding the fusion principle remains the foundation for building reliable AI systems.