# Intelligent Skin Disease Prediction System Combining Vision Transformer and LLM

> An end-to-end AI-driven system that uses the DeiT Vision Transformer model to analyze skin images and predict potential skin diseases, while generating structured health advice via the Gemini large language model.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-04-15T17:45:35.000Z
- 最近活动: 2026-04-15T17:48:24.057Z
- 热度: 141.9
- 关键词: 皮肤病识别, 视觉Transformer, DeiT, 大语言模型, Gemini, 医疗AI, 计算机视觉, 多模态AI
- 页面链接: https://www.zingnex.cn/en/forum/thread/transformerllm
- Canonical: https://www.zingnex.cn/forum/thread/transformerllm
- Markdown 来源: floors_fallback

---

## [Introduction] Intelligent Skin Disease Prediction System Combining Vision Transformer and LLM

This project builds an end-to-end AI-driven intelligent skin disease prediction system. Addressing the issues of uneven distribution of dermatologist resources and difficulty in early diagnosis, it uses the DeiT Vision Transformer model to analyze skin images and predict disease types, and combines the Gemini large language model to generate structured health advice. The system explicitly states that it is for educational demonstration purposes only, does not constitute medical diagnosis, and aims to assist in preliminary screening and health education.

## Project Background and Motivation

Early identification of skin diseases is crucial for timely treatment, but professional dermatologist resources are unevenly distributed, making it difficult to access timely diagnostic services in many areas. With the rapid development of artificial intelligence technology, the combination of computer vision and large language models has brought new possibilities to this field, leading to the construction of this end-to-end system that analyzes skin images in real time and generates personalized health advice.

## Core Technology and Architecture of the System

The system adopts a layered architecture: Users upload images via the Streamlit frontend → FastAPI backend preprocessing → DeiT-III vision model classification and prediction (transfer learning strategy, fine-tuned on the Kaggle skin disease dataset) → Gemini 2.5 Flash generates treatment advice, follow-up steps, and prevention tips. The DeiT model reduces reliance on large-scale labeled data, and Gemini converts classification labels into actionable guidance information.

## API Design and Deployment Methods

The system provides a RESTful API endpoint POST /api/v1/skin/analyze. Clients upload images in multipart/form-data format (supports jpg/png/jpeg, etc.), and the response includes fields for disease type, confidence level, advice, follow-up steps, and prevention tips. Deployment options include Python virtual environment (launched with uvicorn), Docker image, and Streamlit frontend (locally launched interactive test interface).

## Engineering Highlights and Future Directions

Engineering practice highlights: Separation of API layer and business logic, decoupling of machine learning logic and backend services, loading the model only once at startup to improve inference efficiency, and environment variable configuration management. Future plans: Optimize model inference speed, deploy to AWS cloud infrastructure (EC2/ECS), and establish a complete logging and monitoring system.

## Summary and Reflections

This project demonstrates an innovative application model of multimodal AI in the healthcare field: The vision model is responsible for image understanding, and the LLM is responsible for knowledge integration and language generation, collaborating to form a complete intelligent service. Although it is not a medical diagnosis, its value in health education and preliminary screening assistance is significant, making it an excellent case for learning to responsibly build medical AI applications.