Modal Cloud Deployment of Multimodal LLM: A New Image Understanding Solution with InternVL + LMDeploy

This article introduces a multimodal large language model application solution based on the Modal.com platform. By combining the InternVL vision model and LMDeploy inference framework, it achieves cloud-based image understanding and text generation capabilities, providing developers with a low-threshold, highly available multimodal AI deployment solution. It addresses challenges in traditional deployment such as expensive GPU resources, difficulty in elastic scaling, complex inference optimization, and high operation and maintenance costs.

As multimodal models like GPT-4V and Gemini demonstrate strong image understanding capabilities, developer demand is growing, but deployment faces many challenges: expensive GPU resources and difficulty in elastic scaling, complex inference optimization, and high operation and maintenance costs; traditional self-built server solutions require large initial investment and maintenance by a professional MLOps team. Cloud serverless platforms (such as Modal.com) run AI models in a function-as-a-service manner, with pay-as-you-go and automatic scaling, making them suitable for compute-intensive multimodal AI scenarios with uncertain call frequencies.

The project uses InternVL, an open-source model from the Shanghai Artificial Intelligence Laboratory, as the vision-language model. Its innovative architecture decouples the visual encoder from the large language model, achieves cross-modal alignment through a learnable query transformer, supports high-resolution image input while maintaining inference efficiency; compared to closed-source models, it provides fully controllable deployment and lower costs. Inference acceleration relies on the LMDeploy toolset, with optimizations including: continuous batching (dynamically inserting requests to improve throughput), paged attention (paged management of KV cache to reduce fragmentation), and quantization support (AWQ/GPTQ compresses model size to 1/4, reducing memory usage and latency). Combined with Modal's elastic GPU resources, it achieves efficient inference.

The deployment process is simple: configure the Modal API key, run the deployment script, and Modal automatically handles underlying details such as container image building, GPU instance allocation, and load balancing. After the service is started, users can call it via HTTP API or Python SDK: upload image files/URLs, specify prompts, and receive generated text; streaming output is supported for real-time interaction. Billing is based on actual GPU usage time, with low development and testing costs, and automatic scaling in production environments to avoid resource waste.

Application scenarios include: content moderation (identifying sensitive image information), e-commerce (generating product descriptions/answering visual questions), and education (analyzing charts/formulas/handwritten content). The architecture is highly scalable: it can be replaced with models that support LMDeploy such as LLaVA/Qwen-VL; integrate with business systems via Modal Webhooks; modify inference code to add preprocessing/postprocessing logic to meet custom needs.

This project demonstrates the potential of combining open-source multimodal models (InternVL), inference optimization tools (LMDeploy), and cloud-native platforms (Modal), enabling individuals/small teams to build enterprise-level multimodal AI services. As multimodal models evolve and cloud services improve, such deployment solutions will become the standard for AI application development, providing an ideal starting point for quickly validating multimodal AI ideas.