Visual Grounding API: Production-Grade Visual Localization Service Based on LLaVA

The visual-grounding-api project introduced in this article is a production-grade visual localization service based on LLaVA-1.5-7B and LoRA fine-tuning technology. Its core innovation lies in replacing text coordinate parsing with an MLP regression head, solving problems such as inconsistent formats and hallucinations in traditional solutions. On the RefCOCO test set, it improves IoU accuracy by 297.5% compared to the baseline. The project also provides a complete FastAPI service, an interactive demo interface, and a Docker containerization deployment solution, achieving the combination of academic research and engineering practice.

Visual localization is a core task of multimodal AI, which requires locating object bounding boxes based on images and text descriptions, and is applied in scenarios such as image search and intelligent monitoring. Traditional two-stage methods (detection + matching) have fixed category limitations; although multimodal large models (such as LLaVA) have visual understanding capabilities, converting them into precise bounding boxes faces three major challenges: fragility of text parsing (inconsistent formats, coordinate out-of-bounds), trade-off between accuracy and efficiency (high cost of full-parameter fine-tuning), and complexity of production deployment (model optimization, service framework, etc.).

The core innovation of the project is the MLP regression head design: extract the 4096-dimensional hidden state at the [LOC] token position of LLaVA, connect it to a lightweight MLP (4096→512→256→4, GELU+Dropout+Sigmoid), and output normalized coordinates. Advantages include end-to-end differentiability, format guarantee ([0,1] range), and lightweight efficiency. Training uses LoRA fine-tuning: rank 16, alpha 32, target modules are q_proj/v_proj/k_proj, with only 0.14% of parameters trainable; the loss function is an equal-weight combination of L1 and GIoU, balancing position accuracy and overlap optimization.

The experiment is based on the RefCOCO dataset (48,190 samples). The results show: the baseline (LLaVA + text parsing) has an IoU of 0.097; the ablation experiment (freeze LLaVA and only train MLP) improves IoU to 0.284; the main experiment (LoRA + MLP) achieves an IoU of 0.386, an increase of 297.5%. In terms of performance, the inference latency of the main model on A100 is only 78.5ms (baseline is 312.7ms). Bias analysis shows that the IoU of large objects (0.473) is better than medium (0.296) and small objects (0.119); under different thresholds, the accuracy of IoU>0.1 is 75.8%, and IoU>0.75 is only 8.3%.

The project provides a complete production deployment solution: 1. FastAPI service: includes endpoints such as /predict (image + text → bounding box), /health (health check), and /models (model list); 2. Interactive demo: Gradio to compare multi-model results, React Web UI as a production-grade front-end; 3. Docker containerization: based on CUDA12.8/PyTorch2.11, one-click build and deployment. In addition, it includes analysis tools (bias audit, failure case identification, latency benchmark) and CI/CD processes (GitHub Actions, environment check).

This project is a model of the combination of academia and engineering. It solves core problems through architectural innovation (MLP + LoRA) and achieves dual improvements in accuracy and efficiency. Industry insights include: architectural innovation is better than blind stacking, the value of LoRA efficient fine-tuning, and the importance of end-to-end optimization. Potential application scenarios include intelligent image editing, VQA enhancement, auxiliary vision systems, e-commerce product localization, content review, etc., providing high-quality references for the implementation of multimodal AI.