Panoramic Review of Pathology Visual Language Models: Technological Evolution from Contrastive Learning to Agent Systems

This article organizes the curated resource repository Awesome-Pathology-VLMs in the field of Pathology Visual Language Models (Pathology VLMs). The repository is divided into five categories based on technical routes: contrastive learning/dual encoder, generative/instruction fine-tuning, reasoning enhancement/RL, agent systems, and VLM-enhanced MIL, reflecting the evolution of pathology AI from image-text alignment to complex reasoning and autonomous decision-making. Pathology VLMs aim to solve the time-consuming and labor-intensive problem of manual review of Whole Slide Images (WSI), enabling automated analysis and report generation through cross-modal understanding.

Pathology is the gold standard for medical diagnosis. Digitalization has spawned massive WSI data (billions of pixels per image), and manual review is inefficient and experience-dependent. Visual language models bring the possibility of automated analysis through image-text cross-modal understanding. The unique value of the Awesome-Pathology-VLMs repository lies in its scientific classification system: it not only lists papers and code but also classifies them according to five major technical routes, reflecting the evolutionary context of pathology AI technology.

Technical Route 1 (Contrastive Learning/Dual Encoder): The core is image-text contrastive alignment and shared semantic space. Its advantage is high inference efficiency, suitable for pathological image retrieval, but it is difficult to capture fine-grained interactions. Technical Route 2 (Generative/Instruction Fine-tuning): A mainstream direction with an encoder-decoder architecture. It supports VQA, report generation, and multi-turn dialogue through instruction fine-tuning, which meets clinical needs. Instruction fine-tuning is a key link, converting image-text pairs into instruction formats for training.

Technical Route 3 (Reasoning Enhancement/RL): To solve model hallucinations and reasoning errors, it uses Chain of Thought (CoT) supervision to make the model think step by step. It improves the professionalism of answers through preference optimization such as RLHF/DPO, and RLVR uses verifiable medical knowledge as rewards. Technical Route 4 (Agent Systems): A cutting-edge direction that builds agents capable of autonomous planning and tool calling, simulating human reading habits, and multi-scale collaboration (low-magnification overall assessment + high-magnification detailed observation) to improve diagnostic accuracy and interpretability.

Technical Route 5 (VLM-enhanced MIL): Applying VLM as a feature extractor for WSI classification, predicting slide labels through tile feature aggregation, and using VLM's text generation capability to enhance semantic expression. In terms of data resources, the progression from single-task to large-scale multi-cancer datasets drives model advancement; evaluation benchmarks cover multiple tasks and define scientific assessment methodologies. The repository also sets granularity labels (G1 Tile/G2 ROI/G3 WSI) to support multi-granularity operations.

Current Challenges: Data privacy and ethical constraints limit sharing; image domain migration affects generalization; interpretability and uncertainty quantification need to be addressed. Future Directions: Multi-center data collaboration, fine-grained alignment methods, reliable reasoning verification, and clinical process integration. It is expected to transition from a research tool to a core component of clinical auxiliary diagnosis.