WildMatch: A Zero-Shot Wildlife Species Recognition System Based on Visual Language Models

WildMatch is an innovative zero-shot wildlife species classification system that combines Visual Language Models (VLM) and LLM-enhanced taxonomic knowledge bases. It eliminates the need for labeled training data, providing an efficient tool for ecological monitoring and biodiversity research. This system addresses key challenges in traditional wildlife image recognition methods.

Camera traps generate thousands of images daily, but manual identification is time-consuming. Supervised learning methods require large labeled datasets, which are hard to obtain for rare species or newly discovered ones. WildMatch's core innovation lies in its zero-shot approach, using natural language descriptions of species instead of labeled data.

WildMatch offers five zero-shot recognition strategies:

• Pure LLM: Uses LLM to build a species knowledge base from Wikipedia, generates image descriptions via VLM, matches with knowledge base, and uses majority voting.
• CLIP-LLM Fusion: Combines CLIP's visual-text similarity with LLM's semantic matching using weighted fusion (α=0.7 default).
• BLIP-LLM Fusion: Similar to CLIP fusion but uses BLIP for visual-text similarity.
• Pure CLIP: Lightweight method using CLIP's embedding similarity without LLM API calls.
• Pure BLIP: Uses BLIP's embedding similarity, no API calls, suitable for offline scenarios.

WildMatch builds its species knowledge base automatically from Wikipedia using LLM, extracting key features (appearance, habitat, behavior). It supports three datasets: Serengeti (Tanzania), WCS (IUCN), and Caltech camera trap data, covering diverse ecosystems.

WildMatch's zero-shot capability brings several benefits:

• Fast adaptation to new species (add Wikipedia description to knowledge base without retraining).
• Recognition of rare species (no need for labeled samples).
• Multilingual support (translate species descriptions to target languages).
• Cost-effective pure visual methods (CLIP/BLIP) for large-scale deployment.

WildMatch represents an important innovation in wildlife species recognition. Its combination of VLM and LLM breaks the dependency on labeled data. The five methods form a spectrum for different needs (accuracy vs cost/speed). As multi-modal AI advances, WildMatch's zero-shot paradigm will play a key role in ecological monitoring and biodiversity conservation.