In-depth Evaluation of Multi-step Reasoning Capabilities of Small-Parameter Vision-Language Models

This study systematically compares the performance of small vision-language models (VLMs) with 1B-8B parameters and large models on multi-step visual reasoning tasks. It aims to provide empirical evidence for model selection in resource-constrained scenarios (e.g., mobile applications, edge devices) and explore whether small models can handle complex visual reasoning tasks and the gap between them and large models.

Vision-language models (VLMs) have transformed human-computer interaction, but the mainstream trend of pursuing large models (7B+ parameters) brings issues such as high inference costs, high hardware requirements for deployment, and large latency, which are not suitable for mobile, edge, and small-to-medium enterprise scenarios. Therefore, the key research questions are: Can small VLMs (1B-8B parameters) handle complex visual reasoning? What is the gap between them and large models?

A comprehensive evaluation system was built, assessing from three dimensions:

1. VCR (Visual Commonsense Reasoning): Causal inference combining world knowledge (e.g., holding an umbrella → raining);
2. MMMU (Multimodal Multitask Understanding): Covers multiple disciplines, testing the ability to combine visual information with professional knowledge;
3. MathVista: Mathematical visual reasoning, such as geometric figure analysis, function image analysis, etc.

The participating models cover 12 models from 1.8B to 34B parameters:

• Small models (1B-8B): Moondream2 (1.8B), Qwen2-VL-2B/7B, InternVL2-2B/8B, Phi-3-Vision (4.2B), LLaVA-NeXT-7B;
• Large models (13B+): LLaVA-1.5-13B, InternVL2-26B, LLaVA-1.6-34B;
• Closed-source APIs: GPT-4o, Claude. Hardware memory requirements (examples):

  Model	FP16 Memory	8-bit	4-bit
  Moondream2 (1.8B)	~4GB	~2GB	-
  Qwen2-VL-7B	~15GB	~9GB	~5GB
  Small models can run on consumer-grade GPUs (e.g., RTX3060 running 7B 8-bit), while large models require professional hardware.

Key dimensions inferred from the framework:

1. Task complexity and scale: Small models are sufficient for single-step tasks; multi-step reasoning is an area where the gap between small and large models is significant;
2. Quantization impact: 8/4-bit quantization is beneficial for edge deployment, but error accumulation in multi-step reasoning may lead to result deviations;
3. Domain specialization: Small models fine-tuned for specific domains may outperform unoptimized large general models; selection should be based on needs rather than pursuing large and comprehensive models.

Model selection for different scenarios:

• Extremely resource-constrained (mobile/IoT): Moondream2 or Qwen2-VL-2B (8-bit quantization, 2-3GB memory), suitable for simple visual question answering/image description;
• Balanced performance and cost (small-to-medium enterprises/SaaS): 7B-8B models (Qwen2-VL-7B, InternVL2-8B), run smoothly on mid-range GPUs, meeting most commercial scenarios;
• High precision requirements (scientific research/medical): Complex multi-step reasoning requires 13B+ models or closed-source APIs; it is recommended to first build a baseline with small models before deciding to upgrade.

The study provides a complete reproducible toolchain: automated data download, smoke test verification, subset testing, YAML configuration management, and saving results as CSV/JSON. Future directions: Model compression technologies (distillation, pruning, quantization) to expand small model capabilities; multimodal architecture innovation to improve efficiency; model selection should integrate task requirements, resource constraints, and cost-effectiveness; small models are indispensable in AI democratization.