Pi Vision Tool：为非多模态模型赋予视觉能力的智能代理扩展

Pi Vision Tool is an innovative Pi Agent extension that enables pure text large language models (LLMs) to gain visual understanding through tool calls. Key features include flexible image compression, reasoning depth control, and support for multiple image formats, providing developers with a dynamic balance between cost and quality. The project is maintained by xezpeleta and hosted on GitHub (https://github.com/xezpeleta/pi-vision-tool), updated on 2026-06-09.

In the current LLM ecosystem, multimodal capabilities are a key differentiator, but many excellent pure text models (e.g., DeepSeek V4 Pro, certain GPT-5 Codex versions) lack image understanding. Pi Vision Tool addresses this by adding the describe_image tool to Pi Agent, allowing non-multimodal models to delegate image analysis to vision-capable models—preserving text model strengths while expanding their ability to handle visual tasks.

Design Principles:

1. Model-led control: The calling model decides compression, reasoning depth, and custom prompts for each task.
2. Dynamic cost-quality balance: Parameters like compress (reduce size/cost), reasoning (6 levels from off to xhigh), and prompt (custom questions) enable flexible trade-offs. Workflow:
3. Decision & Call: Text model autonomously invokes describe_image with image path, questions, compression, and reasoning params.
4. Visual Model Processing: Tool sends image/prompt to a configured vision model (e.g., Qwen VL).
5. Result Integration: Visual model's text response is returned to the original model, which integrates it into its reasoning chain.

Image Processing Pipeline: When sharp is installed, it optimizes images: reduce max dimension to 1568px, convert RGBA to RGB, and PNG to JPEG (quality 85) — cutting load size by 4x. Disable with compress: false for pixel-precise tasks. Model Registry: Uses Pi's ctx.modelRegistry to dynamically find visual models (config via JSON, supports multiple providers). Persistent Config: The /vision config command saves settings to `/.pi/agent/vision-tool.json` (no restart needed, more user-friendly than env vars).

Installation:

• npm (recommended): pi install npm:pi-vision-tool
• Git: pi install git:github.com/xezpeleta/pi-vision-tool
• Local path: pi install /path/to/pi-vision-tool
• Quick test: pi -e /path/to/pi-vision-tool Configuration:

1. Add visual model to ~/.pi/agent/models.json (set input: ["text", "image"]).
2. Set API key in ~/.pi/agent/auth.json.
3. Configure default model via /vision config or env vars PI_VISION_PROVIDER/PI_VISION_MODEL.

Use Cases:

• Dev debugging: Analyze terminal error screenshots for diagnosis.
• UI analysis: List interactive elements/status or describe page layout.
• Document/chart understanding: Deeply analyze system architecture diagrams (use reasoning: high).
• Image comparison: Find differences between two screenshots.
• Color/style extraction: Get hex colors or extract text from designs. Advantages:
• Decoupling: Text and visual models can be independently selected/upgraded.
• Cost control: Fine-tune cost via compression and reasoning depth.
• Progressive adoption: Easy to add to existing Pi Agent workflows without重构.

Limitations:

• Higher latency (two model calls: text + visual).
• Context window consumption (visual model responses take up space).
• Dependence on external visual models (requires additional setup/quota).
• Error propagation (visual model hallucinations may be amplified). Comparison:

  Dimension	Pi Vision Tool	Native Multimodal Model
  Cost	Controllable (on-demand)	Fixed (usually higher)
  Flexibility	High (swap visual backends)	Low (bound to specific model)
  Latency	Higher (two calls)	Lower (single call)
  Context Efficiency	Needs tool output management	Native fusion (more compact)
  Use Case	Existing text model workflows	New multimodal apps

Summary: Pi Vision Tool is a practical extension that solves the vision gap for text LLMs, offering flexible control and cost balance. It demonstrates the power of tool-based architecture in AI systems. Community: The project is open-source on GitHub, part of the Pi Agent ecosystem (available in Pi's package gallery). Contributions are welcome via GitHub Issues for feedback or feature requests.