AI Content Describer: An NVDA Plugin That Lets Visually Impaired Users 'See' the World

AI Content Describer is an open-source NVDA screen reader plugin that uses multimodal large language models to provide visually impaired users with detailed descriptions of images, interface controls, camera feeds, and more. It supports over ten AI models and local deployment options, helping visually impaired users overcome visual information blind spots and enhance their independence and equality in accessing information.

Traditional screen readers only support OCR text recognition and cannot understand the overall context of images, object relationships, or scene meanings. The rapid development of multimodal large language models (such as GPT-4V, Gemini, Claude, etc.) has enabled a breakthrough from "recognizing text" to "understanding content", bringing new possibilities to the field of assistive technology.

The plugin supports describing various objects such as interface controls, screenshots, clipboard images, and real-time camera feeds. It has a face detection feature to help visually impaired users confirm their own position in the frame during video conferences. Application scenarios include interpreting screenshots for remote work, understanding charts for learning, getting to know software interface layouts, and checking camera angles before online meetings, reducing reliance on others' assistance.

Cloud support includes over ten mainstream multimodal models (such as OpenAI GPT-4 series, Google Gemini, Anthropic Claude, etc.), with Pollinations providing a free GPT-4 access layer. Local deployment supports Ollama (llama3.2-vision), llama.cpp, Seer local service, and LiteLLM Proxy. Optimized for Chinese users, it integrates the vivo BlueLM Vision model, which can be used with a free NVDA-CN account.

Supports multiple image formats including PNG, JPEG, WEBP, and non-animated GIFs. An intelligent caching mechanism saves API quota and costs while improving response speed. A conversational follow-up function allows in-depth information retrieval. Supports Markdown rendering of structured content to enhance readability.

Multiple sets of shortcut keys are designed: NVDA+Shift+I to open the description menu, NVDA+Shift+U to quickly describe navigation objects, NVDA+Shift+Y to describe clipboard images, NVDA+Shift+J for face position detection, and NVDA+Alt+C to open the follow-up dialogue window. All shortcut keys can be customized to adapt to different users' operating habits.

As an open-source project, the global community actively participates, and it already supports multiple languages including Russian, Serbian, French, and Chinese, allowing more non-English users to use it without barriers, reflecting the inclusive value of open-source software.

Current limitations: The integration stability of Ollama and llama.cpp needs improvement, the response quality and speed of the free Pollinations layer fluctuate, and local operation has high hardware requirements. In the future, as model efficiency improves and open-source visual models mature, these issues are expected to be gradually resolved.