Evaluating and Fixing Web Accessibility with Large Language Models: A Cross-Model Validation Study

This study explores the use of Large Language Models (LLMs) to automatically detect and fix web accessibility issues. It conducts cross-model validation of mainstream models based on the W3C ACT Rules Test Suite and provides an open-source benchmark framework. The core uses a two-stage evaluation method: the first stage tests the model's accessibility result prediction and rule retrieval capabilities; the second stage tests the repair code generation capability and validates the quality through cross-model re-evaluation.

Web accessibility ensures that people with disabilities can use online resources equally. According to WHO data, over 1 billion people with disabilities globally cannot fully participate in digital life due to accessibility barriers. Traditional detection and repair rely on professional knowledge and manual review, which are high-cost and limited in efficiency. With the improvement of LLM capabilities, the research team conducted a cross-model validation study to explore the feasibility of LLMs to solve this problem automatically and the capability differences between models.

The project provides an open-source benchmark framework based on the W3C ACT Rules Test Suite. The technical architecture has two pipelines: 1) cross_model_pipeline.py implements accessibility result prediction, blind issue diagnosis, and rule retrieval, including prompt templates, system instructions, and JSON output validation; 2) cross_model_repair_pipeline.py implements repair generation and cross-evaluation, using a 3x3 matrix (code from each repair model is re-evaluated by other models) to eliminate single-model bias.

The tested models include Google Gemini3 Flash Preview, OpenAI GPT5 Mini, and Anthropic Claude Haiku4.5. The dataset comes from the official W3C ACT Test Suite, containing complete HTML test cases and WCAG mapping relationships, covering common issues such as missing image alt text, form label errors, and keyboard navigation barriers.

The project has high transparency: each output record saves the complete system instructions, prompts, and model responses; raw outputs are stored in JSONL format, and error records are saved separately; cross-evaluation data from the repair stage is fully retained, facilitating result reproduction, model behavior auditing, and subsequent improvements.

Practical significance: Provides practitioners with an objective benchmark of LLM capabilities for accessibility tasks; open-source code and datasets support extended research. Limitations: Only tests static HTML fragments (no dynamic interactions/JS), and repair evaluation relies on the model's own judgment. Future directions: Support dynamic content and single-page applications, introduce human expert evaluation, develop targeted fine-tuned models, and explore multimodal models to handle visual accessibility issues.

This cross-model validation study provides empirical data for the application of LLMs in the field of web accessibility and establishes a reproducible evaluation framework. Insights: LLMs are powerful, but critical compliance tasks require careful validation; open-source resources lay the foundation for subsequent research. With the improvement of model capabilities and evaluation methods, AI-driven accessibility tools are expected to become standard equipment for developers.