Multimodal Accessibility Generative Model: AI-Driven Inclusive Content Creation

This project is maintained by nadir-sheikh09 on GitHub (link: https://github.com/nadir-sheikh09/generative-models-multimodal-accessibility). Its core is to generate three types of accessible multimodal content via fine-tuning diffusion models and large language models: rich text alternative descriptions, simplified/high-contrast visual content, and audio description scripts, with support for CoreML export to run on Apple devices. The project aims to address digital content access barriers for over 1 billion people with disabilities worldwide, promote equal rights, and is a typical exploration of AI for good.

There are over 1 billion people with disabilities globally (about 285 million with visual impairments, 466 million with hearing impairments). Digital content accessibility is an equal rights issue, but most current content remains a barrier for users with disabilities. Traditional solutions rely on manual annotation, which is costly and hard to scale. The development of multimodal large models has made AI-generated accessible content possible, and this project is an exploration in this direction.

The project focuses on three types of accessible content generation:

1. Rich Text Alternative Descriptions: Generate detailed scene, action, emotion, and other information for images, supporting screen readers;
2. Simplified/High-Contrast Visual Content: For users with cognitive impairments or low vision, provide simplified, high-contrast, and iconified conversions;
3. Audio Description Scripts: Generate scene and action narratives during dialogue gaps in videos to help visually impaired users understand the story.

Multimodal Model Fine-tuning

• Diffusion Models: Based on Stable Diffusion and others, handle image conversion via LoRA fine-tuning;
• Large Language Models: Based on Llama/Mistral and others, generate description scripts via instruction fine-tuning.

Fairness-Aware Training

Mitigate issues like representational bias through diverse sample supplementation, adversarial training, RLHF, and bias detection.

Quality Assessment

Design metrics for description quality (accuracy/completeness/conciseness/comprehensibility), user experience (screen reader compatibility, etc.), and fairness (consistency across groups, etc.).

CoreML Export

Support converting models to CoreML format for on-device inference on Apple devices (privacy protection, low latency, offline availability).

1. Web Accessibility Enhancement: Batch generate image alt-text to improve WCAG compliance;
2. Educational Material Adaptation: Convert textbook illustrations to simplified versions and generate audio descriptions;
3. Media Content Accessibility: Generate audio scripts for videos and descriptions for image news;
4. Assistive Technology Development: Build applications such as real-time photo description and video audio description.

1. Subjectivity of Descriptions: Provide style adjustment, user feedback, and crowdsourced evaluation;
2. Complex Scene Understanding: Introduce scene graphs, multi-round generation, and preprocessing techniques;
3. Cultural Sensitivity: Multicultural samples, cultural consultant reviews, and localization adaptation;
4. Real-time Requirements: Model distillation and quantization, streaming generation, and on-device deployment.

Positive Impact

Promote inclusion, improve efficiency, empower creation, and educational equity.

Potential Risks and Mitigation

• Description errors: Confidence mechanism + manual review;
• Privacy leakage: On-device inference + data protocols;
• Over-reliance: Clear positioning as AI-assisted;
• Digital divide: Open source and free + digital literacy education.

This project balances technical depth and social value, creating equal information access opportunities for people with disabilities. Future directions include: multilingual support, real-time video description, personalized adaptation, interactive accessibility, and cross-modal fusion. It provides reference implementations for developers, demonstrates technical paths for enterprises, and reminds that technology should serve the groups most in need.