Text-Aware Visual Question Answering System: Innovative Practice of OCR and Multimodal Fusion

This article introduces the Text-Aware VQA project, which builds a text-aware visual question answering system integrating OCR and BLIP models, achieving efficient and accurate image-text understanding through question-guided filtering and multimodal fusion. Core innovations include deep integration of OCR and visual models, question-guided attention mechanism, and lightweight design supporting edge deployment. The system outperforms the baseline BLIP in accuracy (+9.4%), inference speed (+15%), and model size (-36%), and has wide applications in document intelligence, scene interaction, and educational assistance.

Visual Question Answering (VQA) is an AI task that outputs correct answers given images and questions. Traditional VQA focuses on objects, scenes, and relationships, but performs poorly when dealing with questions related to text in images. The Text-Aware VQA project aims to address this pain point and focuses on handling QA tasks for images containing text.

Dual-Branch Feature Extraction

• Visual Branch: BLIP Encoder uses Vision Transformer to encode images into visual tokens, extracts multi-scale features and leverages pre-trained knowledge.
• Text Branch: OCR pipeline completes text detection, recognition, and position encoding, preserving spatial information.

Question-Guided Filtering

1. Encode the question into a query vector; 2. Calculate relevance with OCR text blocks; 3. Dynamically filter relevant text; 4. Weight by confidence.

Multimodal Fusion and Answer Generation

• Early fusion (cross/collaborative attention, gating mechanism) + joint representation learning;
• Supports both classification-style (fixed options) and generation-style (open questions) answers.

Advantages of Question Guidance

Reduce computational complexity, improve accuracy, enhance interpretability (visualize attention areas).

Robust Handling of OCR Errors

Downweight/discard low-confidence results, correct errors via semantic completion, multi-candidate fusion decision-making.

Edge Deployment Optimization

INT8 quantization (reduce memory), inference acceleration (optimize attention), batch processing (efficient concurrency).

Document Intelligence

Form processing, contract review, invoice information extraction.

Scene Interaction

Road sign navigation, product query, menu assistant.

Educational Assistance

Textbook QA, exam paper grading, multilingual learning.

Evaluation Datasets

TextVQA, ST-VQA, OCR-VQA.

Metric Comparison

Metric	Baseline BLIP	Our System	Improvement
Accuracy	52.3%	61.7%	+9.4%
Inference Speed	100ms	85ms	+15%
Model Size	385M	245M	-36%

Ablation Experiments

• Removing question guidance: accuracy drops by 7.2%;
• Removing OCR branch: accuracy of text-related questions plummets;
• Simplifying fusion: accuracy drops by 4.1%.

Current Limitations

Handwriting recognition needs improvement, limited handling of complex layouts, insufficient long text understanding.

Future Improvements

Support for multi-page documents, video text QA, multilingual expansion, end-to-end OCR training.

Open Source Resources

Provide PyTorch code, pre-trained weights, demo scripts, and deployment guides. Quick start: Clone the repository → install dependencies → download weights → run inference.

Conclusion

The project demonstrates the potential of combining OCR and multimodality, with a lightweight design suitable for resource-constrained scenarios, providing innovative ideas for text-aware VQA.