Multimodal Document AI System: Structured Information Extraction via Fusion of CNN+BiLSTM+OCR

This project proposes a multimodal document AI system based on the FUNSD dataset, which integrates Convolutional Neural Networks (CNN), Bidirectional Long Short-Term Memory (BiLSTM), and OCR technology to realize document-level named entity recognition with an accuracy rate of 93%. Through deep fusion of multimodal features, the system solves the problem that traditional document processing ignores the association between visual and semantic information, providing an effective solution for structured information extraction.

Document information extraction is an important research direction in the field of artificial intelligence. Traditional document processing methods often handle visual information and text information separately, ignoring the inherent connection between document layout and semantics. Real-world documents (such as invoices, contracts, and forms) contain rich structured information, which is reflected not only in text content but also in spatial layout. How to understand both the visual features and text semantics of documents has become the key to improving the accuracy of information extraction. Single-modal methods are difficult to capture complete semantics, and multimodal fusion technology provides a new idea for this problem.

The project adopts a three-layer architecture design, organically combining computer vision, natural language processing, and layout understanding:

• CNN Layer: Extracts visual features of document images and learns spatial patterns (layout information such as text area positions and table structures);
• BiLSTM Layer: Processes sequential text features, captures forward and backward context dependencies, and helps understand semantic relationships and long-distance dependencies;
• OCR Layer: Converts image text into processable text sequences, serving as a key bridge connecting visual and text modalities.

The project uses the FUNSD (Form Understanding in Noisy Scanned Documents) dataset for training and evaluation. This dataset contains various real-scenario scanned documents and is annotated with fine-grained entity information (questions, answers, titles, etc.). The task goal is token-level named entity recognition, which labels the semantic role of each token, accurately locates and classifies structured information in documents, and lays the foundation for automated form processing.

The core innovation of the system lies in the deep fusion of multimodal features:

1. Align CNN visual features with OCR text sequences, where each token is associated with image position information to form a spatial-semantic joint representation;
2. When processing text, BiLSTM takes visual features as auxiliary input, considering both word semantics and spatial positions;
3. Through end-to-end joint training, the features of each modality complement each other effectively.

Experiments on the FUNSD dataset show that this multimodal method achieves an accuracy rate of approximately 93%, verifying the effectiveness of the multimodal approach. Compared with single-modal baseline methods, the fusion of visual and text information significantly improves the ability to recognize complex document structures, especially in scenarios involving tables, multi-column layouts, and nested structure documents. It also has good robustness to noisy scanned documents.

This technology has wide application value: in the financial field, it can automatically process invoices, statements, and contracts; in the medical field, it can assist in extracting key information from medical records and inspection reports; in the government affairs field, it supports automated entry of forms and application materials. Multimodal document AI represents the development direction of intelligent document processing. In the future, combining with large language models and multimodal pre-training technologies, the system's capabilities will be further enhanced, and it is expected to realize full automation of document processing.

This project builds an effective multimodal document understanding system through the combination of classic deep learning technologies. The CNN+BiLSTM+OCR architecture is concise yet powerful. For developers in the document AI field, it is a good learning case that clearly demonstrates the basic ideas of multimodal fusion and provides a solid foundation for subsequent advanced technologies such as Transformer architecture and vision-language pre-training models.