NUSKHA: A Multimodal Medical OCR System for Indian Prescriptions

NUSKHA is a multimodal medical OCR system specifically designed to recognize handwritten annotations on Indian prescriptions. By combining image enhancement and vision-language model technologies, it extracts medications, dosages, diagnoses, and SOAP records into structured JSON data. It aims to solve the challenge of handwritten prescription recognition in Indian healthcare scenarios and promote the digital transformation of healthcare.

The global digital transformation of medical records is key to improving service quality and efficiency. However, developing countries like India still rely on handwritten prescriptions, which face issues such as difficulty in preservation and information silos (pharmacies cannot quickly access information, researchers struggle with data analysis, and patients have fragmented referral information). Handwritten prescription recognition is far more challenging than ordinary OCR: large differences in doctors' writing styles, high requirements for professional medical terminology, and mixed abbreviations/dosage annotations/diagnostic codes increase the complexity of understanding.

NUSKHA (Neural Understanding and Structured Knowledge Harvesting from Handwritten Annotations) is an end-to-end multimodal medical OCR system. Its goal is to accurately recognize handwritten annotations on Indian prescriptions and extract key medical data to output standardized JSON. The core innovation lies in using a vision-language model as the understanding engine, fusing computer vision and natural language processing end-to-end, avoiding the error accumulation problem of traditional OCR pipeline design, and improving overall accuracy.

NUSKHA has a built-in image enhancement module to optimize medical document processing:

1. Noise suppression and contrast enhancement: Enhance text clarity while preserving stroke details through adaptive histogram equalization and edge-preserving filtering;
2. Geometric correction and perspective transformation: Automatically detect and correct perspective distortion caused by shooting angles;
3. Handwritten area localization: Use object detection to distinguish between handwritten and printed areas;
4. Binarization optimization: Generate high-quality binary images using an adaptive threshold algorithm to provide optimal input for the vision-language model.

NUSKHA outputs structured JSON data containing fields such as patient_info, medications, diagnosis, and soap_notes. Its application value includes:

• Pharmacy automation: Reduce manual entry errors and speed up the dispensing process;
• Electronic Health Record (EHR) integration: Facilitate connection to EHR systems and support long-term health management;
• Medical research and public health: Provide a data foundation for drug use analysis, disease monitoring, and policy formulation;
• Telemedicine support: Instantly parse prescription photos and improve the efficiency of remote consultations.

Challenges and solutions in developing NUSKHA:

1. Data scarcity: High-quality annotated data is difficult to obtain due to privacy issues; synthetic data generation, transfer learning, and active learning may be used to mitigate this;
2. Multilingual mixing: Indian prescriptions often mix English, Hindi, and dialects; leverage the multilingual pre-training advantages of vision-language models to address this;
3. Domain adaptability: Improve the accuracy of professional vocabulary recognition through fine-tuning with medical domain data;
4. Format diversity: Robust design ensures stable operation across different prescription layouts.

Future development directions of NUSKHA: Expand to healthcare scenarios in more developing countries, support more document types such as test reports and medical records, integrate speech recognition to enable automatic recording of oral prescriptions, and add clinical decision support functions like drug interaction checks. Industry impact: Demonstrate how AI can solve practical problems in resource-constrained environments, allowing advanced technology to benefit a wider range of healthcare workers and patient groups.