EasyHeal: An Intelligent Handwritten Prescription Parsing System Based on Vision-Language Models

EasyHeal is an AI medical application that uses Vision-Language Models (VLM) to parse handwritten medical prescriptions. Its core objectives are to address issues such as poor readability of handwritten prescriptions, low recognition rates of traditional OCR, high risk of medication errors, and complexity in patients' medication management. The project extracts medication information and automatically generates medication reminders using multimodal AI technology. It uses local AI inference (hosted by LM Studio) to ensure privacy, and integrates a human-machine collaboration mechanism to enhance reliability. The project is open-source, and the source code is available on GitHub (link: https://github.com/FaheemaTamton/EasyHeal).

Handwritten prescriptions are still the mainstream form in many regions around the world, but they have many pain points: 1. Doctors' handwriting is hard to read, leading to patients easily misunderstanding medication information; 2. Traditional OCR technology has limited accuracy in recognizing handwritten text (especially medical terms and abbreviations); 3. Misunderstanding prescriptions may lead to serious medication errors, even endangering lives; 4. Chronic patients are prone to mistakes when manually managing their medication schedules. The EasyHeal project was born to convert handwritten prescriptions into structured digital health data and automatically generate medication reminders through multimodal AI technology.

Why Choose VLM Over Traditional OCR?

Traditional OCR struggles to handle challenges such as the diversity of medical terms, differences in handwriting styles, non-standardized prescription formats, and lack of context understanding. VLM can better address these issues by understanding both image content and text semantics simultaneously.

VLM Workflow

User uploads prescription image → Image preprocessing → VLM analysis → Extract medication information → Structured JSON → Database storage → Generate medication reminders → Patient tracking

Advantages of Local AI Inference

Using LM Studio to host models locally achieves: privacy protection (data does not go to the cloud), low latency, offline availability, and cost-effectiveness (no API fees required).

Patient Management Module

Supports profile creation, medical detail recording (medical history/allergies), photo uploads, and guardian information storage.

Prescription Understanding Engine

Extracts the following information: medication name (e.g., Paracetamol, Amoxicillin), dosage (e.g., 500mg, 1 tablet), frequency of administration (e.g., twice daily BID), dietary instructions (e.g., take after meals), and course duration (e.g., 7 days).

Intelligent Medication Reminders

Generates schedules based on frequency (fixed time once daily, morning and evening for twice daily, etc.), provides a snooze function and medication confirmation.

Structured Storage

All information is stored in an SQLite database, supporting prescription history tracing, medication-patient association, and reminder record tracking.

Technology Stack

• Frontend: HTML/CSS/JS responsive design
• Backend: Django framework + SQLite3 database
• AI layer: Vision-Language Model (VLM) + LM Studio local inference

Project Structure

Includes backend (Django configuration), core (business logic), vlm (VLM module), static/media resources, etc.

Image Preprocessing

Steps: Enhance contrast and brightness → Noise removal → Skew correction → Area cropping, to improve VLM recognition accuracy.

Design Reasons

Medical data is sensitive and requires double verification; to handle complex scenarios such as special medical symbols, non-standard abbreviations, and blurry handwriting.

Collaboration Flow

AI extraction → Manual review → Confirmation/correction → Final storage; manual correction feedback is used for continuous model learning.

Patient Side

Simplifies medication management (automatic reminders, history tracking), assists in understanding prescriptions (conversion of professional terms).

Medical Institutions

Digitalization of small clinics, assistance in pharmacy dispensing (reducing errors).

Social Value

Improves medical accessibility (in resource-poor areas), prevents medication errors (WHO states that medication errors are a major cause of medical harm).

Current Limitations

• Relies on LM Studio, which has certain hardware requirements;
• Mainly optimized for English prescriptions;
• Needs to comply with local medical regulations (currently positioned as an educational/portfolio project).

Future Directions

• Expand medical document types (test reports, medical records);
• Integrate drug interaction checks and medication adherence analysis;
• Support more VLM models, cloud deployment, and mobile development;
• Multilingual support and adaptation to different medical systems.