Open OCR LLM Eval: Industrial-Grade Model Selection Evaluation Practice for Small Language Models and OCR

This project systematically evaluates small large language models (SLLM) and OCR models on real business data, constructing a complete model selection framework. It provides reusable methodology for enterprises to choose optimal AI model stacks under resource-constrained scenarios, balancing accuracy, cost, latency, privacy, and maintainability.

Enterprise AI Adoption Dilemma: Enterprises face choices among diverse models (closed-source/open-source SLLM, traditional/deep learning OCR) and need to balance accuracy, cost, latency, privacy, and maintainability.

Initiation Background: Launched by UncommonLab for its document processing service, focusing on real business data, SLLM, end-to-end pipeline evaluation, and 8-week MVP iteration.

Core Objectives: Model research, consistency testing on real data, cost analysis, MVP validation, and production deployment recommendations.

Evaluation Dimensions: Accuracy (CRA, WER, semantic understanding, end-to-end task completion), efficiency (latency, throughput, memory, GPU usage), cost (per-inference, infrastructure, operation and maintenance manpower, licensing), reliability (availability, error recovery, compatibility, compliance).

Test Dataset: Real business data covering printed docs (5k), handwritten forms (2k), low-quality scans (1.5k), complex tables (1k), multilingual (800), with strict annotation quality.

Process: Candidate screening → benchmark testing → deep evaluation (error analysis, stress test, boundary test) → weighted scoring (accuracy 40%, cost30%, latency20%, reliability10%).

OCR Models:

• Chinese scenario: PaddleOCR outperforms Tesseract by 8%.
• Handwriting: TrOCR better but cost 3-5x higher.
• Tables: DONUT retains structure vs traditional OCR's plain text.
• Low-quality input: Commercial APIs more robust than open-source.

SLLM Models: 3B-level SLLMs (Phi-3 Mini:86.7% info extraction accuracy) reach ~90% of 8B models (Llama3-8B:89.2%) in resource-constrained scenarios.

Pipeline Integration: Recommended config: PaddleOCR (Chinese)/TrOCR (multilingual) + Phi3 Mini (English)/Qwen1.8B (Chinese) + serial processing with LLM error correction.

TCO Model:

• Self-built: GPU servers (45%), storage/network (15%), operation and maintenance manpower (25%), power/room(10%), licensing(5%).
• Cloud: Pay-per-use, no upfront cost, good elasticity.

Break-even: Low volume (<10k docs/month) → cloud; medium (1-100k) → self-built SLLM; high (>100k) → self-built full stack.

Hidden Costs: Technical debt (self-built needs model updates), vendor lock-in (cloud), opportunity cost (self-built uses engineering resources).

Architecture: Microservices: Document upload → preprocessing → OCR → LLM → output.

Tech Stack: Backend (FastAPI), OCR (PaddleOCR + custom post-processing), LLM (vLLM for batch inference), deployment (Docker + K8s), monitoring (Prometheus + Grafana).

Performance: Avg end-to-end latency:2.3s/doc; concurrency:50 docs/min; accuracy:91.5% task completion; single A10 GPU supports production load.

Decision Tree:

1. Constraints: Data out-of-domain allowed? Latency requirements? Budget?
2. OCR: Chinese→PaddleOCR; multilingual→EasyOCR/TrOCR; complex layout→DONUT/Nougat; extreme accuracy→commercial API.
3. SLLM: English→Phi3 Mini; Chinese→Qwen; edge→TinyLlama/Gemma2B; complex推理→Llama3-8B.
4. Deployment: Quick validation→cloud API; medium→self-built; large→hybrid (self-built + cloud backup).

Risk Mitigation: Maintain 2-3 candidates; model performance monitoring; open-source priority; modular architecture.

Key Insights:

• SLLM (3B) sufficient for most document tasks; OCR quality is pipeline bottleneck; domain fine-tuning beats general large models; cost can be reduced by 50-70% via optimization.

Limitations: Limited to document processing; East Asian language focus; no long-term stability data.

Future Work: Follow SLLM updates; explore VLM; adaptive model selection; expand to more document types; integrate RAG.