Clinical Text Summarization: A Benchmark Study Comparing Traditional NLP and LLMs

This study systematically compares the performance of traditional NLP pipelines and large language models (LLMs) on medical intent summarization and clinical information extraction tasks using the NIH MeQSum dataset, providing empirical references for technology selection in medical AI applications. The study was published on GitHub by AlessandroClericuzio on June 9, 2026. Project link: https://github.com/AlessandroClericuzio/clinical-summarization-nlp-vs-llm.

Medical text processing has become a challenging scenario for NLP due to the abundance of professional terminology and high accuracy requirements (errors may lead to misdiagnosis). Traditional methods rely on carefully designed NLP pipelines (NER, syntactic analysis, etc.), which are highly interpretable but require extensive expert participation in feature engineering; LLMs demonstrate strong text capabilities, yet there is a question of whether they can replace traditional methods.

Dataset: Uses the NIH MeQSum dataset (paired real patient questions + professional summaries); Comparative Methods:

• Traditional NLP: Extractive parsing, NER for medical entity extraction, structured information reorganization;
• LLMs: Generative prompt-based end-to-end summarization, using in-context learning (few/zero-shot strategies); Evaluation Dimensions: Accuracy (semantic consistency), completeness (key information retention), conciseness (compression ratio), readability (fluency), safety (no misinformation).

Pros and Cons of Traditional NLP: Advantages: Interpretable (clear steps), controllable (parameter/rule adjustment), resource-efficient (no GPU required), domain-adaptable (medical dictionaries/rules); Limitations: High development cost (expert participation), weak generalization (poor adaptability to new texts), heavy maintenance (continuous rule adjustments for knowledge updates). Pros and Cons of LLMs: Advantages: Universal (no domain training needed), high development efficiency (fast adaptation via prompt engineering), strong expression (fluent and natural), knowledge-rich (pre-training includes extensive medical knowledge); Limitations: Hallucination risk (misinformation), black-box nature (hard to interpret), high computational cost (GPU required), consistency challenges (same input may yield different outputs).

• Task complexity determines selection: Traditional NLP is more accurate for structured information extraction (e.g., entity extraction); LLMs may be better for open-ended summary generation;
• Hybrid architecture may be optimal: LLM for initial understanding + traditional NLP for post-processing verification;
• Special requirements for medical scenarios: Accuracy and interpretability are higher than general tasks; the black-box nature of LLMs may hinder adoption in regulatory environments.

• Gradual adoption: Start with low-risk scenarios (e.g., patient education materials);
• Human-machine collaboration: LLMs assist doctors, who then review and edit;
• Safety guardrails: Multiple verifications (knowledge base checks, rule checks, manual reviews);
• Interpretability first: Choose traditional methods or develop LLM interpretability technologies for regulatory scenarios;
• Continuous evaluation: Monitor model performance degradation and edge cases in production environments.

Limitations: Single dataset (MeQSum may not cover all clinical texts), static evaluation (does not consider post-deployment degradation), gap between automatic metrics and human judgment; Future Directions: Multi-dataset/multi-language cross-domain validation, human-machine collaboration effectiveness evaluation, hybrid architecture optimization, LLM fine-tuning strategies for medical scenarios.