Clinical Timeline Reconstruction: Multimodal Alignment Fusing Text Semantics and Structured Temporal Information

This paper proposes a retrieval-augmented multimodal alignment framework that fuses the semantic richness of clinical narrative texts with the precise timestamps of electronic health record (EHR) tabular data to achieve more accurate clinical timeline reconstruction. Experiments on the MIMIC dataset show that this method significantly improves absolute timestamp accuracy, providing strong support for clinical decision-making and research.

Clinical data exists in two complementary but hard-to-integrate forms: unstructured narrative texts (e.g., progress notes, discharge summaries) are semantically rich but temporally ambiguous, often using relative/vague time expressions; structured EHR tabular data (e.g., lab results, medication records) have precise timestamps but incomplete information—over one-third of clinical events exist only in texts. The discrepancy between the two is the core challenge for timeline reconstruction.

Core idea: Texts answer "what happened", while tables answer "when it happened". The workflow has three stages: 1. Extract central anchor events (events with clear timestamps, key clinical nodes, or those that can be linked to structured data); 2. Relative positioning of non-central events (parse relative time, infer event order); 3. Structured data calibration (retrieval-augmented matching of entities/values/time ranges). A dual-encoder architecture, cross-modal attention alignment, and temporal consistency constraints are used.

In the i2m4 benchmark test on the MIMIC dataset: absolute timestamp error was reduced by 30-40%, precise matching rate within 1 hour increased by 25%, and coarse-grained matching rate within 24 hours increased by 15%; temporal consistency improved and sequence conflicts decreased; 34.8% of text events had no table records, and 20% of table events had no text mentions; generalization ability was consistent across different models.

Clinical applications: Supports early sepsis identification, treatment response assessment, and complication prediction; facilitates real-world evidence generation, clinical pathway optimization, and medical quality monitoring. The technical architecture is general and can be extended to fields such as legal document analysis, financial event tracking, and project management.

Current limitations: Data quality (ambiguity/errors/synchronization issues), cross-institutional generalization ability to be verified, insufficient real-time processing, and lack of interpretability. Future directions: Develop robust alignment algorithms, verify cross-institutional generalization, implement real-time updates, and enhance interpretability.