Application of Large Language Models in Suicide Risk Identification: Structured Prompting and Evaluation with Real Conversation Data

This article explores the application of Large Language Models (LLMs) in suicide risk identification, focusing on analyzing the potential of structured prompt engineering methods in the mental health field and verifying model performance based on real conversation datasets. The study examines performance differences among various LLMs, the improvement of judgment accuracy by structured prompts, the robustness of real data processing, and related ethical issues, aiming to provide references for AI-assisted mental health assessment.

Mental health issues are a major global public health challenge. According to WHO statistics, nearly 800,000 people die by suicide each year, and timely identification of high-risk individuals is key to prevention. Traditional assessments rely on professional interviews and scales, but due to limited human resources and subjectivity, many potential individuals are not noticed. In recent years, the natural language understanding capabilities of LLMs have sparked exploration, and their application involves complex issues such as technical feasibility, ethical boundaries, privacy protection, and clinical effectiveness.

This project constructs a systematic evaluation framework to test the suicide risk classification performance of LLMs. Core research questions include performance differences among different models, the improvement of accuracy by structured prompts, the robustness of real conversation data, and risk-related language features. Structured prompts provide a decision-making framework through role definition (activating professional knowledge), task description (risk level standards), input specifications, reasoning requirements (chain of thought), and output format (JSON); and design comparative experiments with variants such as baseline prompts, role enhancement, few-shot learning, chain of thought, and comprehensive optimization.

Obtaining real data requires strict privacy protection, with sources including de-identified forum data, collaborative data from medical institutions, or synthetic data; annotation must be completed by professional experts, using multi-expert independent annotation plus arbitration to ensure reliability; class imbalance issues need to be addressed through metrics such as F1 score, AUC-ROC, and sampling strategies. Evaluation metrics include sensitivity (miss rate), specificity (false positive rate), PPV, F2 score (focusing on recall), and calibration curves; cross-model comparisons cover basic performance, prompt sensitivity, consistency, and interpretability.

Model outputs should not replace professional diagnosis and require manual review; the model's biases toward different genders, ages, and cultural groups need to be checked; experimental settings, prompt design, and evaluation processes must be transparent and auditable to facilitate reproduction and verification.

Future directions can include exploring multimodal fusion (text + voice + physiological signals), longitudinal monitoring (long-term language pattern tracking), human-machine collaboration interface design, domain-specific models (fine-tuning general LLMs with professional data), etc.

LLMs have shown potential to address social issues in the field of suicide risk identification, and structured prompts improve judgment accuracy, but clinical applications require careful verification and strict human supervision. We look forward to multidisciplinary cooperation to promote more reliable, fair, and interpretable AI-assisted solutions to help prevent suicide.