Generative AI Empowers Meta-Research in Psychology: Enhancing Scientific Rigor and Efficiency with Large Language Models

This project explores the use of large language models (LLMs) to address high error rates and time constraints in traditional manual meta-research. Its core objectives include developing prompt workflows for precise data extraction, enhancing research rigor, and open-sourcing tools to empower the scientific community. Focused on the field of psychology, the project promotes meta-research method innovation and open science practices through AI assistance.

Meta-research (meta-analysis/systematic review) is a key link in knowledge integration in psychology, but the traditional process has two major pain points:

1. High error rate: Manual data extraction is prone to omissions and biases, affecting result reliability;
2. Time constraints: Manual processing of hundreds of documents takes months to years, limiting the update frequency and scale of research. Project APVV-24-0278 explores the application potential of LLMs to address these issues.

The project sets three core objectives:

1. Develop precise prompt workflows: Achieve structured data extraction, multi-dimensional coding, and quality control;
2. AI integration to enhance rigor: Improve consistency, expand coverage, and enhance reproducibility;
3. Open-source sharing to empower the community: Open tools, templates, and datasets to accelerate domain progress and establish industry standards.

Prompt Pipeline

• Domain knowledge embedding: Integrate specialized knowledge of psychological methodology;
• Multi-stage processing: Extract research type, method details, and result data in stages;
• Few-shot learning: Provide high-quality examples to guide the model in understanding norms;
• Chain-of-thought: Guide the model to display reasoning processes to improve accuracy.

Validation and Calibration

• Manual audit sampling: Randomly verify extracted results to calculate error rates;
• Consistency check: Compare result consistency across multiple prompts/models;
• Edge case identification: Test and improve prompts to handle complex scenarios;
• Confidence scoring: Prioritize review of low-confidence outputs.

1. Large-scale literature reviews: Shorten the cycle to months and increase update frequency;
2. Real-time evidence synthesis: Provide timely decision support for rapidly evolving fields (e.g., digital mental health);
3. Methodological audit: Identify domain methodological trends and issues (e.g., insufficient sample size);
4. Interdisciplinary integration: Promote knowledge fusion between psychology and fields like neuroscience and education.

1. Data privacy and copyright: Comply with literature database usage agreements and pay attention to the boundaries of factual information extraction;
2. Model bias and fairness: Need to identify and correct theoretical/methodological/cultural biases in training data;
3. Necessity of human-machine collaboration: Judgments such as research quality assessment still require human expertise;
4. Interpretability challenges: Need to improve model decision transparency to ensure result credibility.

1. Dynamic meta-analysis: A continuously updated living system that automatically incorporates new studies;
2. Personalized evidence synthesis: Generate customized reports based on user needs;
3. Cross-language meta-research: Multilingual models reduce English-centric bias;
4. Methodological innovation: Free up researchers' resources to focus on high-level design and interpretation.

Project APVV-24-0278 provides a feasible path for the modernization of meta-research in psychology. AI assistance is a necessary means to cope with data explosion and ensure research rigor. Its open-source commitment promotes community collaboration, which is worthy of attention and participation from meta-research methodology and psychology researchers.