Generative AI Empowers Meta-Research in Psychology: Exploration of Automation and Precision

The Slovak research project APVV-24-0278 uses large language models to overcome the high error rates and time constraints of traditional manual meta-research, developing a prompt engineering process for precise data extraction to enhance research rigor. The project is maintained by viktoriagajdosova, hosted on GitHub with the original title 'Enhancing-Meta-Research-in-Psychology-by-Generative-AI', published on May 26, 2026. Original link: https://github.com/viktoriagajdosova/Enhancing-Meta-Research-in-Psychology-by-Generative-AI. Keywords: large language models, meta-research, psychology, systematic review, prompt engineering, data extraction, open science, evidence synthesis.

Meta-research is the core of the scientific self-correction mechanism. It synthesizes independent research results through systematic reviews and meta-analyses to guide clinical practice, policy-making, and research directions. Traditional manual meta-research faces three major challenges: high time cost (6 months to 2 years), large human resource demand (cross-validation by multiple researchers), and high error rates (10-30%, including numerical transcription errors, unit confusion, etc.). Due to complex experimental designs and exponential growth of literature (PubMed Psychology adds tens of thousands of new papers each year), manual processing capacity in the psychology field is approaching its limit.

The project adopts a systematic approach to develop the LLM data extraction process:

1. Corpus construction: Collect literature from multiple psychology subfields and establish a gold-standard dataset with manual double coding;
2. Prompt engineering iteration: Design prompt templates using few-shot learning and chain-of-thought techniques, requiring LLMs to output structured data and reasoning processes;
3. Validation and calibration: Compare LLM results with the gold standard, calculate metrics such as accuracy and recall, and improve prompts in a targeted manner;
4. Human-AI collaboration exploration: LLMs mark low-confidence cases for manual review, or serve as a second coder for cross-validation.

The project commits to open sharing of all tools: prompt template library (covering multiple data types), validation dataset, evaluation scripts, and best practice guidelines. It plans to develop user-friendly software tools (literature management interface, data extraction workflow, etc.) to lower technical barriers. Open science practices can promote progress in the field and avoid reinventing the wheel.

If the project succeeds, it will have four major impacts:

1. Democratization of systematic reviews: Small laboratories and researchers in developing countries can conduct high-quality evidence synthesis;
2. Real-time evidence update: Support 'living systematic reviews' to continuously monitor new literature;
3. Research quality improvement: Reduce human errors and assist in detecting non-standard reporting in original studies;
4. Open new research questions: Analyze large-scale issues such as literature reporting quality trends and geographic bias.

Ethical considerations:

1. Transparency: Clearly disclose the level of LLM participation;
2. Validation responsibility: LLM outputs need manual review;
3. Data privacy: Focus on privacy protection in scenarios like gray literature. Technical limitations: Current LLMs are insufficient in handling tables, mathematical formulas, and non-English literature. The project will track technological developments to integrate new capabilities.

This project is a cutting-edge application of generative AI in scientific methodology. It promotes the responsible implementation of technology through rigorous validation, transparent sharing, and prudent application. It provides reusable technical infrastructure for the psychology research community and is worth attention and participation.