Systematic Review of AI in Survey Research: The survAI Project Reveals the Current Applications and Limitations of Large Language Models

Title: Systematic Review of AI in Survey Research: The survAI Project Reveals the Current Applications and Limitations of Large Language Models Abstract: survAI is a systematic review study on the applications of large language models (LLMs) in survey research and public opinion research. It analyzes 136 empirical literatures, covering application scenarios in the pre-, during-, and post-data collection stages, and provides researchers with a panoramic reference in this field. Keywords: Large Language Models, Survey Research, Systematic Review, Public Opinion Research, Social Sciences, Data Collection, GPT, Zero-shot Prompting, Human-AI Collaboration, Reproducibility

Introduction: The survAI project was completed by a German-American team. Through a systematic review, it sorts out the current applications, experiences, and limitations of LLMs throughout the entire lifecycle of survey research, and provides open-source data and code, offering a panoramic reference for researchers in this field.

Research Background and Significance

With the rapid development of Large Language Models (LLMs) technology, their application potential in the field of social science research has become increasingly prominent. As core methodologies in social sciences, survey research and public opinion research are facing profound changes driven by AI technology. However, despite the widespread attention to the disruptive impact of LLMs, existing literature reviews often lack systematicness or focus only on specific application scenarios, making it difficult to present a complete picture of this field.

The survAI project emerged as the times require. Completed jointly by research teams from Germany and the United States, it aims to comprehensively sort out the current applications, successful experiences, and potential limitations of LLMs throughout the entire lifecycle of survey research through systematic review methods. The project not only provides detailed research data and code but also establishes a searchable database, offering valuable reference resources for subsequent researchers.

Research Design and Methods

This study adopts a rigorous systematic review method, conducting both quantitative and qualitative evaluations of 136 empirical studies. Starting from the pre-, during-, and post-data collection stages, the research team deeply analyzed the specific application methods of LLMs in different research links.

The core dimensions covered in the study include:

• Application Scenario Classification: Text data classification, survey data generation, survey tool development, etc.
• Technical Implementation Methods: Model selection, prompt strategy, interaction mode
• Language and Cultural Background: Studies are mainly focused on English contexts; multilingual applications remain to be explored
• Effect Evaluation Indicators: Accuracy, generalization ability, reproducibility, etc.

Core Findings and Insights

The study found that as of 2025, the applications of LLMs in survey research are mainly concentrated in three major areas: text data classification, survey data generation, and survey tool development. This distribution pattern reflects the matching degree between current technical capabilities and actual research needs.

Model Usage Patterns

Despite the diverse model choices and interaction methods, most research designs still tend to use a single model from the GPT series and adopt the zero-shot prompting strategy. This convergence has raised researchers' concerns about the generalization ability and reproducibility of individual studies. When the vast majority of studies are based on similar model architectures and prompt strategies, the robustness of research results may be affected.

Performance Characteristics Analysis

The study revealed a key pattern: LLMs perform better when approximating broad, representative aggregation patterns, but are relatively inferior when dealing with subtle individual attitudes, specific topics, or complex constructs. This finding has important guiding significance for research design—researchers need to carefully evaluate the applicability of LLMs according to the characteristics of the research problem.

Positioning of Human-AI Collaboration

The study clearly points out that LLMs should be regarded as tools to assist human researchers, not as alternatives. This positioning emphasizes the irreplaceable role of human expertise in research design, data verification, and result interpretation. The value of technology lies in enhancing human capabilities, not replacing human judgment.

Data Resources and Reproducibility

The survAI project provides rich open-source resources to ensure the reproducibility and scalability of the research:

• survAI.Rmd: R code for reproducing the statistical charts in the paper
• data_survai.csv: Data of quantitative variables encoded in the paper
• corpus_1_included.bib: 136 literatures included in the systematic review
• corpus_2_theoretical.bib: 17 non-empirical theoretical literatures
• corpus_3_surveymethods4llms.bib: 61 survey methodology literatures that can provide references for LLM research
• corpus_4_tools.bib: 38 LLM methodology literatures that can be transferred to the field of survey research

These resources not only provide researchers with a complete data foundation but also lay a methodological foundation for subsequent research in this field.

Future Outlook and Recommendations

Based on the research findings, the author team put forward a series of recommendations to promote the development of this field:

1. Prudent Model Selection: Researchers should choose appropriate models according to specific research needs and avoid blindly following mainstream choices
2. Standardized Reporting Norms: Establish unified reporting standards to improve the comparability between studies
3. Domain-Specific Benchmarks: Develop dedicated evaluation benchmarks for survey research scenarios
4. Cross-Language Research: Expand application research in non-English contexts to enhance the universality of results

Although technology is developing rapidly and may create tension with traditional qualitative evaluation methods, survey research methodologists have unique advantages in adopting LLMs—these models will continue to rely on human-curated input data and validation data.

Conclusion

The survAI project, through systematic review methods, provides an important starting point for understanding the potential, concerns, and gaps of LLMs in survey research. For researchers engaged in survey research, public opinion analysis, or computational social sciences, this project is not only a detailed literature map but also a knowledge base that inspires future innovations. With the continuous evolution of technology, such basic research work will provide important support for the healthy development of this field.