AI4AIR: How Large Language Models Reshape the Entire AI Research Workflow

The AI4AIR survey project released by the FinD Lab of the Institute of Computing Technology, Chinese Academy of Sciences (ICT CAS) systematically sorts out the five core roles of Large Language Models (LLMs) in the entire AI research workflow, covering data engineering, model design and optimization, evaluation, and closed-loop automation. It also constructs a two-dimensional classification framework to provide systematic guidance for AI research automation. The original resources of the project can be obtained on GitHub (link: https://github.com/ICT-FinD-Lab/Awesome-LLMs-for-AI-Research).

For a long time, machine learning research has relied on manual exploration processes with trial and error (data processing, model design, tuning, evaluation, etc.), requiring significant human input. With the evolution of LLM capabilities, it has become possible for AI systems to participate in accelerating the entire lifecycle of AI research. The AI4AIR project is a systematic response to this trend, providing not only a literature review but also a two-dimensional classification framework to understand the multiple roles of LLMs in AI research.

AI4AIR innovatively proposes a two-dimensional classification system: the first dimension covers AI subfields such as natural language processing and computer vision; the second dimension follows the research workflow (data engineering, model design and optimization, evaluation, cross-stage closed-loop automation). Under this framework, the five core roles of LLMs include:

1. Annotator: Automatically generate data labels and annotations to reduce manual costs;
2. Synthesizer: Integrate knowledge to generate hypotheses, literature reviews, and experimental design suggestions;
3. Optimizer: Guide neural network architecture search and hyperparameter tuning;
4. Evaluator: Automatically evaluate model output quality and detect biases;
5. Orchestrator: Coordinate research steps and dynamically adjust experimental directions to achieve closed-loop automation.

LLM-assisted AI research faces three core bottlenecks:

1. Data Contamination: LLM training data contains public benchmark datasets, which may lead to test data leakage and distorted experimental results;
2. Hallucination: Generate seemingly reasonable but incorrect experimental designs, literature citations, or theoretical deductions, affecting scientific research accuracy;
3. Feedback Loop Reliability: Early biases in closed-loop scenarios may be amplified, leading to deviations in research directions.

The AI4AIR project simultaneously releases a GitHub repository, bilingual README documents, and an online homepage, providing directly usable literature indexes and classification systems to help the community form consensus and promote the establishment of tool standards. For researchers, this framework can help quickly locate the applicable roles of LLMs in specific research problems and effectively integrate LLM capabilities into workflows.

The vision depicted by AI4AIR is deep human-machine collaboration: humans focus on core issues, high-level strategies, and creative thinking, while LLMs take on repetitive and exploratory work (literature retrieval, experiment execution, result analysis, etc.). In the future, reliability issues need to be addressed, such as contamination detection mechanisms, hallucination suppression techniques, and robust feedback control systems, to balance efficiency and accuracy.

AI4AIR is the academic community's systematic response to the trend of LLMs empowering scientific research. It not only sorts out existing work but also provides an extensible classification framework, pointing the way for future research and tool development. For practitioners concerned with AI research automation and hoping to improve efficiency, this is a review resource worth reading in depth.