Traditional Statistical Representation Outperforms Generative AI in Identifying Expert Reviewers

This article evaluates six professional domain identification methods and finds that TF-IDF identifies experts in the top 25 recommendations at a rate of 79.5%, while GPT-4o mini only achieves 51.5%. The study shows that fine-grained vocabulary is better at distinguishing subfield expertise than semantic smoothness, and traditional statistical methods perform better in this task.

Peer review is the core of academic publishing, but expert identification faces many challenges: the subdivision of modern scientific fields (e.g., differences between subfields of machine learning and reinforcement learning), increased complexity from interdisciplinary research, outdated classifications for emerging fields, and impacts from regional and language factors. There is an urgent need for automated identification systems.

The study treats expert identification as an information retrieval problem, using data from an international observatory review system as a benchmark (author identity serves as an expert proxy). Six methods are evaluated: TF-IDF (traditional statistics), GPT-4o mini (generative AI), and four others (based on citations, keyword matching, etc.). The core metric is whether the marked expert is included in the top 25 recommendations.

The results show that TF-IDF has a success rate of 79.5%, while GPT-4o mini only has 51.5%. The reasons include: 1. Fine-grained vocabulary vs. semantic smoothness: TF-IDF relies on precise word matching to capture subfield differences, while generative AI's semantic smoothness tends to overgeneralize and blur boundaries; 2. Interpretability: TF-IDF is transparent and auditable, while generative AI is a black box; 3. Computational efficiency: TF-IDF has fast preprocessing, low cost, and is scalable.

1. Not all tasks are suitable for generative AI; tools should be selected based on task requirements; 2. Transparency and interpretability are crucial in professional fields; 3. Future research can explore hybrid methods combining TF-IDF and generative AI to balance precision and semantic understanding.

Limitations: The evaluation is mainly in the field of astronomy, and performance in other fields needs to be verified; the lack of literature in emerging fields may affect TF-IDF's effectiveness; cross-language scenarios are not covered. Future directions can expand to more fields, emerging fields, and cross-language identification.