Automatic Academic Paper Title Generation: Fine-tuned PEGASUS-large Model Outperforms GPT-3.5-turbo

This study focuses on the task of automatic academic paper title generation. By fine-tuning the open-source pre-trained model PEGASUS-large, experiments show that it outperforms GPT-3.5-turbo (zero-shot generation) and the fine-tuned LLaMA-3-8B model in multiple evaluation metrics such as ROUGE and METEOR. The research verifies the feasibility and reliability of AI-generated titles, providing a new solution for researchers to efficiently create high-quality paper titles.

The title of an academic paper is the "face" of the paper, undertaking key functions such as conveying core ideas, attracting readers' attention, facilitating retrieval and discovery, and establishing an academic impression. However, writing an appropriate title poses many challenges: balancing multiple goals such as accuracy and attractiveness, professionalism and readability within a limited number of words, which is especially difficult for non-native English speakers. Therefore, automatic title generation technology has emerged, aiming to assist researchers in efficiently creating high-quality titles.

Dataset Construction: Three datasets were used:

1. CSPubSum dataset: Abstract-title paired data in the field of computer science;
2. LREC-COLING-2024 dataset: Title-abstract pairs from computational linguistics conference papers;
3. SpringerSSAT dataset (newly constructed): Social science journal data to enhance domain diversity and cross-domain generalization capabilities.

Model Comparison:

• Open-source models: PEGASUS-large (dedicated to abstract tasks), LLaMA-3-8B (Meta open-source model), both fine-tuned;
• Closed-source model: GPT-3.5-turbo, using zero-shot setting (generation without examples).

Five automatic evaluation metrics were used to measure title quality from multiple dimensions:

1. ROUGE: Based on n-gram overlap, including ROUGE-1/2/L;
2. METEOR: Introduces stemming and synonym matching to capture semantic similarity;
3. MoverScore: Calculates text movement distance based on word embeddings;
4. BERTScore: Uses the BERT model to compute semantic similarity;
5. SciBERTScore: Uses SciBERT (pre-trained on scientific literature) to evaluate academic semantic quality.

Key Findings: The fine-tuned PEGASUS-large outperforms the fine-tuned LLaMA-3-8B and GPT-3.5-turbo (zero-shot) in most metrics, reflecting the value of task-specific architecture, the importance of fine-tuning, and the competitiveness of open-source models. ChatGPT Advantages: It performs well in creative title generation; although it does not excel in automatic metrics, its titles are more attractive. It is recommended to use them in combination: PEGASUS generates basic titles to ensure accuracy, and ChatGPT polishes them to enhance creativity.

Quality Analysis: AI-generated titles are generally appropriate and reliable, characterized by accurate content, standardized language, and complete structure; however, there are issues such as insufficient innovation, limitations in domain specificity, and neglect of cultural sensitivity. Application Value:

• For researchers: Improve efficiency, gain inspiration, multi-language support, version comparison;
• For publishing institutions: Assist in review, standardization, SEO optimization;
• For academic platforms: Automatic indexing, recommendation systems, data cleaning.

Model Selection Recommendations:

• For accuracy: Choose fine-tuned dedicated models like PEGASUS;
• For creativity: Use large models like ChatGPT;
• Balanced solution: Dedicated model generates basic title + large model polishes it. Future Directions: Multimodal title generation, interactive generation, style transfer, cross-language generation, citation prediction, etc.