RLAIF-SPA: A Breakthrough in Emotional Speech Synthesis via Reinforcement Learning from AI Feedback

RLAIF-SPA is a novel framework integrating automatic speech recognition (Whisper) and large language models (Qwen2-Audio, GPT-4o). It addresses the trade-off between emotional expression and intelligibility in emotional speech synthesis via Reinforcement Learning from AI Feedback (RLAIF), without requiring expensive manual annotations. Key innovations include a four-dimensional fine-grained prosody label system and the GRPO optimization algorithm. Experiments show significant improvements in intelligibility (WER reduced by 26.1%) and speaker similarity (SIM-O increased by 9.1%), providing a successful example of RLAIF application in specific domains.

The field of emotional speech synthesis has long faced a trade-off between emotional expressiveness and speech intelligibility: enhancing emotion often leads to ambiguous pronunciation, while pursuing clarity results in flat and mechanical speech. Additionally, traditional training relies on large amounts of manually annotated data, which is costly and difficult to scale. The RLAIF-SPA project proposes an innovative solution to these pain points.

The core breakthrough of RLAIF-SPA is the introduction of Reinforcement Learning from AI Feedback (RLAIF), which differs from RLHF (Reinforcement Learning from Human Feedback) in that it uses AI models entirely to generate reward signals: Whisper evaluates semantic accuracy (intelligibility), and Qwen2-Audio assesses prosody-emotion label alignment (emotional expression). Meanwhile, the project constructs a four-dimensional fine-grained prosody label system (structure, emotion, speech rate, intonation), automatically generated by GPT-4o, reducing data preparation costs.

RLAIF-SPA is based on the MiniCPM-O 2.6 multimodal model and uses LoRA for efficient fine-tuning. Training employs the GRPO (Group Relative Policy Optimization) algorithm with key hyperparameters: learning rate of 5e-6, batch size of 1, group size of 4, and KL penalty weight of 0.01. The reward function is 0.3×(1-WER) + 0.7×label alignment score, prioritizing emotional expression. The code supports multi-GPU configuration, with different model components allocated to different devices.

RLAIF-SPA has achieved significant improvements in key metrics: compared to the Chat-TTS baseline, the Word Error Rate (WER) is reduced by 26.1% (improved intelligibility), and speaker similarity (SIM-O) is increased by 9.1% (better speech consistency). The results prove that it can achieve or even surpass the performance of traditional methods without manual annotations.

The project includes a complete training pipeline (prosody labeling → audio generation and reward calculation → GRPO optimization) and provides a concise inference script (inference.py). The code is modular (inference.py, label.py, main_grpo.py, etc.), supporting custom label categories (modify qwen_audio_service.py) and reward weight adjustments (modify main_grpo.py) to adapt to different application scenario requirements.

The project has the following areas for optimization: high computational resource requirements (multi-model collaboration and GRPO training require a large number of GPUs), inference latency for real-time applications needs optimization, generalization ability for unseen emotional types or speaker styles needs verification, and evaluation dimensions need to add other metrics such as naturalness.

RLAIF-SPA is not only a breakthrough in emotional speech synthesis but also a successful example of the application of Reinforcement Learning from AI Feedback in specific domains. It proves that high-quality reinforcement learning training without manual annotations can be achieved through an AI evaluation system, providing a complete technical stack for speech synthesis researchers and demonstrating the feasibility of RLAIF to the AI community. With the development of multimodal large models, RLAIF is expected to play a role in more fields, and RLAIF-SPA is an important milestone in this trend.