KSAA-2026 Champion Solution: Regularized Fine-Tuning for Arabic Speech Automatic Diacritization

This article introduces the first-place system of the second task in the KSAA-2026 shared task. The system achieves automatic diacritization of Arabic speech by fine-tuning the CATT-Whisper multimodal model with regularization, under the constraints of only 2327 training samples and no external data allowed, achieving a word error rate (WER) of 23.26% and ranking first in the task.

Arabic is morphologically rich, and diacritics are crucial for understanding word meaning and grammar, but they are often omitted in daily writing, posing challenges for speech recognition. KSAA-2026 Task 2 requires generating fully diacritized text from speech audio and un-diacritized text transcripts. The difficulties include: only 2327 training samples, strict prohibition of external data, and complex Arabic diacritization rules (such as word-end changes, grammatical case changes, etc.).

The champion team Thaka adopted the CATT-Whisper architecture:

• CATT Text Encoder: Character-level processing to capture Arabic word formation rules and morphological changes;
• Whisper Speech Encoder: OpenAI pre-trained model, used in a frozen state to avoid overfitting;
• Multimodal Fusion: Fusing audio and text features via cross-modal attention mechanism for collaborative processing.

To address overfitting in small datasets, the team used the following regularization techniques:

• R-Drop Consistency Regularization: Two forward passes of the same input (with dropout enabled), minimizing KL divergence to enhance robustness;
• Optuna Hyperparameter Optimization: Automatically searching for optimal hyperparameters, finding that higher weight decay can effectively prevent overfitting;
• Focal Loss: Replacing cross-entropy loss, reducing the weight of easily classified samples and focusing on hard-to-classify diacritization positions.

Two integration methods are used in the inference stage to improve performance:

• Monte Carlo Dropout Integration: Keeping dropout enabled, performing 200 random forward passes on the same input, and averaging the softmax probabilities;
• Multi-Checkpoint Integration: Training multiple checkpoints and averaging the prediction results to reduce bias.

In the official evaluation of KSAA-2026, the system achieved a word error rate of 23.26% (including word-end changes and handling of non-diacritized positions), ranking first among all participating teams. The results prove that in small-data scenarios, regularization and inference integration can achieve performance comparable to that of large-data training.

The success of the Thaka team provides experience for low-resource language processing:

1. Freezing pre-trained encoders and fine-tuning task-specific components is an effective strategy for small data;
2. Regularization is more important than model capacity in small datasets;
3. Inference integration can improve prediction stability;
4. Multimodal fusion requires carefully designed mechanisms rather than simple concatenation. These experiences are applicable to other fields such as low-resource languages, medical image analysis, and few-shot learning.