SignWriting Evaluation Toolkit: An Automated Evaluation Scheme for Sign Language Machine Translation

This article introduces an automated evaluation toolkit for the SignWriting sign language writing system, aiming to address the lack of standardized evaluation metrics for sign language transcription and translation models. The toolkit provides standard metrics such as BLEU, chrF, and CLIPScore, along with custom symbol distance measures, offering a standardized automated evaluation solution for sign language technology research.

Sign language is an independent language system with visual-spatial characteristics, making traditional text evaluation metrics difficult to accurately measure the quality of its transcription and translation. SignWriting is an internationally used sign language writing system invented by Valerie Sutton, which uses combinations of symbols to represent elements such as handshape, location, movement, and expression. Evaluating SignWriting faces three major challenges: complex symbol combinations make simple string matching unable to reflect semantic similarity; multiple writing variants exist for the same concept; the relative positions of symbols have semantic importance that needs to be captured.

The toolkit includes multiple evaluation methods:

1. Tokenized BLEU: BLEU with tokenization processing for SignWriting FSW strings, measuring symbol-level n-gram matching, suitable for transcription tasks.
2. chrF: Character-level n-gram F-score, capturing fine-grained similarity, effective for handling writing variants.
3. CLIPScore: Using the CLIP model to calculate semantic similarity of SignWriting images, focusing on visual similarity.
4. Symbol Distance Score: A custom metric that considers symbol category, position, and relative relationships to measure the degree of difference between expressions.

The toolkit is implemented in Python, providing clear API interfaces with independent modules for easy integration. To verify its effectiveness, the authors conducted a nearest neighbor search experiment on the SignBank corpus (approximately 230,000 single-hand sign language symbols): common sign language variants were selected as queries, and the Top10 results of each metric were compared. Experiments show that a single metric cannot perfectly cover all cases, and combining multiple metrics can more comprehensively evaluate model quality.

The toolkit applies to multiple scenarios:

• Sign language transcription tasks: Evaluate the accuracy of ASLR system outputs against manual annotations.
• Sign language translation tasks: Measure the quality of mutual translation between sign language and spoken language.
• Model development: Standardized metrics improve result comparability and guide model optimization.
• Resource construction: Assist in filtering and cleaning SignWriting corpora.

Current limitations: CLIPScore may overfocus on appearance rather than linguistic meaning; Tokenized BLEU and chrF need improvement in handling writing variants; parameters of the Symbol Distance Score need adjustment for different sign languages. Future directions: Support continuous sign language evaluation; develop complex metrics considering grammatical structure; establish standardized benchmark datasets; welcome community contributions via GitHub.

The SignWriting Evaluation Toolkit fills an important gap in sign language technology research, providing a standardized automated evaluation solution. Combining complementary metrics allows for a comprehensive understanding of model performance, which will play an important role in promoting standardization and reproducibility in the field.