SAHM: A New Benchmark for Arabic Financial and Shari'ah Compliance Reasoning

The research team launched the SAHM benchmark, covering 14,380 expert-validated data entries. Evaluations show that Arabic fluency does not equate to evidence-based financial reasoning ability, providing a crucial tool for Arabic financial NLP research. This benchmark focuses on Islamic financial compliance reasoning, filling the gap in Arabic financial AI evaluation.

Current financial AI progress is concentrated in English scenarios; English financial NLP already has a well-established benchmark system, but Arabic financial NLP lacks high-quality evaluation benchmarks. The Arab world has a large financial market, and Islamic finance follows Shari'ah rules (such as prohibiting interest, investing in forbidden industries, requiring risk-sharing, etc.). AI needs cross-domain reasoning, which goes far beyond simple translation or retrieval.

SAHM is a document-anchored benchmark and instruction-tuning dataset. Data sources include AAOIFI regulatory documents, real fatwa legal rulings, professional exam materials, and corporate documents, totaling 14,380 expert-validated instances. Seven tasks are designed: AAOIFI standard Q&A, Fatwa Q&A and multiple-choice questions, accounting and business exams, financial sentiment analysis, extractive summarization, event-causal reasoning, etc., to comprehensively evaluate model capabilities.

Evaluations of 19 top LLMs found: Arabic fluency cannot be converted into evidence-based financial reasoning ability; models perform well in recognition tasks such as sentiment analysis and multiple-choice questions, but their performance in generative tasks (e.g., open-ended answers) and event-causal reasoning tasks drops significantly, with causal reasoning being the biggest shortcoming.

Evaluating financial AI cannot only focus on language fluency; it needs specialized domain benchmarks to test substantive capabilities. Progress in English financial NLP cannot be automatically transferred to Arabic; Islamic finance requires specialized data and training. Financial decisions need traceable evidence, and SAHM emphasizes document-anchored interpretability requirements.

The research team open-sourced the SAHM benchmark data, evaluation framework, and instruction-tuned models. Application scenarios include: Islamic financial compliance AI assistants, Arabic financial education intelligent tutoring systems, compliance review tools, market intelligence analysis (sentiment signal and event-causal extraction), etc.

SAHM has limitations such as geographic coverage (mainly Gulf regions), timeliness (needing regular updates of regulations and Shari'ah rules), multimodal expansion (needing to support charts and tables), and adversarial testing (evaluating robustness). In the future, it is necessary to expand geographic representation, update data, add multimodal capability evaluation, and adversarial testing.