TrustScoreAI: Quantifying and Evaluating the Bias Level of Large Language Models Using the Unified Bias Index

TrustScoreAI is an innovative tool designed to objectively quantify large language model (LLM) bias using the Unified Bias Index (UBI). It addresses the critical gap in fair AI by providing a standardized, multi-dimensional framework to measure and compare bias levels across models. Key features include:

• UBI combines three bias dimensions (magnitude, disparity, distribution shift) into a 0-1 score.
• Supports mainstream LLMs (OpenAI, Google, Anthropic, etc.) and multiple bias dimensions (race, gender, occupation, etc.).
• Offers CLI and Web interfaces for flexible use cases.

As LLMs are increasingly used in high-stakes applications (recruitment, medical diagnosis, legal advice), their inherent biases pose significant risks of unfair treatment to certain groups. However, existing methods lack a unified, quantifiable way to assess and compare bias levels. This gap led to the development of TrustScoreAI, which aims to turn subjective bias perceptions into measurable engineering metrics.

UBI is the core of TrustScoreAI, combining three quantifiable bias dimensions:

1. Bias Magnitude (BM): Measures overall bias strength via language analysis (emotion, assumptions).
2. Disparity (DP): Calculates group selection rate differences: 1 - min(SR_k)/max(SR_k).
3. Distribution Shift (DS): Uses KL divergence to compare model output with a fair baseline.

UBI formula: UBI = α·BM + β·DP + γ·DS (weights configurable). A baseline calibration mechanism (G̃(x,i) = G(x,i) - G(baseline,i)) ensures results are reliable by eliminating model style effects.

TrustScoreAI's pipeline includes four layers:

• Data Layer: Raw prompts (for race, gender, etc.), baseline data, result storage.
• Core Compute: Modules like data_loader.py (preprocessing), llm_connector.py (API integration), pipeline.py (coordination).
• Metrics: bm.py (BM calculation), sr.py (DP), ds.py (DS), aggregator.py (UBI synthesis).
• UI: CLI (batch analysis) and Web interface (interactive visualization, real-time tracking).

TrustScoreAI serves diverse stakeholders:

• Model Developers: Integrate into CI/CD to monitor bias during model updates.
• AI Researchers: Conduct large-scale comparative studies on LLM bias.
• Enterprises: Audit models for compliance (critical for finance/medical sectors).
• Decision Makers: Use UBI scores to select ethically sound models.

Highlights:

• Math rigor (statistical foundation, interpretable metrics).
• Modular design (customizable components).
• Rich visualization and flexible export (JSON, CSV, Excel).

Limitations:

• Dependent on quality of test prompts.
• English-centric (needs validation for other languages).
• Cannot track dynamic bias drift over time.
• Limited support for emerging bias types (e.g., multi-modal).

TrustScoreAI's UBI methodology transforms subjective bias into measurable indices, enabling objective assessment of LLM fairness. While it is a powerful tool, achieving true AI fairness requires collaboration across tech, policy, and society. TrustScoreAI provides a critical starting point for building responsible AI systems and making informed choices about model deployment.