Contradish: An Open-Source Tool for Detecting and Fixing Inconsistencies in LLM Responses

Contradish is an open-source Python tool developed by Michele Joseph, focusing on detecting and fixing answer inconsistencies in large language models (LLMs). It quantifies model output changes via the "CAI Strain" metric (measuring consistency when questions are rephrased) and provides a full pipeline from detection to repair (including prompt engineering, fine-tuning data generation, and real-time firewall). It's applicable to high-risk fields like customer service, healthcare, and law, helping build reliable AI applications.

As LLMs are used in high-risk areas (customer service, medical, legal), a critical issue emerges: the same question phrased differently may lead to opposite answers (e.g., AI customer service refusing a direct refund query but agreeing when phrased as a favor). This inconsistency is called "CAI failure" (Consistency, Alignment, Integrity failure) by Contradish, often due to safety policy loopholes (e.g., refusing harmful requests in English but complying in role-play scenarios).

Contradish is an open-source Python tool for detecting, quantifying, and repairing LLM inconsistencies. Its core innovation is the CAI Strain metric: it measures how much model answers change when questions are semantically equivalent but rephrased (using over 16 paraphrasing techniques). The CAI Strain range is 0.00-1.00: <0.20 (stable), 0.20-0.40 (edge state), >0.40 (unstable).

1. Quick Detection: Install via pip (supports Anthropic/OpenAI/Litellm), run demo with contradish (30s for 12 test cases) or full benchmark with contradish benchmark --model .... 2. Auto Repair: Use contradish improve command to rewrite system prompts, generate fine-tuning data, or set up firewall, reducing CAI Strain (e.g., from 0.42 to 0.13). 3. Production Firewall: Real-time monitoring with memory-aware tracking (stores atomic commitments from past conversations to check consistency in subsequent queries).

Contradish uses three test case types: adversarial (model should stick to stance), real-world tension (model should present both views), representational (model should refactor chaotic premises). It also provides multi-dimensional metrics: SW-Strain (severity-weighted), MT-Strain (multi-turn), CL-Strain (cross-lingual), CAT-Strain (compound attack), SPA-Delta (system prompt anchoring). Additionally, it generates smart findings (e.g., identifying root causes of failures like specific terms).

Fairness: Detects disparate treatment (different answers based on protected attributes like age/nationality) via contradish fairness command. Benchmark: Public benchmark with 2160 strain tests across 20 high-risk areas, using cross-provider judgment to avoid bias. Leaderboard top models: claude-opus-4-6 (0.118), claude-sonnet-4-6 (0.141), gpt-4o (0.179). Comparison: Contradish outperforms traditional tools with multi-dimensional detection, CAI Strain system, auto repair, real-time firewall, memory awareness, fairness testing, and smart insights.

Scenarios: AI customer service (policy consistency), medical consultation (symptom advice), legal Q&A (stable interpretation), education (content consistency), content moderation (uniform handling). Quick Start: Use Python API to define LLM function, create test suite, run tests; or use pre-built policy packages (e.g., Suite.from_policy("ecommerce")).

Consistency is as important as accuracy for LLM applications. Contradish fills the gap with scientific measurement (CAI Strain) and a complete toolchain from detection to repair. It's essential for teams building production-grade AI apps. Project links: GitHub (https://github.com/michelejoseph/contradish), official website (https://contradish.com), technical paper (PAPER.md in repo).