Introducing Intelligence Analysis Techniques into LLM Systems: The sats4llms Project Explores Structured Analysis Methods to Counter AI Cognitive Biases

The open-source sats4llms project adapts Structured Analysis Techniques (SATs) developed by the U.S. intelligence community into architectural patterns and prompt protocols for LLM agent systems. It addresses systemic bias issues of LLMs (such as hallucinations, sycophancy, confirmation bias, anchoring effect, etc.) through adversarial reasoning methods.

Structured Analysis Techniques (SATs) are disciplined reasoning methods developed by the U.S. intelligence community to address cognitive biases of human analysts (such as confirmation bias, anchoring effect, groupthink). The project's core insight: The systemic biases of LLMs are structurally similar to human cognitive biases, so SATs can be adapted as tools to counter LLM biases. Moreover, since LLMs are software systems, interventions can be deterministic protocols rather than advisory guidelines.

The project includes 13 structured analysis techniques (such as ACH Competitive Hypothesis Analysis, Critical Assumption Testing, Devil's Advocate, Red Team Analysis, etc.), each targeting specific cognitive biases or reasoning flaws. It establishes a correspondence between 12 cognitive biases and LLM behaviors (like anchoring effect, confirmation bias, groupthink, etc.), and also focuses on LLM-specific failure modes (sycophancy, hallucinations), with dedicated prompt countermeasures for each failure mode.

The project uses Obsidian as the content creation environment, Quartz v4 to generate static websites, and GitHub Actions for automated deployment. Content organization follows strict knowledge management principles: original materials are fully preserved and immutable; explanations and analyses are only on synthesis pages; extensive wiki links are used; each page has clear type markers and tags; synthesis pages note sources and confidence levels. Tool pages are provided: Bias-SAT Cross-Reference Matrix, SAT Selection Guide, SAT Pipeline.

The project proposes testable hypotheses (e.g., whether ACH reduces premature conclusions, whether Critical Assumption Testing lowers hallucination rates, whether Red Team Analysis improves robustness against adversarial prompts). It invites the community to contribute empirical research; evidence supporting or refuting hypotheses is welcome, distinguishing it from AI safety projects that only stay at the conceptual level.

Traditional AI safety research focuses on model-level improvements (larger datasets, complex architectures, fine-grained alignment). sats4llms demonstrates the importance of system and process-level interventions. SATs do not make LLMs smarter, but make the reasoning process more adversarial towards its own conclusions (embedded into the process). It provides a practical toolbox for LLM application developers, suitable for high-reliability decision support systems or dialogue agents that resist prompt injection.

AI in high-risk fields (medical diagnosis, financial analysis, etc.) requires a reasoning process that can withstand scrutiny. sats4llms adapts human experience in dealing with cognitive limitations to AI, which is a pragmatic path to building reliable AI. Project address: https://github.com/mattdot/sats4llms, Online documentation: https://mattdot.github.io/sats4llms/