LLM Strategic Decision-Making Capability Benchmark: Quantifying Cognitive Biases and Reasoning Flexibility of Large Language Models

The llm-strategy-benchmark project, open-sourced by deokjin-choi, aims to systematically evaluate the strategic decision-making capabilities of large language models (LLMs) in complex business scenarios. It quantifies models' cognitive biases, context dependency, and reasoning flexibility through Tesla's historical cases. This project fills the gap in current LLM evaluations regarding strategic decision-making in real-world scenarios, designs a rigorous experimental framework and five diagnostic indicators, reveals key characteristics such as framing effects and situational sensitivity in LLM decision-making, and provides important insights for AI safety and enterprise-level applications.

Current LLM evaluations mostly focus on dimensions like question-answering accuracy and code generation, but lack systematic tools for assessing strategic decision-making capabilities in complex real-world scenarios. The core motivation of this project stems from the questions: "How do large language models reason when faced with strategic problems? What cognitive biases do they exhibit?" It aims to fill this research gap and diagnose the cognitive biases, context dependency, and reasoning flexibility of LLMs in business strategic decision-making.

Core Hypotheses: 1. LLM strategic recommendations change with situational information, and different models vary in their sensitivity levels; 2. When presenting problems using specific company cases like Tesla, there are systematic differences between model decisions and those from anonymous cases (brand/role bias).

Five Diagnostic Indicators: Technology Leadership Preference Index (strategic path preference), Brand Bias Index (impact of brand on decisions), Context Dependency Index (sensitivity to situational information), Numerical Insensitivity Index (sensitivity to numerical changes), Reason-Choice Consistency Score (consistency of reasoning logic).

Experimental Scenarios: Built around 6 key nodes in Tesla's development history (market entry during the founder period, Roadster quality-delivery balance, Model S transition from niche to mass market, Model X design and manufacturing risks, Model 3 production ramp-up, energy infrastructure diversification).

Variable Control: 1. Problem framing type (general anonymous / specific brand); 2. Dynamic context (adding/removing additional data); 3. Multi-model comparison (6 LLMs including Mistral-7B); 4. Temperature parameters (0.0 for determinism / 0.7 for creative reasoning). Each combination is repeated 30 times to ensure statistical robustness.

1. Impact of Situational Framing: In opportunity-oriented contexts, the proportion of technology leadership strategies increased from 15% to 39%; in adverse fact contexts, niche focus increased from 28% to 33%; pure numerical disturbances had limited impact. 2. PCA Analysis: Basic/random numerical scenarios clustered closely, while opportunity/adverse fact scenarios were significantly separated, proving that decision distribution conditions are separable. 3. Impact of Brand Framing: Brand framing does not simply increase or decrease strategic choices, but subtly changes decision sensitivity.

1. Enterprise Deployment Warning: The framing effect and context dependency of LLMs indicate that fully relying on AI for strategic decisions carries risks. 2. New Evaluation Dimensions: Traditional evaluations need to add cognitive bias and robustness tests. 3. Interpretability Tool: This framework provides a structured tool for studying the internal reasoning mechanisms of LLMs.

The llm-strategy-benchmark is a milestone in LLM research's shift from "what it can do" to "how it thinks", revealing the current capability boundaries of LLM strategic decision-making. The project's open-source nature ensures reproducibility and community participation, and it is of great significance for AI safety researchers, enterprise decision-makers, and model developers to understand and quantify cognitive biases and build reliable AI systems.