EconBench: Evaluating the Economic Rationality of Large Language Models Using Behavioral Economics Experiments

EconBench is a benchmark tool specifically designed to test the economic preferences and rational decision-making abilities of large language models (LLMs). It evaluates AI's decision-making performance in risk, time, and social interaction scenarios through classic behavioral economics experiments. It fills the gap in existing AI benchmarks for the systematic evaluation of economic decision-making capabilities, helps understand the decision logic and "economic personality" of LLMs, and is of great significance for model selection, safety assessment, improvement, and AI alignment research.

Economic rationality is a core concept in decision theory, referring to an individual's ability to make optimal choices under constraints of limited information and resources. Traditionally, economists study human economic behavior through laboratory experiments, but existing AI benchmarks mostly focus on language understanding, code generation, etc., and lack systematic evaluation of economic decision-making capabilities. Therefore, Josh R. Foster developed EconBench, which transforms classic behavioral economics experiments into automatically runnable benchmark tests.

EconBench evaluates LLM economic behavior from three dimensions:

1. Risk and Rationality: Detect violations of the independence axiom of expected utility theory through the Marschak-Machina triangle experiment;
2. Social Preferences: Measure altruistic tendencies and fairness sensitivity through the Dictator Game and Ultimatum Game;
3. Time Preferences: Derive discount rates through intertemporal choice experiments and detect present bias using the Beta-Delta model.

EconBench is developed using Python 3.8+, with a modular architecture including:

• Model Registry: Supports OpenAI (GPT-4o, etc.), Anthropic (Claude series), Google (Gemini series), and open-source models (e.g., Llama-3.1-70B-Instruct);
• Experiment Task Scripts: independence.py (independence axiom test), social.py (social preference test), time.py (time preference test);
• Visualization Dashboard: After running python3 -m http.server 8000, access http://localhost:8000/web/ to view results.

The application value of EconBench includes:

1. Model Selection: Compare the performance of different LLMs in economic decision-making tasks;
2. Safety Assessment: Identify biases and rational flaws of models in financial decision-making or resource allocation applications;
3. Model Improvement: Targeted optimization of training data or fine-tuning strategies;
4. AI Alignment Research: A quantitative tool to help understand model behavioral tendencies.

Limitations: Behavioral economics experiments are designed based on humans, so direct application to AI requires additional validation; model responses are affected by prompts and context, making standardized testing conditions a challenge. Future directions: Expand to scenarios such as auctions and repeated games; evaluate model performance in market environments by combining real financial datasets.

EconBench represents a new AI evaluation paradigm, focusing not only on language capabilities but also exploring decision logic and "economic personality". As AI plays an increasingly important role in business, finance, and policy-making, such tools help understand and trust AI decision-making processes and are open-source projects worth paying attention to and contributing to.