LemonadeBench: Evaluating the Economic Intuition of Large Language Models

LemonadeBench is a benchmark project dedicated to evaluating the economic intuition of large language models (LLMs), aiming to fill the gap in the assessment of LLMs' economic reasoning capabilities. Through the classic lemonade stand scenario, it tests models' reasoning abilities on core economic concepts such as supply-demand relationships, pricing strategies, and market dynamics, which is of great significance for evaluating models' practical reasoning skills.

Large language models excel in mathematical computation, code generation, and natural language understanding, but their performance in economic intuition (such as understanding complex concepts like supply-demand relationships, market dynamics, and cost-benefit analysis) has not been fully evaluated. Economic intuition is a key component of models' practical reasoning abilities, so targeted benchmarks are needed to measure this capability.

The lemonade stand is a classic introductory case in economics education, covering core concepts such as fixed and variable costs, changes in supply-demand curves, price elasticity, and profit maximization strategies. This scenario is concise and close to reality; it requires models to understand the logic behind business decisions rather than just perform numerical calculations, which can fully test models' economic intuition.

LemonadeBench evaluates models from four dimensions:

1. Cost Analysis: Identify fixed costs (e.g., stall rent) and variable costs (e.g., raw materials), and calculate the break-even point;
2. Pricing Strategy: Propose reasonable pricing based on market conditions (e.g., increased demand in hot weather), considering the impact of price on sales volume;
3. Market Dynamics: Strategies to respond to competitor entry or raw material price fluctuations;
4. Long-term Planning: Consistency of multi-cycle decisions, including inventory management, seasonal adjustments, and return on investment analysis.

Testing mainstream LLMs reveals that most models perform well in pure mathematical calculations (cost and profit calculations), but have shortcomings in situational understanding and strategic reasoning (e.g., raising prices without considering demand elasticity, ignoring fixed costs). Some advanced reasoning models can conduct multi-step analysis and consider the interaction of multiple factors, indicating that targeted training can improve economic intuition.

Academic Value: Provides a new perspective for research on LLMs' reasoning abilities, emphasizing practical reasoning and situational application; Application Value: Has direct reference value for fields such as finance, business consulting, and policy analysis; Future Directions: Expand complex scenarios (multi-market competition, macro shocks), explore causal reasoning tests, and optimize model training based on evaluation results.

LemonadeBench reminds us that evaluating LLMs should not only focus on knowledge reserve and computational ability but also pay more attention to reasoning and decision-making abilities in complex real-world scenarios. Economic intuition is an important manifestation of practical intelligence; with the improvement of such benchmarks, it is expected to better understand and enhance the practical application value of LLMs.