Research on Choice Complexity of Large Language Models: A Two-Tier Evaluation Framework from the Perspective of Decision Theory

This article proposes a two-tier framework based on the perspective of decision theory to evaluate the choice complexity of large language models (LLMs). Through two dimensions—CCI (Choice-Set Complexity Index) and ILDC (Internal Decision Difficulty Coefficient)—combined with inference-time control mechanisms, it provides a new theoretical tool for understanding and optimizing the decision-making behavior of LLMs. The core goal is to systematically analyze the performance boundaries of LLMs in complex choice scenarios, helping to improve model reliability and practicality.

LLMs perform well in daily conversations, but still face challenges when dealing with complex decisions involving trade-offs between multiple options. Choice complexity is a classic concept in decision theory, and this study introduces it into the field of LLMs. Classic theories such as bounded rationality (decision-makers cannot always make optimal choices due to cognitive limitations) and prospect theory (humans' asymmetric risk preferences for gains and losses) provide the foundation for this research. Using this framework, key questions can be explored, such as whether LLMs exhibit human-like decision biases, and how model size and training methods affect decision patterns.

The framework consists of two complementary dimensions:

1. CCI (Choice-Set Complexity Index)：Measures the inherent complexity of the external choice set, including the number of options, similarity, size of attribute dimensions, dominance/trade-off relationships, etc. For example, choosing among dozens of laptops with similar configurations has a higher CCI than choosing among three products with obvious differences.
2. ILDC (Internal Decision Difficulty Coefficient)：Reflects the internal difficulty of a specific LLM in handling a task, depending on the model's knowledge reserve, reasoning ability, alignment training method, etc. For the same task, ILDC varies significantly across models of different sizes/training data (e.g., small models have higher ILDC in multi-step reasoning choices).

The framework introduces an inference-time control mechanism, which is different from traditional fixed-configuration evaluation. When high CCI or high ILDC is detected, the system can trigger mitigation strategies: increasing computational budget (e.g., more sampling steps), activating Chain-of-Thought prompts, switching to conservative decision mode, etc. This dynamic adaptation capability helps LLMs maintain stable performance in complex decision scenarios.

The framework can be applied to scenarios such as recommendation systems (evaluating the CCI of recommendation lists to avoid user choice overload), medical decision support (measuring the complexity of treatment plan choices), and financial investment (analyzing the difficulty of portfolio choices). Experimental design steps: Construct choice sets with different CCI levels → Let LLMs make decisions → Measure ILDC indicators (inference time, confidence, consistency, etc.) → Analyze the relationship between CCI and ILDC to reveal the model's decision characteristic curve.

Implications for LLM Development: Need to focus on performance in complex decision scenarios (not just simple question-and-answer accuracy); systematically identify model weaknesses to guide improvements; assist alignment research (identify inconsistent behaviors) and efficiency optimization (dynamically allocate computational resources). Future Directions: Cross-model comparison (decision complexity curves of different architectures/sizes), dynamic difficulty adjustment (real-time adjustment of task difficulty based on ILDC), human-machine collaboration (optimal decision right allocation), multimodal extension (complexity measurement of visual-language decisions), etc.

The choice-complexity-llm project provides a new perspective for LLM evaluation and optimization through the lens of decision theory. The two-tier framework of CCI and ILDC accurately describes the decision challenges faced by LLMs and provides a theoretical basis for inference-time control. As LLMs are increasingly applied in real-world complex decision scenarios, understanding and controlling choice complexity will become key to improving model reliability and practicality.