AI Portfolio Experiment: Benchmarking Large Language Models in Financial Decision-Making Environments

This article introduces the open-source project ai-portfolio-experiment, which builds a standardized experimental framework aimed at systematically testing and comparing the performance of different large language models in financial investment decision-making environments, exploring their application potential in the financial field, and filling the gap in the objective evaluation of LLM's financial decision-making capabilities.

Large language models have shown outstanding performance in text understanding, reasoning, and decision support, sparking interest in their application in the financial investment field. However, how to objectively and standardly evaluate the performance of these models in real financial decision-making environments remains an open question, so the ai-portfolio-experiment project was created to fill this gap.

The project builds a simulated portfolio management environment where models need to make buy, sell, or hold decisions based on market information (historical data, news, financial reports, etc.). The core challenge of the experiment is to create real and controllable test scenarios, with evaluation metrics covering multiple dimensions such as return rate, Sharpe ratio, risk control, and decision consistency, ensuring that model comparisons are statistically meaningful.

The evaluation system includes profit ability (cumulative return, Sharpe ratio, maximum drawdown), risk management (position control, diversified investment, stop-loss strategy), and information processing (extracting valuable information from unstructured text and converting it into trading signals). This method reveals the advantages and limitations of models, such as some models having good trend judgment but insufficient risk control, or being sensitive to news sentiment but struggling to handle complex financial data.

The project provides complete code implementation and standardized datasets (including data preprocessing, model interfaces, backtesting engines, and evaluation metric calculations), supporting researchers to connect custom models or modify parameters. Standardized processes are crucial for AI financial research, as financial markets are highly random, and strict environmental control is required to draw meaningful conclusions.

LLMs show potential in financial decision-making, but have limitations such as overfitting to historical patterns, insufficient response to extreme events, and difficulty in explaining decisions. These findings guide future research and applications: financial institutions can refer to the benchmark test to evaluate AI maturity, researchers can explore model improvement directions based on the framework, and this platform will play an important role between AI and financial practice.