LLM Financial Decision Evaluation Framework: Subjecting AI Traders to the Rigorous Testing of Quantitative Strategies

Abstract: An empirical research framework for evaluating the performance of large language models in financial trading decisions, supporting multi-level memory systems, five trading personality simulations, and rigorous comparative analysis with traditional quantitative strategies. Keywords: LLM, Quantitative Trading, Financial AI, Backtesting Framework, Behavioral Finance, Memory System, Trading Personality, Statistical Validation, GitHub Open Source

Original Author/Maintainer: tns-research Source Platform: GitHub Project Name: llm-finance-framework Project URL: https://github.com/tns-research/llm-finance-framework Release Date: June 4, 2026

This framework aims to systematically evaluate the performance of LLMs in financial trading decisions, compare their differences with traditional quantitative strategies through rigorous empirical methods, and explore behavioral biases and confidence consistency issues in AI trading.

With the widespread application of LLMs across various industries, the financial sector is exploring the integration of AI into trading decisions, but core questions remain unresolved: Can AI trading capabilities match traditional quantitative strategies? Do they exhibit human-like behavioral biases? Is there consistency between confidence and actual performance?

This open-source framework provides a rigorous empirical methodology, supporting the testing of LLM trading performance on historical data and statistical comparison with mature quantitative strategies to systematically answer the above questions.

Trading Decision Process

The framework simulates an intraday trading cycle: LLMs receive daily market data and technical indicators, and need to make three choices—buy (long), hold (cash), sell (short). Position management is simplified to focus on decision quality.

Five-Layer Prompt Engineering Architecture

The hierarchical memory system mimics human traders:

1. System Prompt Layer (fixed rules and indicator definitions)
2. Raw Market Data Layer (current situation + 20-day technical history)
3. Strategy Log Layer (decisions and explanations from the last 10 trading days)
4. Memory Block Layer (weekly/monthly/quarterly/annual summaries)
5. Performance Summary Layer (real-time comparison with benchmark assets)

Dual-Track Technical Indicator System

• Daily historical sequence: detailed data such as 20-day lagged RSI, MACD histogram, etc.
• Aggregated memory context: statistical summaries (mean, percentage) of indicators over weekly/monthly cycles
• Real-time analysis layer: current RSI, MACD, etc.

Five Trading Personality Simulations

Configurable LLMs can adopt different personalities: Prudent (risk-averse), Aggressive (pursuing excess returns), Balanced (risk-return balance), Momentum (trend-following), Contrarian (reverse positioning). This facilitates analysis of how behavioral frameworks affect decisions.

Memory and Learning Dynamics Research

• Evaluation of hierarchical memory system effectiveness
• Analysis of multi-scale temporal learning and adaptation patterns
• Impact of historical context integration on decisions
• Assessment of adaptive behavior based on performance feedback
• Influence of emotional states on decision quality

Probability Calibration Analysis

• Quantitative measurement of overconfidence/underconfidence patterns
• Calibration analysis by decision type (buy/sell/hold)
• Evaluation of long-term calibration stability

Behavioral Bias Detection

• Quantification of loss aversion
• Identification of disposition effect
• Appropriateness of risk management under uncertainty

Statistical Validation Methods

• Bootstrap resampling test
• Out-of-sample validation
• Risk-based HOLD decision evaluation
• Multi-dimensional comparison with traditional quantitative strategies

Architecture Evolution

• Phase3: Decoupled trading engine, split modules like performance tracking and strategy logs, reducing main process complexity by 29%
• Phase4: Optimized data pipeline, eliminated 54 warnings, batch operations improved DataFrame memory efficiency
• Phase5: Integrated chain-of-thought, supporting structured step-by-step reasoning (can be enabled independently)

Practical Application Value

1. Model Selection Reference: Quantitatively compare the performance of different LLMs on financial tasks
2. Prompt Engineering Optimization: Study the impact of prompt structure on decision quality
3. Risk Management Research: Understand AI behavioral patterns in extreme market conditions
4. Regulatory Compliance Preparation: Provide methodology for auditing and interpretability of AI trading systems

The llm-finance-framework represents an important direction in AI financial research—systematically understanding how AI trades. Through rigorous comparative experiments, multi-level memory systems, and behavioral personality simulations, it provides a scientific methodology for researching the capability boundaries and limitations of LLM financial decisions.

For researchers and practitioners in the intersection of AI and finance, this is an open-source project worth exploring in depth.