Using LLM to Analyze Wall Street Journal Headlines for S&P 500 Prediction: A Practical Guide to Quantitative Trading Strategies with Financial Text

This project focuses on 146,000 Wall Street Journal headlines from 2016 to 2023. It builds a strategy to predict the next-day movement of the S&P 500 index using FinBERT sentiment analysis and LSTM deep learning models. It also compares the risk-adjusted returns of three strategies (momentum, mean reversion, and surprise), conducts rigorous evaluations using methods like Fama-French factor attribution, and explores the feasibility and practical paths of quantitative trading with financial text.

Price prediction in financial markets is a core challenge in quantitative investing. Traditional methods rely on technical indicators and macroeconomic data but ignore news text information. With the rise of LLMs, extracting trading signals from unstructured text has become possible. As a course practice project, the core question is: Can we use daily Wall Street Journal headlines to predict the next-day movement of the S&P 500? This question is challenging due to news noise, metaphors, and non-linear market reactions.

Core Data Sources: 146,000 Wall Street Journal headlines from 2016 to 2023, daily price data of the S&P 500, Fama-French three-factor data, and a subset of 16,000 headlines with sentiment polarity labeled by FinBERT.

Preprocessing Flow: TF-IDF vectorization to capture word importance → PCA dimensionality reduction to mitigate high-dimensional sparsity → K-Means clustering to discover potential topic clusters.

Baseline Model: TF-IDF + PCA + Logistic Regression. Its advantage is transparency, as it can intuitively show the contribution of words to sentiment classification.

Advanced Model: LSTM neural network. Through tokenization and padding, LSTM layers to learn sequence dependencies, and a classification head to output sentiment predictions, it can understand subtle semantic differences and long-distance dependencies.

Based on the daily aggregated sentiment score signal St, three strategies are built:

1. Momentum Strategy: Go long on positive sentiment, short/hold cash on negative sentiment (assuming sentiment persistence);
2. Mean Reversion Strategy: Go long on extremely negative sentiment, short on extremely positive sentiment (assuming recovery after overreaction);
3. Surprise Strategy: Go long on sudden sentiment increases, short on sudden drops (based on 30-day rolling average deviation, assuming reaction driven by expectation gaps).

Distinguish between in-sample (2016-2021) and out-of-sample (2022-2023) periods. Evaluation metrics include: Basic Returns (cumulative/annualized return); Risk Adjustment (annualized volatility, Sharpe ratio, maximum drawdown, Calmar ratio); Factor Attribution (annualized Alpha and significance obtained via Fama-French three-factor regression); Cost Realization: Calculate net returns considering 5bps/10bps fees.

Visual Insights: TF-IDF word contribution chart, K-Means topic clustering, classification metric radar chart.

Methodological Insights: 1. Text data has a forward-looking information advantage; 2. A trade-off between model complexity and interpretability is needed; 3. The same signal can derive multiple strategies; 4. Small fees have a significant impact on high-turnover strategies.

Current Limitations: Single data source, only using headlines (losing details), simplified binary sentiment classification, insufficient linear factor models.

Potential Expansions: Multi-source data fusion (social media/earnings calls, etc.), fine-grained sentiment (intensity/specific emotions), high-frequency implementation, reinforcement learning to optimize strategy parameters.