From TF-IDF to Transformer: The Evolution of Sentiment Classification and New Explorations in Ensemble Learning

This article discusses the evolution of sentiment classification tasks from traditional machine learning to modern Transformer models. Through comparative experiments on the IMDb movie review dataset, it reveals that RoBERTa leads with an accuracy rate of 93.02% and verifies the effectiveness of the soft voting ensemble strategy.

The development of sentiment classification reflects the evolution of NLP technology: early methods relied on handcrafted features like TF-IDF and statistical models such as Naive Bayes and Logistic Regression, but ignored context; after the rise of deep learning, RNN/LSTM could model sequence information, while the Transformer architecture and pre-trained models (e.g., BERT/RoBERTa) achieved better semantic understanding.

The experiment is based on the IMDb movie review dataset. The compared models include traditional machine learning (Naive Bayes, Logistic Regression, SVM), ensemble learning (LightGBM), deep learning (LSTM), and pre-trained Transformers (RoBERTa, DistilBERT); multiple metrics including accuracy, precision, recall, F1, and ROC-AUC are used for evaluation.

RoBERTa ranks first with an accuracy rate of 93.02%. Its optimized training strategies (more data, larger batches, removal of the NSP task) enable it to learn more robust language representations; soft voting ensemble improves performance by weighted averaging the prediction probabilities of each model, reflecting the complementarity of models.

The self-attention mechanism of Transformer can globally focus on text positions and capture long-distance dependencies; the pre-training-fine-tuning paradigm uses massive unlabeled data to learn general language knowledge, with significant transfer effects. Compared to traditional methods that rely on handcrafted features and LSTM's low efficiency in sequential processing, Transformer has obvious advantages.

1. Prioritize RoBERTa when resources are sufficient; 2. Use lightweight models like DistilBERT for real-time scenarios; 3. Consider soft voting ensemble for critical businesses; future directions include multimodal, aspect-level, and cross-language sentiment analysis, with Transformer remaining the core.

From TF-IDF to Transformer, sentiment classification has achieved a leap from statistical word frequency to semantic understanding. RoBERTa's high accuracy marks an important progress in machine sentiment understanding. Future technologies will play a valuable role in more fields, helping humans utilize text sentiment information.