Discriminative Hidden State Readout: A New Paradigm for Multimodal Large Model Sentiment Analysis

This paper proposes the Discriminative Hidden State Readout new paradigm, targeting the Multimodal Sentiment Analysis (MSA) task, and proves that it is more accurate and efficient than traditional generative decoding. The study is based on the Tongyi Qianwen native full-modal model Qwen2.5-Omni-7B, using a lightweight regression head to directly predict continuous sentiment scores from the model's hidden states. Combined with 4-bit quantization and QLoRA optimization, it can be trained on consumer-grade GPUs. Key findings: Generative readout has issues such as precision loss and parsing failures, while discriminative readout achieves SOTA performance on the CMU-MOSI/MOSEI datasets. The original paper is from arXiv (June 4, 2026), link: http://arxiv.org/abs/2606.05713v1.

Multimodal sentiment analysis aims to infer emotional states from linguistic, acoustic, and visual signals. The current mainstream generative readout method uses prompts to let the model generate sentiment scores in text form, but it has fundamental flaws: binding the continuous regression problem to discrete autoregressive decoding introduces computational overhead, easily leads to format errors and numerical out-of-bounds issues, and masks the decisive impact of the readout mechanism on performance.

Core of the discriminative readout paradigm: Bypass text decoding and directly predict continuous values from the model's hidden states.

• Technical architecture: Based on the Thinker module of Qwen2.5-Omni-7B, add a lightweight regression head to the last layer of hidden states, map the hidden state of the last non-padding token to a sentiment score, requiring only a single forward pass.
• Efficient training: 4-bit quantization compresses weights, QLoRA trains only 1.14% of parameters, peak memory usage is 10-21GB on RTX5090 (32GB), which can be reproduced by ordinary researchers.

On the CMU-MOSI and CMU-MOSEI datasets, fix the backbone network, training data, and LoRA configuration to isolate the impact of the readout mechanism:

• Performance comparison: The MAE of discriminative readout (0.551/0.506) is much lower than that of generative readout (>1.1/1.0), and the Corr is higher (0.888/0.790), reaching SOTA.
• Generative issues: Double precision loss, 2.8% of zero-shot outputs cannot be parsed, higher latency, and poor stability.
• Modality ablation: Text dominates in CMU-MOSI, and the performance of text-only is close to the complete model, suggesting the need for more refined modality complementarity analysis.

Key insight: The readout method of large models is as important as the training method. Engineering practice suggestions:

1. Prioritize discriminative readout for continuous value prediction tasks (more accurate and faster);
2. Consumer-grade GPUs can fine-tune 7B full-modal models (QLoRA + 4-bit quantization);
3. Fix other factors when comparing methods to ensure reliable conclusions.

Current limitations: Only verified on sentiment analysis tasks, not extended to other continuous tasks such as time series prediction and physical quantity estimation; discriminative readout sacrifices the interpretability of generative methods (the model cannot explain the reason for the score). Future directions: Verify the applicability of discriminative readout in more tasks, and explore the balance between high accuracy and interpretability.

This study proves through rigorous experiments that discriminative hidden state readout is significantly superior to generative decoding in multimodal sentiment analysis, with higher accuracy, lower latency, and avoids parsing failure issues. For researchers and engineers using large models for continuous value prediction, this is a technical option worth considering.