Application of Frozen Multimodal Embeddings in Psychological Assessment for Asynchronous Video Interviews: Solutions for the ACM Multimedia AVI Challenge 2026

The research team proposes using frozen multimodal encoders (CLIP, Whisper, RoBERTa, etc.) for personality and cognitive ability assessment in asynchronous video interviews (AVI). They achieved results significantly better than the baseline in the ACM Multimedia AVI Challenge 2026, while revealing potential dataset shortcut issues in cognitive ability prediction.

New Frontiers of Asynchronous Video Interviews

Asynchronous video interviews (AVIs) have transformed recruitment assessment methods. They require automatic evaluation of psychological traits from visual, acoustic, and linguistic signals in videos, but labeled data is limited, posing a challenge for multimodal learning.

AVI Challenge 2026 Tasks

• Track1: Personality Trait Prediction: A regression task to predict continuous scores for the six HEXACO dimensions (Honesty-Humility, Emotionality, Extraversion, Agreeableness, Conscientiousness, Openness).
• Track2: Cognitive Ability Classification: A classification task to categorize candidates into different cognitive ability levels.

Reasons for Choosing the Frozen Strategy

1. Data scarcity: Limited labeled samples make fine-tuning prone to overfitting;
2. Representation quality: Pre-trained models already have high-quality general representations;
3. Computational efficiency: Freezing reduces training costs;
4. Generalization ability: Maintaining pre-trained weights is beneficial for generalization.

Multimodal Encoder Combination

• Visual: CLIP captures facial expressions, body language, etc.;
• Acoustic and Transcription: Whisper provides acoustic features like intonation and speech rate, as well as text transcription;
• Text: RoBERTa (general understanding), E5 (semantic similarity), DeBERTaV3 (long-distance dependencies).

Downstream Model Design

• Lightweight linear layers/small MLPs;
• Train a separate model for each trait;
• Late fusion of multimodal information.

Progressive Improvement Path

1. Global Model: A single model predicts all traits with an MSE of 0.3189;
2. Single Trait Modeling: Train independently for each trait with an MSE of 0.2871;
3. Single Trait Late Fusion: Integrate multimodal information at the trait level with an MSE of 0.2696.

Performance Comparison

• Official baseline MSE: 0.3334;
• Relative improvement of the final model: 19.1%;
• Stable performance on the validation set with statistical significance.

Unexpected Results

• Official baseline accuracy: 0.4062;
• Multimodal ensemble model: 0.5313;
• Simple topic attribute baseline (metadata like age, education): 0.5781 (better than the multimodal model).

Dataset Shortcut Hypothesis

• Systematic differences exist in the distribution of topic attributes between the validation and training sets;
• Topic attributes (e.g., education level) are highly correlated with cognitive labels;
• Models rely on shortcuts rather than AVI content to infer cognitive ability.

Challenges in Robust Cognitive Inference

Cognitive ability is complex, with high variability in performance and context dependence, making it difficult to accurately assess from short video clips.

1. Specific Trait Modeling: Different traits rely on different modal cues, so independent modeling is better;
2. Late Fusion Strategy: Integrate high-level information after independent encoding of each modality to avoid early fusion noise;
3. Beware of Dataset Shortcuts: Use simple baseline tests to identify potential issues;
4. Effectiveness of Frozen Encoders: Balance representation quality and complexity in small-sample scenarios to avoid overfitting.

• Data scale limitation: Small samples restrict generalization; need to explore semi-supervised/self-supervised methods to utilize unlabeled data;
• Cross-dataset validation: Need to validate cross-cultural and cross-domain generalization on diverse datasets;
• Cognitive assessment improvement: Fine-grained decomposition of cognitive abilities, multi-task learning, adversarial debiasing techniques.

This study achieved significant progress in the AVI personality assessment task through the frozen multimodal embedding strategy, while revealing potential challenges in cognitive ability prediction. The core contributions lie not only in technical methods but also in methodological insights: AI psychological assessment needs to pursue both performance improvement and mechanism understanding, and high accuracy must be based on models truly learning from content. This lays the foundation for building more reliable and interpretable AVI psychological assessment systems.