Dynamic Alignment: Rethinking Human-AI Value Alignment Evaluation Through Longitudinal Research

Core argument of this paper: Current human-AI alignment evaluation methods assume static user preferences and rely on immediate feedback (e.g., RLHF/DPO), but in reality, user preferences change over time and context. The study proposes a longitudinal, contextualized alignment measurement framework and validates it through the BITE browser system, finding significant differences between users' immediate preferences and subsequent reflections, thus revealing the limitations of traditional methods.

The Neglected Temporal Dimension

Current LLM alignment research (e.g., RLHF, DPO) assumes static user preferences and relies on immediate feedback at the end of interactions. However, real-world decisions are time-extended: the consequences of AI-assisted decisions lead users to re-evaluate later.

Limitations of Immediate Feedback

1. Time-extended nature of decisions: The actual consequences of AI outputs (e.g., email replies, travel experiences) affect final evaluations, but existing datasets ignore this dimension.
2. Cognitive limitations: Immediate judgments are influenced by cognitive biases (availability heuristic, anchoring effect), while evaluations after careful consideration are more rational.

Three-Pronged Framework

1. In-context preference capture: Collect immediate feedback as the starting point for tracking.
2. Context-triggered subsequent reflection: Trigger re-evaluation at key decision points (e.g., receiving an email reply, completing a purchase).
3. Privacy-preserving behavior trajectory: Collect de-identified behavior data to explain preference changes, with a user-led consent mechanism.

BITE System Implementation

• Key interaction detection: Identify impactful LLM interactions (decision-making, planning, etc.).
• Progressive consent: Request permissions in stages; users can manage data at any time.
• Contextualized reflection triggering: Prompt reflection at relevant moments (returning to view outputs, new operations linked to old decisions).

Differences Between Immediate and Subsequent Preferences

A two-week study with 8 participants showed:

• Accuracy: Some answers marked as 'accurate' immediately were later rated as 'partially accurate' or 'misleadingly accurate' (missing key information).
• Relevance: Many outputs initially considered 'relevant and useful' were found to not solve the problem after actual application.

Patterns of Preference Change

1. Satisfaction → Disappointment: Good on the surface but with practical limitations;
2. Skepticism → Recognition: Found effective after verification;
3. Context-dependent shift: The same output is evaluated differently in different contexts.

Implications for Existing Datasets

Datasets based on immediate feedback may have systematic biases, leading to overestimation of model performance and misjudgment of alignment levels.

Significance

• AI Safety: Detect long-term issues earlier (reward hacking, value drift);
• User Experience: Products need to provide re-evaluation mechanisms to avoid solidifying user profiles.

Limitations

1. Small sample size (8 participants);
2. Short time span (two weeks);
3. Self-selection bias (participants have high technical acceptance);
4. Scenario limitation (browser environment only).

Future Exploration Directions

1. Large-scale longitudinal datasets: Build datasets spanning months/years with thousands of users;
2. Dynamic alignment training: Develop algorithms that use longitudinal signals (online learning, continuous adaptation);
3. Cross-cultural research: Explore preference changes across different cultures;
4. Automated detection: Automatically identify preference changes without explicit queries.

Conclusion

Human-AI alignment needs to embrace the temporal dimension; static preference signals cannot reflect real needs. The longitudinal perspective is key to building truly aligned AI systems, especially when AI is involved in important decisions.