Study Reveals: How Large Language Models Infer User Identity from Conversation History and Impact Advice Quality

The University of Mannheim team in Germany conducted a study focusing on two core questions: Can large language models (LLMs) infer user identity from conversation history? If yes, does such inference change the advice, recommendations, and other high-impact responses given by the models? The study covers key issues including fairness and bias, consistency of advice quality, and user privacy boundaries, aiming to explore the potential risks and governance directions of LLM personalization effects.

LLMs are increasingly integrated into daily life (writing assistance, professional consulting, etc.). The interactive mode of multi-turn conversations between users and models raises a question: Will models remember user characteristics from conversation history and adjust their responses? The University of Mannheim team systematically explores this issue, focusing on identity inference capabilities and their impact on advice.

Understanding the personalization effects of LLMs is of great significance:

1. Fairness and Bias: If models infer background information such as gender or race based on language style and adjust their answers, it may constitute hidden algorithmic bias;
2. Advice Quality and Consistency: In key decision-making scenarios, users expect objective advice—if influenced by subjective impressions, system reliability will be compromised;
3. Privacy Boundaries: Users may not be aware of revealing identity information, and the implicit construction of user profiles by models involves privacy issues.

Although specific details are not fully disclosed, the methodological framework is speculated to include:

1. Conversation History Manipulation: Design experimental conversations to manipulate user characteristic cues such as language style, topic preferences, and situational information;
2. Identity Inference Testing: Test whether the model forms specific identity inferences through direct questions or indirect probing;
3. Advice Bias Measurement: Parallel scenario testing, coverage of sensitive fields (medical/legal, etc.), and quantitative bias indicators to evaluate the impact of identity inference on advice quality.

The study is expected to contribute knowledge at multiple levels:

• Technical Level: Reveal the context understanding capabilities and limitations of LLMs; if systematic biases exist, it indicates problems with stereotypes in training data;
• Ethical Level: Provide an empirical basis for AI ethics discussions and help formulate targeted governance strategies;
• Application Level: Guide developers to introduce forgetting mechanisms or fairness constraints to ensure fair and consistent advice in high-risk scenarios.

Regardless of the study results, the following design principles are indicated:

1. Transparency and Interpretability: Users should understand how models use conversation history to personalize responses and be provided with control options;
2. Scenario-Specific Fairness Constraints: Mandatory fairness checks are required in sensitive fields to avoid advice being unduly influenced by user profiles;
3. Regular Audits and Testing: Conduct regular fairness audits after launch to detect response biases among groups;
4. User Control Rights: Grant users the right to delete conversation history and reset the model's impression of them.

The study has limitations:

• Model Coverage: It may focus on mainstream models, and the applicability of results needs verification;
• Cultural Context: Identity cues vary across cultures, so the cross-cultural applicability of results needs to be tested;
• Dynamic Evolution: LLMs iterate rapidly, so the problem may change. Future directions: Develop automated fairness detection tools, explore technical means to reduce identity inference, and conduct interdisciplinary research on users' perceptions of AI personalization.