Research on Discourse Strategies of Large Language Models When Facing Information Scarcity

This article explores the discourse strategies of large language models (LLMs) when facing knowledge gaps, including phenomena such as vague expressions, fictional information, and strategic avoidance. It uses the little-known legend of Fanciulla di Vagli from the Tuscany region of Italy as a case for in-depth analysis. The study reveals the complex behaviors of LLMs in responding to cognitive uncertainty, which is of great significance for improving AI safety and reliability.

Although LLMs are trained on massive amounts of data, they are not omniscient. When dealing with niche topics with insufficient training data coverage, their response methods are oversimplified by the traditional label of 'hallucination'; in reality, there are multiple complex discourse strategies—from cautious vagueness to confident fabrication, and then to strategic avoidance.

Case Selection

The extremely niche legend of Fanciulla di Vagli (a mysterious female figure from Tuscany, Italy) was selected. Due to the scarcity of relevant literature, it serves as an ideal object to test LLMs' response strategies to information scarcity.

Methodological Contributions

Adopt a discourse analysis framework:

• Go beyond accuracy metrics and focus on 'how to answer' rather than just right or wrong
• Contextually evaluate model behavior under information scarcity
• Cross-model comparison of the performance of different architectures and training strategies

1. Vagueness Strategy: Use qualifiers like 'may' or 'it is said' to reduce the certainty of statements and imply knowledge boundaries
2. Fictional Construction: Generate content that seems reasonable but is false, which is hard for ordinary users to distinguish from the truth
3. Strategic Avoidance: Shift topics, provide general information, or suggest consulting materials
4. Knowledge Attribution: Cite non-existent literature, make vague attributions, or confuse similar events

1. Tension in Training Objectives: With the goal of predicting tokens, prioritize generating content that 'sounds correct' rather than factually accurate
2. Side Effects of Instruction Fine-tuning: Trained to be a 'helpful assistant', prefer to give uncertain answers rather than admit 'I don't know'
3. Confidence Bias: Express facts and speculations with similar confidence levels, making it difficult for users to distinguish reliability

1. Retrieval Quality Threshold: Set quality standards for retrieval results and prompt users when they are insufficient
2. Uncertainty Quantification: Distinguish between high and low confidence statements and proactively declare knowledge boundaries
3. User Education: Help users understand the limitations of LLMs and cross-verify outputs on niche topics

Developers

• Implement uncertainty detection and mark low-confidence content
• Design an elegant fallback response for 'I don't know'
• Integrate source verification functions
• Strengthen uncertainty expressions for high-risk fields

Users

• Maintain critical thinking and treat outputs as a starting point
• Be alert to model outputs on niche topics
• Request verifiable sources
• Cross-verify key facts from multiple sources

"Answering without knowing" is a core feature of LLMs when facing knowledge gaps. This study reveals their discourse strategies and reminds us to clearly recognize their limitations while enjoying the convenience of AI. As LLMs are increasingly applied in key fields, understanding and addressing these strategies will become core issues for AI safety and reliability.