Machine Mental Imagery: Resolving Representational Ambiguity in Dialogue with Visual Scaffolding

The research team proposes an active visual scaffolding framework that incrementally converts dialogue states into a persistent visual history to address the "representational ambiguity" problem in situational dialogue. Tests of this framework on the IndiRef benchmark show that hybrid multimodal representations significantly outperform text-only methods, providing a new path for dialogue systems to maintain precise common ground.

Human dialogue relies on a "common ground" formed by shared contexts, which naturally maintains precise understanding of specific objects. However, current text-only dialogue systems based on large language models tend to lose distinguishing information in text descriptions when handling specific objects in physical environments (e.g., subtle differences between a red apple and a red tomato), leading to "representational ambiguity". This ambiguity accumulates, causing "false grounding" and making it difficult to maintain precise reference chains, especially in long-term dialogues.

Researchers drew inspiration from "mental imagery" in human cognitive science. When thinking and communicating, humans rely on descriptive, persistent, manipulable, and language-complementary visual internal simulations. With the development of multimodal models (e.g., GPT-4V, Gemini), it has become possible to endow dialogue agents with similar "mental imagery" capabilities, with the core being the active construction and maintenance of visual situational representations.

The core of the framework is to incrementally convert dialogue states into visual representations, maintaining two parallel representations: text (handling abstract concepts) and visual (handling spatial relationships and details). Key mechanisms include: 1. Incremental externalization: Whenever new visual information is introduced in the dialogue, actively generate/update visual representations (e.g., add objects, adjust positions); 2. Scene commitment: When generating visual representations, specific visual decisions must be made (e.g., object size, color) to enforce concreteness and reduce ambiguity.

On the IndiRef (multi-turn reference resolution) benchmark, four settings were compared: text-only baseline, incremental externalization, complete visual scaffolding, and hybrid multimodal. Results show: Incremental externalization itself improves performance; the complete framework significantly outperforms text-only; hybrid multimodal (flexible use of text/visual) performs best, verifying the value of multimodal complementarity.

The current framework has limitations: 1. Visual generation quality constraints (insufficient handling of complex scenes and details); 2. High computational cost; 3. Focus only on the visual modality; 4. Lack of long-term memory integration. Future directions include: optimizing visual generation technology, reducing computational costs, expanding multimodal coverage, integrating long-term memory, etc.

The study shows that multimodality is a necessity rather than a luxury for situational dialogue; actively constructing visual representations (instead of passive processing) is key to advanced dialogue capabilities; future dialogue systems need to rethink cognitive architectures, integrating multimodal representations and reasoning mechanisms to break through the bottlenecks of text-only systems.