ShoggothBench: Quantifying Role Consistency Deviation of Large Language Models

This article introduces the ShoggothBench project maintained by nikakogho, an evaluation benchmark for measuring the deviation between large language models (LLMs) and role selection models, aiming to help researchers understand and improve the role-playing consistency of AI systems. The project source is GitHub, and the release date is May 31, 2026.

Although LLMs can simulate specific roles in role-playing tasks, they tend to deviate from the settings during long conversations and revert to their base training personas, affecting user experience and applications in scenarios like entertainment and education. The project name "Shoggoth" is derived from the shape-shifting creature in Lovecraftian mythology, metaphorically representing that LLMs can imitate various roles but remain unchanged in essence. The benchmark aims to quantify the extent to which models maintain their roles and the exposure of their base characteristics.

ShoggothBench uses a "role selection model" as the reference benchmark, quantifying role deviation by comparing the output differences between the tested LLM and this model. It designs four evaluation dimensions: Style Consistency (vocabulary, sentence structure, tone matching), Knowledge Consistency (alignment with the role's knowledge background), Behavioral Consistency (alignment with character and behavioral habits), and Temporal Stability (consistency maintenance in long conversations).

The dataset covers types such as historical figures, fictional characters, and professional roles, with each role accompanied by a detailed setting document. The test uses an adversarial dialogue design: after presenting the role settings, multiple rounds of dialogue are conducted, and "probe questions" are inserted to induce the model to expose base training knowledge or default style, then responses are analyzed to evaluate the role maintenance ability.

Experiments reveal: 1. Role complexity is negatively correlated with consistency—more unique roles are more prone to deviation; 2. System prompts that clarify role boundaries can reduce deviation rates; 3. The relationship between model size and consistency is non-monotonic—medium-sized models may perform better in specific roles.

Application Value: Guiding model training optimization and prompt engineering design. Limitations: It does not involve multimodal roles (consistency of expressions and actions); it does not consider "beneficial deviation" scenarios (such as correcting errors or safety information). Future work needs to expand evaluation dimensions and incorporate this concept.

ShoggothBench provides a systematic evaluation tool for LLM role-playing capabilities. By quantifying deviation degrees, it helps identify areas for improvement and promotes continuous optimization of AI in terms of role consistency.