ShoggothBench: Quantifying Personality Deviation and Behavioral Uninterpretability of Large Language Models

Title: ShoggothBench: Quantifying Personality Deviation and Behavioral Uninterpretability of Large Language Models Abstract: ShoggothBench is a benchmark framework for measuring behavioral deviations of large language models under role pressure. By comparing differences between declared personality, other personality patterns, general strategy behaviors, and residual uninterpreted behaviors, it helps identify elusive "Shoggoth candidate" behavioral patterns. Source Information: Original author/maintainer: nikakogho; Source platform: GitHub; Original link: https://github.com/nikakogho/ShoggothBench; Release date: 2026-05-31 Core Value: Provides AI safety researchers with a quantifiable tool to explore the consistency between the model's internal mechanisms and its surface personality settings.

Background and Motivation

LLM alignment training often relies on the "personality choice" assumption, but whether this assumption reflects the model's internal mechanisms is questionable. The name ShoggothBench comes from the Shoggoth in Cthulhu Mythos (an amorphous entity that can mimic forms), metaphorically referring to the possibility that models may have elusive internal behavioral patterns inconsistent with their surface personalities. Its creation aims to answer: Are model behavior changes driven by declared personality, or do they stem from other uninterpreted internal mechanisms?

Core Objectives and Design Philosophy

ShoggothBench v0's core objective is to test whether the model's behavior under role pressure can be categorized into the following four patterns:

1. Declared Personality: The role or guidelines the model is explicitly told to follow;
2. Other Personality Patterns: Actual patterns followed that differ from the declared personality;
3. General Strategy/Rejection Behavior: Standard rejection or conservative responses to sensitive topics;
4. Shoggoth Candidate Category: Residual behaviors that cannot be explained by the first three patterns, implying potential elusive internal mechanisms. Note: This framework does not directly detect the model's "internal goals"—it only operationalizes uninterpreted behavioral changes.

Technical Implementation and Evaluation Methods

ShoggothBench is presented as a Jupyter Notebook, serving both as a runnable codebase and an interactive research report. Researchers can:

• Reproduce the evaluation process;
• Visualize behavior distribution under different test conditions;
• Compare performance differences between models;
• Explore cases of behavioral anomalies. The core contribution is converting abstract "personality consistency" into measurable quantitative indicators of deviation.

Practical Significance and Application Scenarios

For AI Safety Researchers:

• Verify the effectiveness of alignment training and check if the model truly follows the preset personality;
• Identify potential safety risks and discover uninterpreted behavioral patterns;
• Standardize tests to compare the stability of different models.

For Model Developers:

• Detect the phenomenon of "personality drift" (deviation from initial settings during long-term operation or complex scenarios).

Limitations and Future Directions

Current Limitations:

• Limited test coverage, focusing on specific role pressure scenarios;
• Lack of strict theoretical basis for determining "Shoggoth candidate" behaviors;
• No large-scale benchmark dataset to support cross-model comparisons.

Future Directions:

• Expand test scenarios to cover more types of role pressure;
• Establish a more refined behavioral classification system;
• Combine tools like activation patching and sparse autoencoders to understand uninterpreted behaviors from a mechanistic perspective.

Conclusion

ShoggothBench is an important attempt in LLM safety research, converting abstract "personality consistency" into an actionable evaluation framework. While it does not claim to reveal the model's "true internal goals", it provides a starting point for identifying and quantifying behavioral phenomena that cannot be explained by existing theories. In an era where AI systems are becoming increasingly complex, this humble and pragmatic research attitude holds significant value.