Guess What AI Thinks: Understanding the Cognitive Patterns of Vision-Language Models Through Games

This project is an interactive game where players predict the label choices of the vision-language model SigLIP for images. It aims to help people intuitively understand how AI "sees the world", revealing its cognitive biases and decision-making patterns. The project combines gameplay and educational value, encourages community participation through open-source models, and provides a new path for AI interpretability and literacy education.

With the widespread application of vision-language models (VLMs) in fields like image recognition and autonomous driving, the "black box" problem of their decision-making logic has become prominent—how do models "see" the world, and do they have cognitive biases? Traditional evaluations only focus on metrics like accuracy, making it difficult to reveal the real logic behind decisions. This project uses gamification to allow ordinary users to experience the cognitive process of AI.

Core Design: Players need to predict the label choices of the SigLIP model for images (not the "correct answer"), shifting the human-machine interaction perspective to "humans understanding AI". Technical Architecture:

• Core model: Google's SigLIP2 (google/siglip2-base-patch16-224), which calculates image-text matching degree through contrastive learning;
• Application framework: Interactive web application built with Streamlit;
• Game mechanism: Custom theme pack labels → SigLIP scoring → Probability distribution → Display Top3 predictions and confidence levels.

The project includes 4 theme packs:

1. Animal Pack: Test the ability to recognize common/rare animals, compare the performance of similar-looking species (e.g., wolves and huskies);
2. Food Pack: Explore the cognition of global cuisines, reveal cultural biases in training data;
3. Tech Items Pack: Examine the ability to distinguish between modern electronic devices and traditional tools;
4. Illusion Pack: Test model robustness—when human vision is deceived, will AI "see wrong"?

The game flow is simple and intuitive:

1. Select a theme pack → 2. Observe the image → 3. Predict the AI's label choice → 4. View the model's actual results and confidence levels →5. Track scores/accuracy/consecutive correct answers. The design creates "aha moments": when predictions do not match the model's output, players will think "why does the AI think this way", deepening their understanding of decision-making logic.

Model Insights:

• Images with clear features: The model performs stably and is easy to guess correctly;
• Confident mistakes: Giving wrong answers with high confidence, revealing that AI only relies on statistical patterns rather than true "understanding";
• Probability distribution: Top3 shows whether the model is "hesitant" or "confident";
• Biases and blind spots: Reflect uneven distribution of training data. Educational Applications:
• AI researchers: Quickly test model behavior;
• Educators: Interactive teaching aids for AI courses;
• Product managers: Understand the boundaries of AI capabilities;
• General public: Lower the threshold for understanding AI, cultivate rational cognition.

Expansion Possibilities: Multilingual label support, model interpretation visualization (attention heatmaps), multiplayer battles, community theme pack market; Open-Source Information: The project is open-sourced under the MIT license, released by Melidi Georgii in 2026, and encourages community participation—educators can use it for teaching, researchers can build experimental platforms, and developers can contribute new features/theme packs.

Guess What AI Thinks carries a serious proposition in the form of a game: understanding AI's way of thinking is crucial for human-AI collaboration. AI's cognitive patterns (statistics, probability, pattern matching) are completely different from humans' (causality, common sense), but they have their own logic. The value of the project lies in revealing AI's capabilities and limitations, cultivating a rational view of AI, being a beneficial attempt at AI literacy education, and providing a cognitive foundation for building a future of human-AI collaboration.