Chartographer: A Counterfactual Evaluation Framework for Visual Language Models' Chart Reasoning Capabilities

Chartographer is an open-source chart counterfactual generation framework from the Computational Linguistics Lab at the University of Waterloo, designed to systematically evaluate whether visual language models (VLMs) truly possess chart reasoning capabilities. Its core idea is to construct original charts, basic reconstructions, and seed-controlled counterfactual variants, using these variants to test whether models rely on visual shortcuts rather than genuine reasoning. Additionally, the framework uses executable Python code to verify answers, ensuring objective and reproducible evaluations. The project is open-source (GitHub link: https://github.com/compling-wat/Chartographer), with supporting resources including an arXiv paper (https://arxiv.org/abs/2605.27311) and a Hugging Face dataset (https://huggingface.co/datasets/1fanjz/Chartographer).

Existing evaluations of VLMs' chart understanding have fundamental issues: models may answer questions using visual shortcuts or prior knowledge instead of truly understanding the chart content. For example, when answering 'What is the highest value in the chart?', a model might rely on memorized common patterns rather than analyzing visual elements. This 'pseudo-understanding' is hard to detect in standard benchmarks because test samples are distributed similarly to training data. The Chartographer project was created to address this challenge by generating counterfactual chart variants to build a more rigorous testing framework.

Chartographer's core methodology is to transform chart QA samples into counterfactual chart-question families, which include three key components:

1. Original Chart: Keep the data unchanged, serving as the starting point for benchmark testing.
2. Basic Reconstruction: Generate charts from scratch using a reconstruction model to test the model's ability to understand chart structures.
3. Seed-Controlled Counterfactual Variants: Generate variants with different visual presentations but the same data by controlling random seeds (e.g., changes in color, layout, chart type). Key insight: Models relying on shortcuts will have fluctuating performance, while models with genuine reasoning will have stable performance.

Executable QA Logic

The framework uses executable Python code to verify answer correctness, eliminating subjectivity:

• Extract precise values from chart data
• Perform mathematical calculations (sum, average, etc.)
• Verify the accuracy of model answers

System Workflow

1. Chart Reconstruction Phase: Initiated by make reconstruction-workflow, including analyzing original charts, generating descriptions and data extraction, reconstructing visualizations, and multi-round self-optimization (controlled by REVISION_ROUNDS).
2. QA Generation Phase: make qa-workflow generates QA pairs with verification code.
3. Counterfactual Generation Phase: make seed-workflow generates seed-controlled variants.
4. Dataset Export and Evaluation: make export-family-dataset packages the dataset; make prediction-workflow runs VLM predictions and evaluates reasoning stability.

Supported Models and APIs

Chartographer supports multiple VLMs:

• OpenAI API (GPT-4V series)
• Anthropic API (Claude3 visual version)
• Local Hugging Face models (weight path needs to be specified)

Dataset Configuration

Defined using JSON configuration files: local template path, question/image/answer column mappings, variant columns, and family ID columns, adapted to existing chart QA datasets.

Code Structure

Layered architecture:

• src/clients: API and local VLM clients
• src/common: Dataset, answer, and prediction I/O tools
• src/config: Model aliases and task prompts
• src/pipeline: Modules for reconstruction, QA, dataset export, prediction, etc.

Use Cases

1. VLM Researchers: A rigorous evaluation tool to identify gaps between real capabilities and surface performance, guiding model improvements.
2. VLM Developers: A regression testing tool to ensure that reasoning capabilities do not degrade after model updates.
3. Data Visualization Field: Counterfactual chart families can be used to explore differences in sensitivity to visual elements between humans and machines.

Project Status

Chartographer is open-source (Apache 2.0 license), providing complete documentation, example configurations, and Makefile workflows. Access methods:

• GitHub: https://github.com/compling-wat/Chartographer
• arXiv paper: https://arxiv.org/abs/2605.27311
• Hugging Face dataset: https://huggingface.co/datasets/1fanjz/Chartographer

Chartographer represents an important advancement in AI evaluation methodology. In today's era of rapidly improving VLM capabilities, distinguishing between 'genuine understanding' and 'pattern matching' is crucial. Through counterfactual generation and executable verification, the framework provides a systematic solution to this challenge.

Outlook: As multimodal large models become widespread, such rigorous evaluation frameworks will become key infrastructure to ensure the reliability of AI systems. The open-source nature of Chartographer is expected to promote the development of more robust visual language models.