CoTLab: A Research Toolkit for Chain-of-Thought Reasoning and Interpretability of Large Language Models

CoTLab is an open-source toolkit dedicated to researching chain-of-thought (CoT) reasoning, faithfulness, and mechanistic interpretability of large language models (LLMs). It supports multiple experiment types and reasoning backends, providing a systematic research framework for understanding the internal working mechanisms of LLMs. Its core goal is to help researchers deeply explore the faithfulness of CoT and its relationship with the model's internal representations, addressing the key question of whether CoT truly reflects the model's internal computation process.

The chain-of-thought reasoning ability demonstrated by large language models has significantly improved the accuracy of tasks such as mathematical problem-solving and logical reasoning. However, fundamental questions remain: Do these chain-of-thought processes truly reflect the model's internal computation? Is the model 'truly thinking' or just generating a reasonable format? This question touches the core of AI interpretability—if CoT is disconnected from the actual decision-making mechanism, audits, alignment, and safety assessments based on CoT will lose their foundation. CoTLab was created to address this challenge.

CoTLab is built on the Hydra configuration system, modular and configurable, supporting flexible experiment combinations and batch runs. Its core functions cover three main directions:

1. Chain-of-Thought Faithfulness Evaluation: Quantify the contribution of reasoning steps to the final answer through CoT ablation, faithfulness testing, etc.;
2. Mechanistic Interpretability Analysis: Track neural circuits corresponding to reasoning using activation patching, logit lens, attention head probing, etc.;
3. Prompt Strategy Comparison: Evaluate the impact of strategies like direct answer, CoT, and adversarial prompts.

Dual-backend design:

• vLLM Backend: High performance, suitable for large-scale generation experiments (e.g., CoT faithfulness testing), but no access to internal activations;
• Transformers Backend: Supports access to model internal states, used for mechanistic interpretability experiments (e.g., activation patching), but slower. Users can switch backends via the command line.

CoTLab supports multiple experiment scenarios, with the following typical examples:

• Logit Lens Analysis: Decode hidden states layer by layer to observe when the model locks onto the correct answer. Command: python -m cotlab.main experiment=logit_lens model=medgemma_4b;
• Sycophancy Head Detection: Identify attention heads sensitive to sycophantic bias. Command: python -m cotlab.main experiment=sycophancy_heads model=medgemma_4b;
• Chain-of-Thought Ablation: Remove/modify CoT steps to observe the impact on answers. Command: python -m cotlab.main experiment=cot_ablation dataset=pediatrics;
• Multi-Prompt Strategy Comparison: Batch run different prompt formats. Command: python -m cotlab.main -m prompt=chain_of_thought,direct_answer,sycophantic.

CoTLab collaborates with the DRIVE Digital Innovation Unit of Great Ormond Street Hospital (GOSH) in the UK to conduct medical AI research. It is optimized for medical large models like MedGemma, with built-in configurations for MedGemma 2B/4B/27B models and designed clinical task workflows such as radiology report generation. The toolkit supports direct loading of compatible models via Hugging Face model IDs, automatically inferring the number of layers and attention heads; for models with special architectures, users can fine-tune parameters via configuration files.

CoTLab uses a Python 3.11+ environment and uv as the package management tool, supporting hardware such as NVIDIA GPU (vLLM), AMD ROCm (Docker/ROCm PyTorch), and Apple Silicon (vLLM-Metal plugin). The configuration follows the Hydra hierarchical structure, allowing parameter overriding via the command line for easy hyperparameter scanning. The code organization separates experiment configuration, model definition, dataset processing, and result output to ensure reproducibility. Documentation is hosted on GitHub Pages, including installation guides, API references, and tutorials.

CoTLab fills the tool gap in LLM interpretability research, providing a complete chain from prompt engineering to internal representation analysis, helping to systematically test the CoT faithfulness hypothesis. Its application value includes: an 'honest' reasoning evaluation method for AI safety research, a tool for model developers to fix reasoning defects, and promoting the transformation of AI from 'black box' to interpretable. Future directions: expand multi-modal CoT support, integrate causal inference methods, develop automated faithfulness metrics, and establish cross-model standardized benchmarks.