Reasoning and Misalignment: A Comparative Empirical Study of Three Open-Source Language Models

A master's thesis study that systematically compares the performance of three open-source large language models on reasoning tasks, revealing the potential tension between model capabilities and their alignment training.

Original author/maintainer: haavardos Source platform: GitHub Original title: master-thesis-ikt590-reasoning-misalignment Original link: https://github.com/haavardos/master-thesis-ikt590-reasoning-misalignment Source publication/update time: 2026-06-01T15:05:02Z

Keywords: large language models, alignment training, reasoning ability, RLHF, open-source models, AI safety, empirical study

Training of large language models usually involves two stages: pre-training (learning language and world knowledge) and alignment (using RLHF or similar techniques to make model behavior conform to human preferences). While alignment training significantly improves the usefulness and safety of models, an increasing number of studies show that excessive alignment constraints may impair the original reasoning ability of models.

This phenomenon of "alignment tax" has attracted academic attention: when we require models to be more "obedient" and "safe", do we inadvertently limit their ability to solve complex problems? This study is an empirical exploration of this question.

The study selected three open-source language models for comparative analysis (based on the research time and open-source ecosystem, the tested models are likely to include Llama 2, Mistral, or similar popular open-source models). The study uses a systematic comparative method to evaluate the performance of each model on the same set of reasoning tasks.

Selection of Reasoning Tasks

• Logical Reasoning: Conditional reasoning, syllogism, logical puzzles
• Mathematical Reasoning: Arithmetic, algebra, geometry problems
• Common Sense Reasoning: Tasks requiring world knowledge and causal inference
• Multi-step Reasoning: Complex problems requiring chain-of-thought

Evaluation Metrics

In addition to simple accuracy, the study also focuses on:

• Completeness of reasoning chain: Whether the model correctly shows intermediate reasoning steps
• Stability of answers: Whether the same question with different expressions gets consistent answers
• Rejection rate: Whether the model overly conservatively refuses to answer certain questions

The study found a negative correlation between the alignment degree of models and their reasoning performance, which is specifically manifested as follows:

Cost of Over-caution

Models that have undergone extensive safety alignment may be overly cautious in reasoning tasks. For example, when facing hypothetical scenarios (such as "If all cats could fly..."), they refuse logical deduction due to focusing on real-world constraints, and the safety-first tendency becomes an obstacle.

Trade-off Between Instruction Following and Reasoning Depth

Alignment training strengthens instruction-following ability, but may lead models to give superficial answers instead of in-depth analysis, and prematurely converge to seemingly reasonable answers in multi-step reasoning tasks.

Pattern of Differences Between Models

• Base Model: Pre-trained version without alignment, with the strongest reasoning performance but less "useful" output
• Standard Aligned Model: Typical RLHF-processed model, balancing safety and capability
• Over-aligned Model: Version with additional safety reinforcement, showing a significant decline in performance on certain reasoning tasks

The methodological design of this study is worth noting:

Importance of Controlled Variables

When comparing different models, confounding factors such as scale and training data volume are controlled, and open-source models of similar scale are selected, making alignment strategy the main source of difference.

Targeted Task Design

Questions that require both reasoning ability and may trigger safety filters are specifically designed to highlight when the alignment mechanism will "overreact".

Combination of Manual and Automatic Evaluation

A mixed method is adopted: automated metrics for large-scale screening, and manual evaluation for in-depth analysis of typical cases to judge the quality of reasoning chains.

Refinement of Alignment Strategy

Future alignment training needs:

• Domain-aware safety strategy: Relax certain safety constraints in mathematical reasoning tasks
• Explicit protection of reasoning ability: Add reasoning task performance monitoring during alignment
• Dynamic alignment: Adjust model behavior patterns according to task types

Value of Open-Source Models

Open-source models allow access to weights and training details, facilitating in-depth analysis of the impact of alignment mechanisms, while the black-box nature of closed-source models hinders such research.

Evolution of Evaluation Benchmarks

Traditional evaluation focuses on the correctness of the final answer; it needs to pay more attention to the reasoning process, distinguishing between "correct answer but chaotic reasoning" and "rigorous reasoning but wrong answer" cases.

Research Limitations

• Limited number of models: Only three models are not enough to draw universal conclusions
• Task coverage: The types of tasks are not comprehensive enough
• Timeliness: The open-source model ecosystem is developing rapidly, and the results need to be reproduced on new models

Future Directions

Expand the sample size, include more models (such as Llama 3, Mixtral, etc.), and explore specific technical solutions to mitigate the "alignment tax".