# Reasoning and Misalignment: A Comparative Empirical Study of Three Open-Source Language Models > A master's thesis study that systematically compares the performance of three open-source large language models on reasoning tasks, revealing the potential tension between model capabilities and their alignment training. - 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm) - 发布时间: 2026-06-01T15:05:02.000Z - 最近活动: 2026-06-01T15:27:21.450Z - 热度: 148.6 - 关键词: 大语言模型, 对齐训练, 推理能力, RLHF, 开源模型, AI安全, 实证研究 - 页面链接: https://www.zingnex.cn/en/forum/thread/llm-github-haavardos-master-thesis-ikt590-reasoning-misalignment - Canonical: https://www.zingnex.cn/forum/thread/llm-github-haavardos-master-thesis-ikt590-reasoning-misalignment - Markdown 来源: floors_fallback --- ## [Introduction] Reasoning and Misalignment: A Comparative Empirical Study of Three Open-Source Language Models A master's thesis study that systematically compares the performance of three open-source large language models on reasoning tasks, revealing the potential tension between model capabilities and their alignment training. Original author/maintainer: haavardos Source platform: GitHub Original title: master-thesis-ikt590-reasoning-misalignment Original link: https://github.com/haavardos/master-thesis-ikt590-reasoning-misalignment Source publication/update time: 2026-06-01T15:05:02Z Keywords: large language models, alignment training, reasoning ability, RLHF, open-source models, AI safety, empirical study ## Research Background: Tension Between Alignment and Capability Training of large language models usually involves two stages: pre-training (learning language and world knowledge) and alignment (using RLHF or similar techniques to make model behavior conform to human preferences). While alignment training significantly improves the usefulness and safety of models, an increasing number of studies show that excessive alignment constraints may impair the original reasoning ability of models. This phenomenon of "alignment tax" has attracted academic attention: when we require models to be more "obedient" and "safe", do we inadvertently limit their ability to solve complex problems? This study is an empirical exploration of this question. ## Research Design: Comparative Methods for Three Open-Source Models The study selected three open-source language models for comparative analysis (based on the research time and open-source ecosystem, the tested models are likely to include Llama 2, Mistral, or similar popular open-source models). The study uses a systematic comparative method to evaluate the performance of each model on the same set of reasoning tasks. ### Selection of Reasoning Tasks - **Logical Reasoning**: Conditional reasoning, syllogism, logical puzzles - **Mathematical Reasoning**: Arithmetic, algebra, geometry problems - **Common Sense Reasoning**: Tasks requiring world knowledge and causal inference - **Multi-step Reasoning**: Complex problems requiring chain-of-thought ### Evaluation Metrics In addition to simple accuracy, the study also focuses on: - **Completeness of reasoning chain**: Whether the model correctly shows intermediate reasoning steps - **Stability of answers**: Whether the same question with different expressions gets consistent answers - **Rejection rate**: Whether the model overly conservatively refuses to answer certain questions ## Key Findings: Negative Correlation Between Reasoning Ability and Alignment Degree The study found a negative correlation between the alignment degree of models and their reasoning performance, which is specifically manifested as follows: ### Cost of Over-caution Models that have undergone extensive safety alignment may be overly cautious in reasoning tasks. For example, when facing hypothetical scenarios (such as "If all cats could fly..."), they refuse logical deduction due to focusing on real-world constraints, and the safety-first tendency becomes an obstacle. ### Trade-off Between Instruction Following and Reasoning Depth Alignment training strengthens instruction-following ability, but may lead models to give superficial answers instead of in-depth analysis, and prematurely converge to seemingly reasonable answers in multi-step reasoning tasks. ### Pattern of Differences Between Models - **Base Model**: Pre-trained version without alignment, with the strongest reasoning performance but less "useful" output - **Standard Aligned Model**: Typical RLHF-processed model, balancing safety and capability - **Over-aligned Model**: Version with additional safety reinforcement, showing a significant decline in performance on certain reasoning tasks ## Methodological Insights: Importance of Controlled Variables and Mixed Evaluation The methodological design of this study is worth noting: ### Importance of Controlled Variables When comparing different models, confounding factors such as scale and training data volume are controlled, and open-source models of similar scale are selected, making alignment strategy the main source of difference. ### Targeted Task Design Questions that require both reasoning ability and may trigger safety filters are specifically designed to highlight when the alignment mechanism will "overreact". ### Combination of Manual and Automatic Evaluation A mixed method is adopted: automated metrics for large-scale screening, and manual evaluation for in-depth analysis of typical cases to judge the quality of reasoning chains. ## Implications for Model Development: Refined Alignment and Value of Open Source ### Refinement of Alignment Strategy Future alignment training needs: - **Domain-aware safety strategy**: Relax certain safety constraints in mathematical reasoning tasks - **Explicit protection of reasoning ability**: Add reasoning task performance monitoring during alignment - **Dynamic alignment**: Adjust model behavior patterns according to task types ### Value of Open-Source Models Open-source models allow access to weights and training details, facilitating in-depth analysis of the impact of alignment mechanisms, while the black-box nature of closed-source models hinders such research. ### Evolution of Evaluation Benchmarks Traditional evaluation focuses on the correctness of the final answer; it needs to pay more attention to the reasoning process, distinguishing between "correct answer but chaotic reasoning" and "rigorous reasoning but wrong answer" cases. ## Limitations and Future Directions: Expanding Research Scope and Technical Exploration ### Research Limitations - **Limited number of models**: Only three models are not enough to draw universal conclusions - **Task coverage**: The types of tasks are not comprehensive enough - **Timeliness**: The open-source model ecosystem is developing rapidly, and the results need to be reproduced on new models ### Future Directions Expand the sample size, include more models (such as Llama 3, Mixtral, etc.), and explore specific technical solutions to mitigate the "alignment tax".