CodeTalkers: Revealing the Hidden Costs of Instruction Tuning for Large Language Models in Code Tasks

This study proposes the concept of "Instruction Tuning Tax", pointing out that while instruction tuning improves the instruction-following ability of code large language models (such as Qwen2.5-Coder, DeepSeek-Coder), it may impair their core programming capabilities like code completion and infilling. The study quantifies this hidden cost through comparative experiments and explores mitigation strategies, which has important guiding significance for model selection and application.

Code LLMs (e.g., Qwen2.5-Coder, DeepSeek-Coder) exhibit strong programming capabilities through pre-training, but are usually instruction-tuned to enhance human-computer interaction. The industry once assumed that instruction tuning was a "free improvement", but this study questions: Does instruction tuning have hidden costs? That is, when models follow natural language instructions, do they sacrifice pure code completion ability? This question is crucial for code assistance functions relied on by developers (such as auto-completion, intelligent prompts).

The study defines "Instruction Tuning Tax" as the loss of core programming capabilities of code models during instruction tuning. It also distinguishes two types of tasks:

• Instruction-driven tasks: Generate code based on natural language instructions (e.g., HumanEval, MBPP benchmarks);
• Code completion tasks: Predict subsequent content based on code context (e.g., HumanEval-Infilling, ClassEval benchmarks). Hypothesis: Instruction tuning optimizes instruction understanding but weakens sensitivity to pure code context.

The experiment compares Base (pre-trained) and Instruct (instruction-tuned) variants of mainstream code LLMs:

• Models: Qwen2.5-Coder (1.5B/7B/14B/32B), DeepSeek-Coder (1.3B/6.7B/33B);
• Benchmarks: Code completion category (HumanEval-Infilling, ClassEval, etc.), instruction-following category (HumanEval+, MBPP+, etc.);
• Research questions: RQ1 (Impact of instruction tuning on completion ability), RQ2 (Changes in behavior patterns), RQ3 (Exploration of mitigation strategies).

The experiment confirms the existence of Instruction Tuning Tax:

1. Instruct models are systematically lower than Base models on completion benchmarks, especially with significant gaps in middle infilling and fine-grained completion tasks;
2. Task type affects the tax magnitude: Tasks relying on code structure intuition (like code infilling) have higher taxes, while tasks aligned with instruction goals (like complete function generation) have lower taxes;
3. Scale effect: Smaller models have a higher tax ratio, while larger models still have relative taxes;
4. Behavior changes: Instruct models increase attention to natural language tokens, reduce sensitivity to code structure, and tend to generate comments, leading to inaccurate completions.

Mitigation attempt: Fine-tuning Qwen2.5-Coder-7B using the Magicoder process partially restores completion performance while maintaining instruction capabilities. Practical suggestions:

• Model selection: Prioritize Base models for IDE completion, choose Instruct models for chat-style assistants;
• Product development: Code editor plugins need to evaluate whether to switch back to Base models, and AI assistants need to balance completion and instruction capabilities;
• Future research: Explore unified models with dynamic switching modes, training methods that balance both task types, model fusion technologies, etc.

The project provides complete code to support reproduction:

• Environment setup: git clone https://github.com/arkosioscambions/CodeTalkers.git && cd CodeTalkers && pip install -r requirements.txt;
• Code generation: python generate.py --model <qwen|dscoder> --model_id <model_id> --dataset <dataset_name>;
• Evaluation: Use different scripts for corresponding benchmarks (e.g., evaluate_classeval_completion.py for ClassEval-Completion);
• Behavior analysis: python3 generate_rq2_table7.py generates the metric report.