Knowledge Distillation Energy Efficiency Evaluation Framework: Slimming Large Models While Saving Power

This article introduces the open-source project Slimming-Models-Saving-Watts, a knowledge distillation research framework for HPC cluster environments. It supports three mainstream distillation paradigms and integrates GPU/CPU energy consumption telemetry, providing a quantitative evaluation tool for energy efficiency optimization of large language models. Its goal is to resolve the conflict between model scale and computational resource consumption.

Large language models consume enormous energy during training and inference. As a model compression technique, knowledge distillation theoretically enables slimming and efficiency improvement, but traditional KD research only focuses on accuracy metrics and ignores systematic evaluation of energy consumption. This project addresses this gap by building a complete framework for HPC environments, integrating energy efficiency evaluation into the core of the KD process.

The project modularly implements three mainstream distillation paradigms:

1. Responsive Distillation: Fits the output probability distribution of the teacher model. It is simple but may lose intermediate layer information.
2. Feature Distillation: Forces the intermediate layer representations of the student model to align with those of the teacher, transferring deep semantics but requiring inter-layer mapping design.
3. Relational Distillation: Preserves relative distances between samples to transfer knowledge, suitable for tasks that need to retain data structure characteristics. Researchers can flexibly combine or use them individually.

The framework has a built-in energy consumption telemetry system (monitor.py) that real-time collects GPU power consumption/utilization, memory/temperature, CPU usage, and timestamps, with data recorded in JSONL format. It can calculate metrics such as E_run (total energy consumption), EPT (energy per token), OM_perf (performance retention rate), and Eff_overall (comprehensive efficiency) to quantitatively answer the power-saving effect of distillation methods.

The project is optimized for Slurm scheduling systems and NVIDIA GPUs (H100/A100/RTX series). Data preprocessing uses a sharding strategy to improve I/O performance and deterministic sampling. It integrates the lm-evaluation-harness and lighteval evaluation systems, covering mainstream tasks like MMLU. Evaluation results are visualized via Jupyter Notebook (energy consumption curves, accuracy-energy efficiency trade-off graphs) to assist in generating research reports.

The framework applies to multiple scenarios:

• Cloud service providers: Find the optimal balance between accuracy and energy consumption under hardware configurations to provide cost-effective model services.
• AI research teams: Compare the energy efficiency performance of different distillation strategies to support method selection.
• Environmental organizations: Quantify the carbon emission reduction effect of model compression to meet ESG report requirements.

Slimming-Models-Saving-Watts advances KD research to a new stage where both accuracy and energy efficiency are emphasized, and has important practical value against the backdrop of expanding AI computing power demand. The project is open-source and supports mainstream model families like Llama 3.1 and Qwen2.5, providing a solid model optimization infrastructure for academia and industry.