OPSD: A Large Language Model Inference Optimization Tool Based on In-Strategy Self-Distillation

OPSD is a local large language model inference optimization tool for Windows platforms. Its core uses a "student-teacher" dual-role architecture to implement in-strategy self-distillation, and improves the token-level output quality of models in tasks such as logical reasoning and mathematical computation through contrastive learning. This tool does not rely on external labeled data, realizes a closed loop of inference and learning, and allows the model to continuously evolve during use.

Large language models often struggle to generate high-quality intermediate thinking processes in complex reasoning tasks, and traditional supervised fine-tuning has limitations. Self-distillation technology has gradually gained attention because it allows models to learn from their own outputs without external data. The OPSD project was born in this context, proposing an innovative training paradigm where the same model plays both student and teacher roles.

• In-strategy self-distillation: Breaks the traditional dual-model architecture. The same model generates outputs from two perspectives—student (only sees the problem) and teacher (sees the problem + reference answer)—and uses contrastive learning to guide optimization, enabling immediate feedback during inference.
• Token-level optimization: Refines the optimization granularity to each generation position, avoiding subsequent deviations caused by errors in intermediate steps, and allowing each token decision to receive fine-grained gradient feedback.

OPSD is a Windows desktop application, and its key architecture includes:

1. Dual input channels: The student channel receives the original problem, while the teacher channel appends reference answers/thinking processes;
2. Inference-learning closed loop: Generate initial answers through inference → evaluate the difference between student and teacher outputs → encode the difference into gradient fine-tuning parameters → update the model for the next round of inference, realizing continuous evolution.

Applicable tasks: Logical reasoning (puzzles, causal analysis), mathematical problem-solving (showing steps), answer quality evaluation, output tracking and review. Operation interface: Prompt input box, model selector, run button, output panel, setting area; parameters such as model path, batch size, context length, and log level can be configured.

Advantages of local operation: Data privacy (local processing), offline availability, cost control (no API fees), low latency (local GPU inference). Hardware requirements: Windows10/11, 8GB memory (larger is recommended), 10GB disk space; optimization suggestions: reduce batch size, close memory-intensive applications, clean up disk space.

Current limitations: Only supports Windows, limited model compatibility, steep learning curve for non-technical users, lack of standardized benchmark tests. Future directions: Multimodal support, distributed training, cloud synchronization, community model market.

OPSD encapsulates self-distillation technology into an easy-to-use desktop tool, allowing more people to access cutting-edge methods. It reflects the trend of model optimization shifting from large-scale pre-training to refined post-training. Although it has limitations, the core concept of "models learning from their own outputs" has broad application prospects.