miLLM: A Self-Hosted Large Language Model Inference Server Integrating Sparse Autoencoders and Feature Steering

miLLM is a powerful self-hosted large language model (LLM) inference server that innovatively integrates Sparse Autoencoder (SAE) technology. It enables real-time monitoring of the model's internal activation states and precise manipulation at the feature level, providing a new tool for research on LLM interpretability and controllability.

With the widespread application of large language models (LLMs) across various fields, understanding the internal working mechanisms of models and improving their controllability and interpretability have become important topics in artificial intelligence research. Traditional LLM inference services mainly focus on throughput and latency optimization, but pay less attention to the transparency and manipulability of the model's internal states.

Sparse Autoencoder (SAE) technology has shown great potential in neural network interpretability research in recent years. Through SAE, researchers can decompose model activations into interpretable features, thereby understanding what the model "is thinking" during inference. However, integrating SAE technology into production-level inference services and achieving real-time feature manipulation remains a technical challenge.

The miLLM (monitored inference LLM) project, developed by the hitsainet team, is an open-source self-hosted LLM inference server. The main innovation of this project lies in the deep integration of SAE technology into the inference architecture, providing a complete toolchain from activation monitoring to feature manipulation.

Compared to traditional inference servers (such as vLLM, TensorRT-LLM, etc.), miLLM's unique value lies in its "observability-first" design philosophy. It not only provides efficient model inference capabilities but also allows users to gain insight into the model's internal decision-making process and perform fine-grained regulation at the feature level.

Sparse Autoencoder Integration

One of miLLM's core technologies is the seamless integration of Sparse Autoencoders (SAE). SAE is a neural network architecture that learns to map high-dimensional activation vectors to low-dimensional sparse representations, enabling interpretable decomposition of the model's internal states.

In miLLM, SAE integration is reflected in the following aspects:

1. Activation Capture Layer: Insert activation capture hooks at key layers of the model (such as attention layers and feedforward network layers) to extract intermediate activation states in real time.

2. Online Encoding: Encode the captured activations into sparse feature representations in real time; these features usually correspond to interpretable concepts (such as "numbers", "negation words", "person names", etc.).

3. Feature Dictionary: Maintain a learnable feature dictionary that maps sparse feature indices to human-understandable semantic descriptions.

This design allows users to observe which features are activated and how these features affect the final output while the model is inferring.

Activation Monitoring and Visualization

miLLM provides rich activation monitoring functions to help users understand the model's internal states:

• Real-time Activation Heatmap: Visually display the activation intensity distribution of different layers and attention heads.

• Feature Activation Tracking: Track the activation trajectory of specific features during the generation process to understand how the model gradually constructs the output.

• Anomaly Detection: Automatically identify abnormal activation patterns to help detect potential model biases or erroneous behaviors.

These monitoring functions not only serve researchers but also provide important tools for model operation and maintenance in production environments.

Feature Steering and Guidance

miLLM's most innovative function is its Feature Steering capability. Users can enhance or suppress specific features during inference to guide the model's behavior:

• Feature Enhancement: Increase the activation intensity of features related to the desired output, making the model more inclined to generate specific types of content.

• Feature Suppression: Reduce the activation of features related to undesirable outputs, used for content security control or bias mitigation.

• Multi-feature Combination: Support simultaneous manipulation of multiple features to implement complex generation control strategies.

This fine-grained control capability provides new possibilities for building safer and more controllable AI applications.

LLM Interpretability Research

For AI researchers, miLLM provides a powerful experimental platform. Researchers can verify hypotheses about the model's internal working mechanisms and discover new interpretability rules by observing feature activation patterns.

For example, by analyzing differences in feature activation across different tasks (such as question answering, summarization, translation), researchers can better understand the multi-task learning mechanism of large models.

Content Safety and Alignment Optimization

Content safety is a key consideration when deploying large models. miLLM's feature steering function can be used for:

• Identifying and suppressing features related to harmful content generation
• Enhancing features related to beneficial and safe outputs
• Real-time monitoring of risk feature activation during the generation process

This method provides a more proactive and precise safety control approach compared to traditional output filtering.

Model Debugging and Error Analysis

When the model produces incorrect outputs, miLLM's activation monitoring function can help quickly locate the root cause. By analyzing the activation patterns corresponding to incorrect outputs, developers can identify which features caused the error and adjust the model or training data accordingly.

Personalized Generation Control

For application scenarios requiring personalized outputs (such as creative writing, style transfer), miLLM allows users to achieve fine-grained generation control by manipulating specific style features without retraining the model or adjusting complex prompts.

Efficient SAE Inference Optimization

Online encoding of SAE requires additional computational overhead. miLLM ensures inference efficiency through the following optimization strategies:

• Sparse Computing Acceleration: Utilize the sparsity of features and adopt sparse matrix operations to accelerate the encoding process.

• Inter-layer Parallelism: Perform SAE encoding while capturing activations to maximize hardware utilization.

• Optional Modes: Provide "monitoring mode" and "standard mode", allowing users to choose whether to enable SAE functions based on their needs.

Modular Architecture Design

miLLM adopts a highly modular architecture, with core components including:

• Inference Engine: Based on an efficient inference framework, supporting multiple model architectures.
• SAE Module: A pluggable sparse autoencoder component that supports custom feature dictionaries.
• Monitoring Service: An independent monitoring data stream that does not affect the performance of the main inference path.
• Control Interfaces: RESTful API and WebSocket interfaces for easy integration into various applications.

Open-Source Ecosystem Compatibility

miLLM was designed with full consideration of compatibility with the existing open-source ecosystem:

• Supports Hugging Face model format
• Compatible with OpenAI API interface specifications
• Provides Docker deployment solutions
• Supports integration with frameworks such as LangChain and LlamaIndex

Although miLLM provides powerful functions, there are still some notable limitations:

• Computational Overhead: SAE encoding and activation monitoring bring additional computational costs, which may require trade-offs in latency-sensitive scenarios.

• Feature Interpretability: Although the features extracted by SAE are sparse, not all features have intuitive human interpretability, and the construction of feature dictionaries still requires manual participation.

• Model Support Range: Currently, it mainly supports decoder-only models of the Transformer architecture; support for other architectures needs to be expanded.

Future development directions may include:

• More efficient SAE algorithms to reduce monitoring overhead
• Automated feature semantic annotation
• Support for monitoring and manipulation of multi-modal large models
• Integration with reinforcement learning to implement adaptive feature steering strategies

The miLLM project represents an important direction in the evolution of LLM inference services—from "black-box inference" to "white-box inference". By integrating SAE technology into the inference architecture, miLLM provides a practical tool platform for research on LLM interpretability and controllability.

For AI practitioners, the value of miLLM lies not only in its technical implementation but also in the possibilities it demonstrates: future LLM services can not only provide high-quality outputs but also allow users to understand why such outputs are generated and have the ability to finely control the model's behavior. This transparency and controllability will be the key foundation for building trustworthy AI systems.