Hybrid Rule and AI Log Noise Reduction System: LLM-Noise-Filtering-System

LLM-Noise-Filtering-System is an intelligent log filtering system that combines rule engines and large language models. It uses a hybrid architecture to efficiently identify and eliminate noise data, and has practical value in cybersecurity and log analysis scenarios. Project goals include significantly reducing the proportion of log noise, accurately identifying security-critical information, building a scalable AI pipeline, and verifying performance through manual annotation. Project address: https://github.com/mUchiha26/LLM-Noise-Filtering-System

Logs are core data sources for system monitoring, security auditing, and troubleshooting. However, the expansion of scale leads to a surge in noise data, which wastes resources and obscures key events. Traditional rule-based methods are efficient but struggle to handle complex formats and new types of attacks; pure LLM methods have strong understanding capabilities but are high-cost and high-latency. Balancing efficiency and intelligence is a key industry challenge.

The system adopts a layered processing architecture: Raw Logs → Rule Filter → LLM Classifier → Scoring System → Clean Output. Core components include: 1. Rule Filter (uses regex to pre-filter known noise); 2. Text Splitter (splits logs while preserving context); 3. LLM Classifier (supports API/local modes); 4. Scoring System (outputs confidence based on comprehensive results).

Application example: When inputting logs containing DEBUG, successful login, and SQL injection entries, the system filters DEBUG noise and retains key entries. Performance evaluation uses manually annotated datasets (marking relevant/noise entries), and calculates accuracy by comparing model predictions with annotated results to ensure system reliability.

The project's value lies in its hybrid intelligence design (balancing cost and effectiveness with rules + LLM), evaluability (manual annotation ensures credibility), modular architecture (easy to expand and customize), providing practical references for AI pipeline construction, and demonstrating the controllability and interpretability of integrating LLMs into traditional processes.

Future plans include: 1. APIization (building real-time processing interfaces with FastAPI); 2. Scoring optimization (fine-grained confidence calibration); 3. Cost optimization (reducing LLM token consumption); 4. Security integration (deep integration with SIEM systems).

The tech stack is based on Python, relying on regex libraries, LLM API clients, etc. Configuration supports API mode (set environment variable LLM_MODE=api and secret key) and local mode (LLM_MODE=local). The running command is python main.py data/sample.txt, and the code structure is clear and modular.