Neuro-RIT: Neuron-level Precise Regulation Makes RAG Systems More Robust and Effectively Suppresses Noise Interference

Neuro-RIT addresses the core issue of Retrieval-Augmented Generation (RAG) systems being highly sensitive to retrieval quality and vulnerable to noise interference. Based on the neuron sparsity property of LLMs, it identifies neurons that distinguish relevant/irrelevant contexts through attribution analysis, adopts a strategy of functionally shutting down noise-related neurons plus two-stage instruction fine-tuning, and consistently outperforms baseline methods on multiple knowledge-intensive QA benchmarks, achieving a significant improvement in RAG system robustness.

The RAG architecture relies on external retrieval to reduce hallucinations, but retrieval noise significantly degrades model performance, and LLMs struggle to effectively distinguish between relevant and irrelevant information. Existing robustness improvement methods mostly update parameters at the layer or module level (e.g., adding special tokens, designing attention mechanisms, adversarial training), which are coarse-grained, inefficient, and tend to interfere with the model's ability to perform other tasks.

LLM feedforward networks exhibit neuron-level sparsity—only a small number of neurons are activated during each forward pass. This implies that different functions are handled by different subsets of neurons, specific tasks depend on a small number of parameters, and precise regulation is more effective than global updates. Neuro-RIT is based on this insight and focuses on neuron-level precise intervention.

1. Attribution Mining

Using attribution analysis methods such as integrated gradients, calculate the contribution of neurons to relevant/irrelevant documents, and identify relevant neurons (positive contribution) and irrelevant neurons (activated by noise and negative contribution).

2. Functional Suppression

During training, force the output of irrelevant neurons to zero or suppress them, directly shutting down noise-related neural pathways.

3. Two-Stage Fine-Tuning

• Stage 1: Train with noisy samples and apply functional suppression;
• Stage 2: Optimize with clean samples to enhance evidence extraction ability.

Tested on multiple QA benchmarks including Natural Questions, TriviaQA, and HotpotQA, comparing with standard RAG, adversarial training, and other methods:

• Accuracy improved by 5-15 percentage points;
• Performance degradation under noise is slower;
• Better generalization to unseen noise types. Ablation experiments show that removing neuron mining, functional shutdown, or single-stage training leads to significant performance drops.

• Attribution method: Choose an approximate version of integrated gradients to balance accuracy and efficiency;
• Functional shutdown: Can use hard masking (set to zero) or soft masking (multiply by a small coefficient);
• Computational optimization: Control additional overhead through caching attribution results and batch processing.

• Paradigm shift: From dense parameter updates to sparse neuron precise regulation;
• Interpretability: Identify noise-related neurons to facilitate debugging and improvement;
• Modular design: Can build noise filtering and evidence extraction modules corresponding to specific neuron sets.

• Attribution accuracy: Need more precise causal inference methods;
• Cross-task transfer: Explore cross-task generality of neuron patterns;
• Dynamic adaptability: Design mechanisms to adjust neuron activation in real time;
• Technology integration: Integrate with better retrievers, re-ranking models, etc., for optimization.