White-box Method Research on Hallucinations in Large Language Models: A Comprehensive Experimental Framework for Decoding Strategies, Retrieval Augmentation, and Parameter-Efficient Fine-Tuning

This article introduces an open-source white-box research framework that systematically controls decoding parameters, retrieval contexts, and PEFT fine-tuning techniques to deeply analyze the generation mechanisms and mitigation strategies of hallucination behaviors in large language models (LLMs). The framework aims to address the problem that traditional black-box research struggles to understand the internal mechanisms of hallucinations, providing support for the reliable application of LLMs in high-risk fields such as healthcare and law.

Large language models (LLMs) often generate "hallucinations"—information that seems plausible but is actually incorrect—seriously restricting their practical application in high-risk fields such as healthcare, law, and finance. Traditional hallucination research mostly treats models as black boxes, making it difficult to deeply understand the internal mechanisms of hallucination generation.

The sanskarmodi8/whitebox-hallucinations-llms project adopts a white-box research approach, establishing a reproducible experimental framework by systematically controlling hyperparameters in the training and inference stages to help researchers and developers understand the nature of hallucination behaviors.

The project constructs a research system from four key dimensions:

1. Decoding Strategy Control

Systematically study the impact of decoding parameters such as temperature, top-k sampling, top-p sampling, and repetition penalty on hallucination frequency and model confidence, observing reliability performance under different randomness and diversity settings.

2. Retrieval-Augmented Grounding

Evaluate the mitigation effect of Retrieval-Augmented Generation (RAG) technology on hallucinations, analyze the improvement of factual accuracy supported by external knowledge, and distinguish between "hallucinations caused by missing model knowledge" and "hallucination tendencies inherent in the model generation mechanism".

3. Parameter-Efficient Fine-Tuning (PEFT/LoRA)

Study the impact of parameter-efficient fine-tuning techniques like LoRA on hallucination behaviors, explore the possibility of improving model reliability through fine-tuning with limited computing resources, and analyze cases where fine-tuning reduces or introduces hallucinations.

4. Combined Intervention Strategies

Study the combined effects of the above technologies, analyze the synergistic or conflicting relationships between different intervention measures, and provide a basis for balancing reliability and computational cost in practical deployment.

The project adopts a modular experimental architecture to ensure reproducibility:

• configs/: Experimental configuration files
• datasets/: Dataset loading and preprocessing module
• src/generation/: Decoding strategy implementation
• src/finetuning/: PEFT/LoRA training code
• src/evaluation/: Hallucination detection and evaluation metrics
• src/pipeline/: Experimental workflow orchestration
• notebooks/: Exploratory analysis notebooks
• experiments/: Experimental log records
• results/: Result tables and visualizations

This architecture complies with the reproducibility principles of scientific research.

The project focuses on the following core research questions:

1. How do decoding parameters in the inference stage affect hallucination frequency and model confidence?
2. Under what circumstances can fine-tuning reduce hallucinations, and when might it be ineffective?
3. Which hallucinations originate from the model itself, and which from missing contextual information?
4. What is the trade-off between reliability and computational cost for different mitigation strategies?

Expected outputs include: hallucination behavior analysis reports, comparative evaluations of mitigation strategies, practical reliability guidelines for LLM deployment, and a reproducible research framework.

Currently, the project is in the initialization phase, designing the evaluation process, determining dataset selection, implementing the baseline generation and scoring system, and experimental results and analyses will be added gradually.

The project is open-source under the MIT License, developed by Sanskar Modi, Aryan Dhanuka, and Priyanshu Kumar Singh under the guidance of Ashwani Kumar. Researchers and engineers interested in LLM reliability, hallucination detection, and mitigation are welcome to participate.