Research on Sensitivity Analysis of Evaluation Awareness in Large Language Models

Undergraduate thesis project in Data Science at the University of Buenos Aires, focusing on sensitivity analysis of Large Language Models (LLMs) regarding evaluation awareness. Complete code, datasets, and experimental framework are provided, and the project is open-sourced on GitHub.

With the widespread application of LLMs in natural language processing, traditional evaluation methods mostly focus on performance against standard benchmarks, but pay less attention to models' awareness of the evaluation process. This project aims to reveal the behavioral differences of LLMs when facing different evaluation strategies through sensitivity analysis, providing new perspectives for model selection and optimization.

Code Implementation

Provides a modular codebase for sensitivity analysis algorithms, facilitating experiment reproduction and extension.

Datasets

Includes carefully designed datasets that capture subtle differences in models' evaluation awareness.

Experimental Framework

Establishes a standardized experimental process, forming a closed loop from data preparation and training to result analysis.

Impact of Evaluation Strategies

Explore whether models adjust their output strategies upon realizing they are being evaluated.

Differences in Model Architectures

Analyze systematic differences in evaluation awareness among different architectures (e.g., Transformer, RNN).

Role of Training Data

Study the impact of evaluation-related information in training data on models' awareness capabilities.

Data Preprocessing

Includes steps such as text standardization, label encoding, and dataset splitting.

Model Configuration

Unified interface to manage multiple mainstream LLM architectures, facilitating comparative experiments.

Evaluation Metrics

Combines traditional metrics (accuracy, recall) with metrics dedicated to evaluation awareness.

Theoretical Contributions

Deepen the understanding of LLMs' behavioral mechanisms, especially behavioral adjustments in evaluation environments.

Practical Value

Help developers select appropriate models and design reasonable evaluation strategies.

Future Directions

Explore complex evaluation scenarios, diverse model architectures, and the relationship between awareness and other characteristics.

The project is open-sourced on GitHub; code and datasets can be obtained to reproduce experiments. Contribution directions include: improving algorithm efficiency, expanding dataset scenarios, adding model support, and perfecting documentation and tutorials.