ALUE: A Professional Large Language Model Evaluation Framework for the Aerospace Domain

MITRE's ALUE (Aerospace Language Understanding Evaluation) framework provides a standardized solution for evaluating large language models (LLMs) in the aerospace domain. This framework fills the gap in vertical domain model evaluation, supporting local GPU inference, remote API calls (such as TGI and OpenAI-compatible endpoints), custom datasets, and quantitative metrics to facilitate scientific evaluation and selection of models in the domain.

As LLMs are widely applied across industries, general benchmark tests struggle to meet the high requirements for safety, accuracy, and domain knowledge in the aerospace field. General model evaluation tools cannot capture performance differences in special scenarios of this domain. MITRE launched the ALUE framework precisely to address this issue and fill the gap in professional domain model evaluation.

The ALUE framework is user-friendly and highly configurable, supporting multiple operation modes:

• Local inference: Run open-source models like Llama and Mistral using local GPUs
• TGI (Text Generation Inference): HuggingFace's high-performance inference service, which has been tested to reduce the inference time of the Mistral-7B-v0.1-Instruct model from 15 minutes 45 seconds to 4 minutes 43 seconds (based on 586 questions)
• OpenAI-compatible endpoints: Support various remote services compatible with the OpenAI API These modes allow users to choose flexibly based on resources, significantly improving inference efficiency.

The core advantage of ALUE lies in its domain specificity:

• Built-in aerospace-specific datasets
• Allows users to create/import custom datasets, define domain-specific evaluation metrics, and configure custom prompt templates In addition, the framework maintains a public online leaderboard showing the performance of different models on aerospace datasets, providing references for model selection and driving technological progress in the domain.

ALUE uses uv as the package management tool, supporting Python 3.10/3.11. Installation only requires the uv sync command to automatically create a virtual environment and install dependencies. Model configuration is implemented via config.py, supporting local models (specify path) and remote endpoints (configure api_endpoint). The operation flow is: Configure model → Select operation mode → Execute evaluation script → View quantitative results.

For aerospace enterprises: Can evaluate model performance in tasks such as flight manual understanding and Q&A, maintenance document analysis, aviation regulation compliance checks, and safety report processing; For researchers: Can establish domain benchmark standards, compare the professional performance of models with different architectures, identify model knowledge gaps and biases, and promote the development of domain-specific models.

ALUE is not just an evaluation tool but an open ecosystem. Project documents detail how to create custom datasets, encouraging the community to contribute aerospace domain test cases. Through open collaboration, the evaluation system is continuously improved to better meet actual application needs.

ALUE represents an important direction for vertical domain LLM evaluation, demonstrating the limitations of general benchmark tests and the feasibility of building targeted evaluation frameworks for specific industries. As the digital transformation of the aviation industry deepens, ALUE will provide a scientific basis for model development and selection, which is expected to enhance aviation safety and optimize operational efficiency. It is recommended to pay attention to and participate in this open-source project.