LLM-based Multi-Agent Metamorphic Testing for FMU Simulation Models: A New Automated Verification Solution

This article introduces an innovative solution that uses an LLM-driven multi-agent workflow to automatically generate metamorphic relations from specifications, addressing the testing challenge of FMU (Functional Mock-up Unit) simulation models lacking explicit expected outputs. This solution comes from the paper "Multi-Agent Specification-based Metamorphic Testing of FMU-Based Simulations" published on arXiv on May 24, 2026 (link: http://arxiv.org/abs/2605.25101v1).

What are FMI and FMU?

FMI (Functional Mock-up Interface) is a widely adopted standard for simulation model exchange in industry, allowing models developed with different tools to be packaged into FMU format for exchange and facilitating cross-organizational collaboration.

Testing Challenges

1. Black-box nature: FMU is a binary file, so white-box testing is ineffective;
2. Lack of expected outputs: Complex dynamic systems have no known "correct outputs" as a benchmark;
3. State space explosion: Input space is infinite, making exhaustive testing impossible.

Core Idea of Metamorphic Testing

Instead of directly judging output correctness, it checks the reasonable relationships between outputs (metamorphic relations, MRs). For example: sin(-x) = -sin(x). Industrial MRs include scaling, monotonicity, invariance, conservation laws, etc.

LLM Multi-Agent Framework

Collaborated by 5 types of agents:

1. Specification Parsing Agent: Reads specification documents and identifies variables and requirements;
2. Requirement Extraction Agent: Identifies MR sources (symmetry, conservation laws, etc.) from specifications;
3. MR Generation Agent: Generates formal MRs using the Given-When-Then pattern;
4. Test Generation Agent: Converts MRs into executable test cases;
5. Execution and Verification Agent: Coordinates FMU simulation, verifies MRs, and generates reports.

Advantages of the Given-When-Then Pattern

High readability, clear structure, easy to automate, and traceable.

Examples of Automatically Generated MRs

• MR-1: Load-temperature monotonicity (Under steady state, increasing heat load leads to a monotonic rise in oil temperature);
• MR-2: Flow conservation (In a closed loop, the inflow and outflow of the radiator are equal);
• MR-3: Cooling efficiency boundary (When ambient temperature is fixed, adjusting fan speed to 100% results in a reasonable drop in oil temperature).

Experimental Results

• Successfully generated physically reasonable MRs;
• Significantly reduced manual workload;
• Discovered anomalies in model boundary conditions;
• Improved test coverage.

Advantages

• Multi-agent collaboration: Specialized, interpretable, scalable, and robust;
• LLM role: Natural language understanding, domain knowledge reasoning, creative generation, and formal transformation.

Limitations

• Dependent on the quality of specification documents;
• LLMs may generate hallucinated MRs;
• High computational cost;
• Adaptability to other physical domains needs verification.

For Simulation Model Developers

1. Emphasize the quality of specification documents as the foundation for automated testing;
2. Adopt metamorphic thinking for test design;
3. Human-machine collaboration: Use LLMs to generate initial MR drafts, then refine them with expert review.

For Test Engineers

1. When "correct outputs" cannot be defined, use metamorphic relations to verify relationships between outputs;
2. Specification-driven testing: Extract test basis from the requirement phase;
3. AI as an assistant rather than a replacement; final decisions depend on human judgment.

Summary

This solution combines LLM multi-agent and metamorphic testing, providing a new idea for industrial simulation model verification. It has been proven to reduce manual workload and improve test coverage.

Future Directions

1. Develop automatic MR quality assessment methods;
2. Introduce active learning to optimize MR generation;
3. Support multi-modal specification documents;
4. Explore real-time test generation in CI/CD pipelines.