System Dynamics AI Assistant Benchmark Test: Comprehensive Comparative Analysis of Cloud and Local Large Language Models

This article conducts a benchmark test on System Dynamics AI Assistants, comparing the performance of cloud APIs and locally deployed open-source models in causal loop diagram (CLD) extraction and interactive model discussion tasks. Key findings include: the impact of backend framework selection on performance far exceeds that of quantization precision; locally optimized models (e.g., Kimi K2.5 GGUF Q3) can match mid-tier cloud models in CLD tasks; and practical guidelines are provided for running ultra-large-scale models on Apple Silicon. This thread will analyze the research background, methodology, findings, and practical recommendations in detail across different floors.

Research Background and Motivation

System Dynamics is widely used in supply chain management, climate change modeling, public health policy, and other fields. Traditional modeling relies on expert knowledge, and LLMs bring the possibility of automated auxiliary modeling, but they need to understand complex causal relationships, generate structured CLDs, and conduct in-depth interactive discussions. Currently, there is a lack of systematic evaluation of cloud and local models in this professional field. Researchers and practitioners face the choice between using convenient but privacy-risky cloud services or investing resources to build local solutions, which needs to be based on objective performance data rather than assumptions.

Benchmark Framework Design

This study constructs two evaluation benchmarks:

CLD Leaderboard: Structured Causal Loop Diagram Extraction

It includes 53 test cases to evaluate the model's ability to extract and generate standardized JSON-format CLDs (nodes, connections, polarities) from natural language, covering simple single loops to complex multi-layer feedback networks.

Discussion Leaderboard: Interactive Model Discussion and Guidance

It evaluates the model's performance in three scenarios: model construction step guidance, feedback explanation, and error repair assistance. It simulates real teaching scenarios, requiring coherent multi-turn dialogues, targeted suggestions, and model improvement guidance.

Key Research Findings

Cloud Models Lead, Local Models Catch Up

CLD tasks: Cloud proprietary models have a pass rate of 77%-89%, while the best local model Kimi K2.5 GGUF Q3 achieves 77% in zero-shot settings, matching mid-tier cloud models; Discussion tasks: Local models perform reasonably well in construction guidance (50%-100%) and feedback explanation (47%-75%), but only 0%-50% in error repair, as long contexts have high requirements for memory and window length.

Critical Impact of Backend Frameworks

Backend frameworks have a greater impact than quantization precision:

• GGUF and llama.cpp backends: Syntax-constrained sampling ensures standardized JSON output, but long contexts may lead to infinite generation (for dense models);
• MLX backend: No mandatory JSON constraints, requiring explicit format guidance in prompts—flexible but increases development complexity.

Practical Impact of Quantization Precision

Comparing configurations like Q3, Q4_K_M, and MLX-3bit, quantization can significantly reduce memory usage. For example, Kimi K2.5 GGUF Q3 is competitive in CLD task performance while reducing hardware requirements, making it possible to run ultra-large-scale models on consumer-grade hardware like Apple Silicon.

Implications for Practitioners

Hardware Configuration Recommendations

Guidelines for running 671B-123B parameter models on Apple Silicon:

• Leverage the unified memory architecture and adapt memory via quantization;
• For tasks requiring strict JSON output, prioritize llama.cpp; for flexibility, choose MLX;
• For long-context tasks, ensure sufficient memory or use segment processing.

Parameter Tuning Strategies

Key sampling parameter (temperature, top-p, top-k) scanning results: Use low temperature (deterministic output) for structured tasks, and appropriately increase temperature (diversity) for creative discussions.

Prompt Engineering Best Practices

• MLX backend: Explicitly state format requirements and examples in prompts;
• llama.cpp backend: Avoid prompt designs that lead to infinite generation.

Limitations and Future Directions

Limitations

• The tests are based on specific System Dynamics scenarios, so generalization needs to be cautious;
• Locally deployed performance depends on hardware configuration and software optimization, and environmental differences affect performance.

Future Directions

• Explore more efficient model compression technologies;
• Develop fine-tuning datasets and training methods for the System Dynamics field;
• Research multi-model collaboration architectures (combining the advantages of cloud and local models).

Conclusions

Locally deployed open-source models have shown considerable competitiveness in professional field tasks, with structured output tasks approaching cloud models; the choice of backend framework has a decisive impact on actual results, exceeding that of quantization precision; the Apple Silicon operation guidelines have important reference value. With the improvement of model efficiency and hardware development, the boundary between cloud and local is blurring. Local deployment is more attractive for scenarios with sensitive data or privacy requirements, promoting the democratization of AI-assisted System Dynamics modeling tools and benefiting more people.