# Less is More: The Art of Precise Regulation of LLM Engagement in Code Analysis Scenarios

> When integrating LLMs into static analysis tools, does more LLM engagement mean better results? This article compares three architectures with different LLM engagement levels and finds that the structured intermediate representation scheme outperforms direct generation and Agentic generation in effectiveness, while its token consumption is only 1/8 of the latter, providing important insights for LLM applications in formal domains.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-04-23T14:51:18.000Z
- 最近活动: 2026-04-24T02:57:57.698Z
- 热度: 147.9
- 关键词: LLM参与度, 静态分析, 代码分析, 结构化中间表示, Agentic生成, 形式化领域, Joern, CPGQL
- 页面链接: https://www.zingnex.cn/en/forum/thread/llm-77f8f2f1
- Canonical: https://www.zingnex.cn/forum/thread/llm-77f8f2f1
- Markdown 来源: floors_fallback

---

## [Introduction] Less is More: Core Findings on Precise Regulation of LLM Engagement in Code Analysis

This article compares three architectures with different LLM engagement levels (direct generation, structured intermediate representation, Agentic generation) and challenges the intuitive assumption that "more LLM engagement equals better results". Core findings: The structured intermediate representation scheme achieves the best performance, and its token consumption is only 1/8 of the Agentic scheme, providing important insights for LLM applications in formal domains.

## Background: The Intuitive Trap of Integrating LLMs into Static Analysis Tools

Large language models (LLMs) are widely used in software engineering, and natural language interfaces for static analysis tools (e.g., Joern, CodeQL) are typical scenarios. However, the issue of LLM engagement is often overlooked: existing systems range from direct query generation to Agentic multi-round calls, and these differences are treated as implementation details rather than independent variables. This article studies "LLM engagement" as an independent variable and challenges traditional intuitions.

## Comparative Design of Three LLM Engagement Architectures

The study designs three architectures distributed along the engagement spectrum (targeting Joern's CPGQL):
1. Direct Query Generation: LLM outputs complete CPGQL end-to-end, maximizing engagement;
2. Structured Intermediate Representation: LLM outputs an intermediate representation following a predefined JSON schema, which is then compiled into CPGQL by a deterministic converter;
3. Agentic Generation: LLM acts as an Agent to iteratively build queries via multi-round tool calls (schema querying, validation, etc.), with the highest engagement.

## Experimental Design and Evaluation Methods

The experiment constructs a benchmark of 20 code analysis tasks (three levels: simple/medium/complex), adopts a 2×2 model design (2 families × 2 scales), and repeats each configuration three times to ensure statistical significance. The core evaluation metric is "result matching rate" (the generated query returns a result set equivalent to the reference, which is stricter than grammatical correctness).

## Core Findings: Validation of Less is More

Experimental results show:
1. The structured intermediate representation scheme performs best: it improves by 15-25 percentage points over direct generation on large models and outperforms the Agentic scheme (which consumes 8x more tokens but yields worse results);
2. Model scale effect: Large models benefit more from structured constraints, while small models are limited by JSON schema compliance bottlenecks;
3. Token efficiency: The Agentic scheme consumes 8x more tokens than the structured scheme, leading to higher costs.

## In-depth Analysis: Why Do Constraints Bring Improvements?

Reasons for the advantages of the structured scheme:
1. Specificity of formal domains: Code analysis query languages have strict syntax and semantics, and LLMs have limited mastery of formal languages; constraining the output space avoids risks;
2. Value of deterministic post-processing: Separates semantic understanding by LLMs from formal query construction, achieving separation of concerns;
3. Error propagation control: Agentic multi-round interactions tend to accumulate errors, while the structured scheme's single-round generation + deterministic conversion reduces uncertainty.

## Implications for Practice

Practical guidelines:
1. Decision framework: Prioritize structured intermediate representation in formal domains, match model capabilities (large models are more suitable), and consider cost constraints;
2. Intermediate representation design: Balance expressiveness, learnability, and convertibility; iterative optimization is recommended;
3. Hybrid strategy: Choose based on task characteristics (direct generation for simple tasks, structured for complex ones, Agentic for tasks requiring dynamic information).

## Limitations and Future Directions

Limitations: The study focuses on code analysis; generalization to domains like SQL generation and configuration management needs verification; manual design of intermediate representations has a high threshold. Future directions: Explore automated learning of intermediate representations, optimize Agentic interaction modes (selective tool calls, early termination, etc.).
