Less is More: The Art of Precise Regulation of LLM Engagement in Code Analysis Scenarios

This article compares three architectures with different LLM engagement levels (direct generation, structured intermediate representation, Agentic generation) and challenges the intuitive assumption that "more LLM engagement equals better results". Core findings: The structured intermediate representation scheme achieves the best performance, and its token consumption is only 1/8 of the Agentic scheme, providing important insights for LLM applications in formal domains.

Large language models (LLMs) are widely used in software engineering, and natural language interfaces for static analysis tools (e.g., Joern, CodeQL) are typical scenarios. However, the issue of LLM engagement is often overlooked: existing systems range from direct query generation to Agentic multi-round calls, and these differences are treated as implementation details rather than independent variables. This article studies "LLM engagement" as an independent variable and challenges traditional intuitions.

The study designs three architectures distributed along the engagement spectrum (targeting Joern's CPGQL):

1. Direct Query Generation: LLM outputs complete CPGQL end-to-end, maximizing engagement;
2. Structured Intermediate Representation: LLM outputs an intermediate representation following a predefined JSON schema, which is then compiled into CPGQL by a deterministic converter;
3. Agentic Generation: LLM acts as an Agent to iteratively build queries via multi-round tool calls (schema querying, validation, etc.), with the highest engagement.

The experiment constructs a benchmark of 20 code analysis tasks (three levels: simple/medium/complex), adopts a 2×2 model design (2 families × 2 scales), and repeats each configuration three times to ensure statistical significance. The core evaluation metric is "result matching rate" (the generated query returns a result set equivalent to the reference, which is stricter than grammatical correctness).

Experimental results show:

1. The structured intermediate representation scheme performs best: it improves by 15-25 percentage points over direct generation on large models and outperforms the Agentic scheme (which consumes 8x more tokens but yields worse results);
2. Model scale effect: Large models benefit more from structured constraints, while small models are limited by JSON schema compliance bottlenecks;
3. Token efficiency: The Agentic scheme consumes 8x more tokens than the structured scheme, leading to higher costs.

Reasons for the advantages of the structured scheme:

1. Specificity of formal domains: Code analysis query languages have strict syntax and semantics, and LLMs have limited mastery of formal languages; constraining the output space avoids risks;
2. Value of deterministic post-processing: Separates semantic understanding by LLMs from formal query construction, achieving separation of concerns;
3. Error propagation control: Agentic multi-round interactions tend to accumulate errors, while the structured scheme's single-round generation + deterministic conversion reduces uncertainty.

Practical guidelines:

1. Decision framework: Prioritize structured intermediate representation in formal domains, match model capabilities (large models are more suitable), and consider cost constraints;
2. Intermediate representation design: Balance expressiveness, learnability, and convertibility; iterative optimization is recommended;
3. Hybrid strategy: Choose based on task characteristics (direct generation for simple tasks, structured for complex ones, Agentic for tasks requiring dynamic information).

Limitations: The study focuses on code analysis; generalization to domains like SQL generation and configuration management needs verification; manual design of intermediate representations has a high threshold. Future directions: Explore automated learning of intermediate representations, optimize Agentic interaction modes (selective tool calls, early termination, etc.).