Prompt Engineering New Discovery: Code Generation Improvement Comes from Structure Rather Than Content—A Pre-Registered Controlled Study on Popperian Prompt Techniques

The latest pre-registered controlled study reveals: The key to the code generation improvement effect of prompt techniques that guide LLMs to act as Popperian falsifiers comes from structural frameworks rather than specific content. Through a two-level ablation experiment, the study found no significant difference in code correctness between the complete prompt technique and the framework that only retains labels, providing an important calibration basis for prompt engineering practice.

In recent years, LLMs have been widely used in tasks such as code generation. To improve performance, "prompt techniques" (e.g., guiding models to act as Popperian falsifiers) have become popular practices. However, the effects of such techniques are mostly evaluated through "LLM-as-a-judge", which has biases such as position and self-preference, raising a core question: Does the effect come from Popperian content or the organizational effect of structured frameworks?

The study uses a pre-registered two-level ablation experiment with three control conditions: 1. Length-matched placebo (to control length effect); 2. Label-only framework (retains structure, strips content); 3. Execution oracle (uses HumanEval + unit tests as objective indicators). Vocabulary halo sentinels and self-judgment audits are also added to capture biases. Model selection: cutting-edge model Claude Sonnet 4.6 (N=163), small model Qwen2.5-Coder-0.5B (N=164), to observe the consistency of effects across models of different scales.

1. Cutting-edge model (Claude Sonnet4.6): Performance under all conditions is close to the ceiling, with no significant differences; 2. Small model (Qwen2.5-Coder-0.5B): Structured prompts (complete technique / label-only framework) improve by 20-22 percentage points compared to the unstructured baseline, with both having an accuracy rate of 34.8% (no significant difference); the placebo is only 2.4 percentage points behind (limited contribution from length); 3. Self-judgment of small models fails: Performance does not exceed random, with 60% of choices concentrated on a single index, confirming that LLM-as-a-judge is unreliable for small models.

1. Structure first: When designing prompts, priority should be given to information organization and attention guidance rather than over-pursuing specific content; 2. Evaluation reflection: Caution is needed when relying on LLM-as-a-judge; execution correctness (e.g., unit tests) should be prioritized; 3. Value of negative results: Define the effective boundaries of prompt techniques to avoid resource waste; 4. Reusable protocol: Provide a standardized ablation scheme to facilitate the verification of other prompt techniques.

Limitations: The conclusions are limited to a specific family of prompt techniques, not an evaluation of Popperian methodology itself; the ceiling effect of cutting-edge models suggests that existing benchmarks are insufficient. Future directions: Explore whether other prompt techniques follow the "structure > content" pattern; the importance of content specificity in complex tasks; design hybrid prompt strategies that combine structure and domain knowledge.