Conversational Programming Assessment: When AI Meets Code Understanding, How Do We Verify That Students Truly Learned?

This article focuses on the new dilemma in programming education in the LLM era—students can use AI to generate correct code but lack true understanding ("unproductive success"). Traditional Automatic Programming Assessment Systems (APAS) struggle to address this challenge, so the study proposes the Hybrid Socratic Framework, which uses conversational verification as a supplementary layer, combining the advantages of rule engines and LLMs to verify students' understanding of code, providing a new paradigm for programming education assessment.

LLM tools (such as ChatGPT) bring convenience to programming learning, but also lead to "unproductive success"—students submit functionally correct code but do not understand the logic. Traditional APAS rely on unit tests and static analysis, which become ineffective after the popularization of LLMs: students can generate perfect code via AI without actual mastery. This undermines educational fairness and effectiveness, so a new assessment method is urgently needed to verify code understanding.

The team from the University of Innsbruck followed the PRISMA guidelines and searched for literature (from Google Scholar, ACM Digital Library, etc.) after 2018 (post-Transformer era), identifying three conversational assessment technology routes:

1. Rule/template-based: High certainty but insufficient flexibility;
2. LLM-based: Natural interaction but with hallucination risks;
3. Hybrid system: Combines the advantages of the first two, balances quality and risk, and is considered the most practical.

The Hybrid Socratic Framework aims to supplement traditional APAS, with core components including:

• Deterministic code analysis layer: Static/dynamic code analysis to extract objective data such as structure and execution paths;
• Dual-agent dialogue layer: A "questioner" (Socratic tutor) guides explanations, and an "assessor" judges the depth of understanding to reduce bias;
• Knowledge tracking module: Records knowledge point mastery and builds personalized knowledge graphs;
• Scaffolded questioning: Adjusts question difficulty based on answers, provides hints or follow-up questions;
• Runtime fact anchoring: Binds questions to the actual execution state of the code (e.g., variable value changes) to avoid vague answers.

To address students using LLMs to generate dialogue answers, the framework designs the following strategies:

• Proctoring mode: Restricts access to external AI tools (browser locking, network monitoring, etc.);
• Randomized tracking questions: Randomly selects states from code execution traces to ask questions, making the dialogue path unique;
• Step-by-step reasoning requirement: Requires showing the reasoning process instead of just the final answer;
• Local model deployment: Supports local deployment of open-source models (Llama, Mistral) to ensure data privacy.

The framework has the following limitations:

1. Large-scale deployment requires significant computing resources;
2. The LLM hallucination problem is not fully resolved, which may lead to misjudgment of answers;
3. Privacy and academic integrity issues need continuous research. In the future, it is necessary to verify the framework's effect in more educational scenarios, explore more efficient large-scale solutions, and improve anti-cheating mechanisms.

Programming education assessment in the LLM era needs to keep pace with the times. The Hybrid Socratic Framework does not replace traditional tests but serves as a supplement, using AI to assist in verifying students' understanding. Its core is human-AI collaboration: technology enhances teachers' judgment ability to identify students who truly master knowledge. This model may become the new normal of programming education assessment.