Automatic Learning and Execution of Context-Sensitive Constraints: Enabling Large Language Models to Master Generation Rules Independently

This paper proposes a framework for automatically learning context-sensitive constraints. Through two stages—syntax exploration and constraint exploitation—LLMs can independently learn constraint rules from interactions and execute them. Experiments show that small models with only 1B parameters can achieve perfect constraint compliance, outperforming larger models and state-of-the-art reasoning models. This framework lowers the threshold for applying context-sensitive constraints and provides a new path for controlled generation of LLMs.

LLMs have strong text generation capabilities, but ensuring outputs comply with specific formats, syntax, or domain constraints is a challenge. Traditional methods relying on Context-Free Grammars (CFG) cannot capture context-sensitive constraints (e.g., SQL column names must exist in the FROM table, variables must be declared before use, etc.). Context-sensitive constraints usually need to be manually specified by humans, relying on domain expertise, which has a high threshold and limits their application.

The framework consists of two stages: 1. Syntax Exploration: Allow LLMs to freely generate samples (including legal and illegal ones), and build constraint awareness through iterative feedback from verification (compilers/parsers); 2. Constraint Exploitation: Encode the learned constraints into executable forms, dynamically filter valid tokens during generation, and ensure outputs comply with rules.

Experiments show that small models with 1B parameters can perfectly follow complex context-sensitive constraints with the support of this framework. Their performance does not rely on model scale stacking and outperforms larger models and state-of-the-art reasoning models. Large models are prone to rule-violating "hallucinations", while reasoning models focus on semantic reasoning rather than structural constraints. The mechanism designed by this framework for constraint compliance is more effective.

This work is the first to realize the deep integration of context-sensitive grammar learning and LLM generation, filling a technical gap; it lowers the threshold for applying constraints, allowing developers to avoid manually writing rules; it reveals the non-linear relationship between constraint compliance and model scale, providing insights for efficient model design. It eliminates the need for manual specifications, saves manpower, and may discover implicit rules.

The framework can be applied to scenarios such as code generation (complying with syntax/type constraints), data extraction (following schemas), NL2SQL/NL2Code (maintaining semantic integrity), and dialogue systems (executing response format strategies). As LLMs are increasingly used in critical tasks, this framework will enhance their reliability and practicality.