Discovering Shared Logical Subspaces: Guiding LLM Reasoning via Alignment of Natural Language and Symbolic Perspectives

Core research findings of this paper: There exist cross-perspective shared logical subspaces (spanning natural language and symbolic perspectives) within LLMs. These subspaces can be extracted using canonical correlation analysis (CCA), and a training-free reasoning guidance method is designed to generate content directionally along these subspaces, achieving up to an 11-percentage-point accuracy improvement on logical reasoning benchmarks. This result provides a new path for understanding the logical reasoning mechanisms of LLMs and for neural-symbolic fusion.

Although LLMs perform well in tasks like text generation, multi-step logical reasoning remains a weakness. Existing solutions have limitations:

Pure Natural Language Methods (e.g., Chain-of-Thought)

• Unstable format, prone to logical jumps
• Lack of verification mechanism, intermediate conclusions are prone to hallucinations

External Symbolic Solvers

• Natural language to symbol conversion is error-prone
• Capability fragmentation; models do not truly learn to reason
• Unable to handle informal problems

Core Hypothesis

There exist low-dimensional shared logical subspaces in LLMs that encode logical reasoning capabilities in both natural language and symbolic forms, independent of surface forms.

Discovery Methods

1. Data Construction: Create a parallel reasoning corpus (natural language and symbolic solutions for the same problem)
2. Activation Collection: Input paired content and collect residual activations from each layer
3. CCA Analysis: Use canonical correlation analysis to find the low-dimensional subspace with maximum correlation between the two types of activations, confirming the existence of shared subspaces.

Guidance Mechanism

1. Generate initial reasoning steps
2. Extract residual activations from the current layer
3. Project onto the shared logical subspace
4. Integrate expected activations from the symbolic perspective
5. Adjust the model state to the ideal logical state
6. Continue generating the next step

Advantages

Training-free (no parameter fine-tuning/extra data needed), can be plug-and-play for any pre-trained model.

Benchmark Datasets

LogiQA, ReClor, ProofWriter, FOLIO

Results

• Average improvement of 7-8 percentage points
• Up to 11 percentage points improvement on ProofWriter
• Outperforms pure CoT prompting and simple integration methods

Generalization Capability

Effective across domains (e.g., mathematical proof → logical QA), indicating that the subspaces capture general logical capabilities.

Ablation Experiments

Single-perspective only has limited effect; combining both perspectives yields the best results.

Logical Patterns

• Deductive reasoning (e.g., modus ponens), inductive reasoning, and abductive reasoning form clear clusters
• Logical connectives (AND/OR/IF-THEN) have unique representation directions

Hierarchical Differences

• Shallow layers: Capture syntactic logical structures
• Middle layers: Capture semantic reasoning patterns
• Deep layers: Capture abstract logical relationships

Implications

1. There exist extractable logical reasoning capabilities within LLMs
2. New path for neural-symbolic fusion: Discover symbolic structures inside the network
3. Provide new tools for LLM interpretability

Limitations

• CCA assumes linear relationships
• Subspaces are static and not task-adapted
• Guidance process increases computational overhead

Future Work

• Explore nonlinear subspace discovery methods
• Dynamic subspace adaptive adjustment
• Extend to mathematical/physics/causal reasoning
• Optimize guidance algorithms to reduce overhead