AI Achieves Perfect Score on LSAT for the First Time: A New Milestone in Reasoning Ability

A research team has for the first time documented a large language model achieving a perfect score on the Law School Admission Test (LSAT), indicating that AI reasoning ability has reached or exceeded the top human level. The study verified through controlled experiments that the result was not accidental, and revealed the critical role of chain-of-thought in reasoning performance. It also explored directions such as the limitations of distilled models and the optimization of process reward models, which have far-reaching cognitive and industry significance.

Since 1948, the LSAT has served as a gatekeeper for elite legal education, testing high-order human cognitive abilities such as logical reasoning and analytical reading. AI achieving a perfect score on the LSAT (completing all questions with zero errors) means its reasoning ability has reached the upper limit of human cognitive capabilities, marking a significant breakthrough in AI cognitive development.

The research team designed multiple controlled experiments to eliminate interference: testing different prompts had no substantial impact; shuffling option orders ruled out the possibility of memory-based positioning; and multiple samplings yielded consistent results. These experiments prove that the AI's perfect score stems from genuine reasoning ability, not chance or trickery.

Ablation experiments showed that removing chain-of-thought (the model's intermediate reasoning process) reduced the accuracy of cutting-edge models by up to 8 percentage points, primarily affecting the logical reasoning section. This confirms the importance of explicit reasoning processes; the quality of chain-of-thought is more critical than its form, providing directions for model improvement.

Comparing cutting-edge models with distilled models revealed that although distilled models can generate chain-of-thought in the same format, their performance is far lower. This indicates that knowledge distillation may replicate surface forms but fail to transfer deep reasoning strategies, suggesting that reasoning ability involves complex cognitive architecture, and simply compressing models may sacrifice core reasoning capabilities.

The study attempted to fine-tune a Process Reward Model (PRM) on LSAT explanation materials using QLoRA technology, combined with the Best-of-5 strategy to select the optimal answer. This successfully narrowed the performance gap between distilled models and cutting-edge models, with improvements focusing on the logical reasoning section, providing new ideas for the development of efficient reasoning models.

This breakthrough redefines the boundaries of cognitive ability, prompting reflection on the education evaluation system and changes in the legal industry, marking progress toward Artificial General Intelligence (AGI). However, AI reasoning still has limitations: optimization in specific domains, the gap between exams and reality, and interpretability challenges. Future research can focus on ability transfer, efficient optimization, and new models of human-AI collaboration.