Comparative Study on Analogical Reasoning Capabilities of Transformer Architectures: Systematic Evaluation of BERT, RoBERTa, GPT-2, and T5

This study systematically evaluates the analogical reasoning capabilities of five mainstream Transformer models: BERT, RoBERTa, DistilBERT, GPT-2, and T5, and explores the impact of different architectural designs (such as bidirectional/unidirectional attention mechanisms, pre-training objectives, etc.) on the models' ability to understand structured relationships. The results provide empirical evidence for model selection, architectural improvement, and understanding of machine cognitive mechanisms, with key findings including the superior performance of bidirectional encoder models, the effectiveness of training strategy optimization and knowledge distillation, etc.

Analogical reasoning is the core of human intelligence, enabling the recognition of structured relationships between concepts and their mapping to new scenarios. With the breakthroughs of Transformer models in NLP tasks, a key question emerges: Do these models truly possess analogical reasoning capabilities, or do they only simulate it superficially? This question is crucial for technical evaluation and understanding the essence of machine intelligence.

The study designed a prompt dataset using the classic analogy format (A is to B as C is to [MASK]), testing models in different ways: encoder models (BERT/RoBERTa/DistilBERT) use mask prediction, autoregressive models (GPT-2) use conditional probability completion, and T5 is converted into a sequence-to-sequence task. This framework can quantify the models' mastery of relationship types such as semantics and functions.

The models involved in the evaluation represent different design philosophies: BERT is a bidirectional encoder; RoBERTa optimizes BERT's training strategy (removing next sentence prediction, using larger batches/data); DistilBERT compresses parameters via knowledge distillation; GPT-2 is a unidirectional autoregressive decoder; T5 adopts a unified text-to-text framework.

Bidirectional encoder models (BERT/RoBERTa) performed excellently, with RoBERTa being superior due to training optimization; DistilBERT has fewer parameters but retains similar performance; GPT-2 performed weakly due to unidirectional modeling; T5's performance depends on the quality of prompt engineering and can be competitive when properly converted.

The study reveals that attention directionality (bidirectional vs. unidirectional) is a key factor—bidirectional attention facilitates the recognition of cross-word pair relationships; differences in pre-training objectives (masked vs. autoregressive) shape the type of representations. For strong analogical reasoning, bidirectional encoders are better; to balance generation and reasoning, hybrid architectures or new pre-training objectives can be explored.

The results can guide fields such as educational technology (intelligent tutoring), knowledge graphs (entity relationship discovery), and creative tools (cross-domain concept transfer). Future directions include expanding analogy types (cross-modal, abstract concepts), evolution of large-scale models, and building cognitive evaluation systems by combining multiple reasoning types.

Current models perform well at the lexical level, but structured reasoning still needs improvement. Architectural design (attention, pre-training objectives, training strategies) deeply affects cognitive capabilities, providing practical value for model development and application selection. This study lays the foundation for understanding the boundaries of machine cognition and building the next generation of cognitive models.