Text Embedding-Based Cognitive Diagnosis: A New Paradigm for Evaluating Large Language Model Capabilities

The Text-Embedding-CDM-LLM project proposes an innovative cognitive diagnosis method that uses text embedding technology to perform fine-grained capability evaluation of large language models (LLMs). It addresses the problem where traditional evaluations can only measure correctness but fail to reveal the causes of defects, and does not require manual annotation of knowledge points, providing a new perspective for model assessment and optimization.

With the rapid development of LLMs, traditional evaluations rely on standardized test sets and can only calculate the matching degree between outputs and standard answers, but cannot reveal the reasons for model errors or specific knowledge point defects. Cognitive diagnosis has been introduced into the AI field, but traditional models require manual annotation of knowledge points, and the complex capability dimensions of LLMs make exhaustive annotation difficult, which has become an application bottleneck.

The core solution of the project is to use text embedding technology to realize automated cognitive diagnosis: encode questions and model answers into high-dimensional embedding vectors, compare semantic space distribution patterns to identify capability differences, without manual definition of knowledge point labels, fully data-driven, and highly scalable.

Embedding Encoding Layer

Models such as Sentence-BERT and OpenAI's text-embedding series are used to convert text into dense vectors that capture deep semantic meanings.

Cognitive State Modeling Layer

Based on the Item Response Theory (IRT) and Cognitive Diagnosis Model (CDM) frameworks, the similarity measurement in the embedding space is used to replace the manually annotated knowledge point associations.

Diagnostic Reasoning Layer

Through Bayesian inference or neural networks, the mastery level of each capability dimension of the model is inferred from the response patterns, and a fine-grained capability profile is output.

• Model Selection and Comparison: Fine-grained capability comparison helps select appropriate models according to scenarios;
• Model Optimization Guidance: Identify deficiencies and collect data or adjust architectures in a targeted manner;
• Education Field: Evaluate the subject knowledge mastery of teaching assistant models;
• Safety Assessment: Diagnose capabilities in dimensions such as ethics, bias, and harmful content recognition.

This method opens up a path for unsupervised/weakly supervised cognitive diagnosis, solving the limitation of traditional methods relying on manual annotation and adapting to the rapid development of LLMs. In the future, it is expected to combine with model interpretability research to reveal how models "know" and why they "make mistakes", laying a foundation for building reliable and controllable LLMs.