Framework for Analyzing the Quality of Student-LLM Collaboration: A New Learning Analytics Approach in Higher Education Writing Scenarios

This article introduces the Student-LLM Collaboration Quality Analysis Framework proposed by the student-llm-collaboration-analysis project. Addressing the pain points in educational assessment after AI has become a learning partner, it provides a systematic and actionable learning analytics tool. The framework quantifies the depth of human-AI collaboration through multi-dimensional indicators, helping educators distinguish between high-quality collaboration and low-quality dependency, and offering a new method for evaluating AI-assisted learning in higher education writing scenarios.

The traditional educational assessment system is based on the assumption that students complete tasks independently, but the popularity of AI-assisted writing has made the original assessment dimensions insufficient. The core research questions of the project include: Are there quality differences in student-LLM interactions? What is the difference between high-quality collaboration and dependency? How to design quantitative indicators? The writing scenario was chosen because it is a cognitively intensive activity and a common use case for LLM assistance.

The framework design follows three principles:

1. Process-oriented: Focus on the interaction process (number of dialogue turns, questioning strategies, etc.) rather than just the final text;
2. Multi-dimensional characterization: Evaluate from perspectives such as cognitive engagement and metacognitive awareness;
3. Interpretable and actionable: Indicators are clearly defined, making them easy for educators to apply and reproduce.

The framework includes four assessment dimensions:

• Interaction depth: Measure the complexity of dialogue (multi-turn conversations, follow-up clarifications, etc. vs. simple copy requests);
• Cognitive strategy: Analyze the cognitive level in prompts (clear goals, questioning and verifying AI outputs, etc.);
• Metacognitive awareness: Evaluate the ability to monitor the learning process (asking AI to explain reasoning, recognizing its limitations);
• Creative transformation: The key to distinguishing collaboration from ghostwriting, measuring the degree to which AI outputs are transformed into original expressions.

The technical implementation uses mixed methods:

1. Data collection: Collect complete dialogue logs (prompts, responses, editing history);
2. Feature extraction: Use NLP technology to extract interaction features (number of turns, vocabulary complexity, etc.);
3. Quality assessment: Build scoring standards based on educational theory and train evaluators to label samples;
4. Model training: Train models using labeled data to achieve large-scale automated assessment.

The framework's application scenarios include:

• Optimization of teaching feedback: Provide students with personalized AI collaboration suggestions;
• Academic integrity assessment: Distinguish between legitimate assistance and academic misconduct;
• Curriculum design guidance: Design teaching activities targeting collaboration misconceptions;
• Learning analytics research: Provide standardized tools to promote the accumulation of domain knowledge.

Current framework limitations: Focused on writing scenarios, generalization to other disciplines needs verification; LLM evolves rapidly, so indicators need to be updated. Future directions:

• Expand to tasks such as programming and mathematics;
• Develop real-time feedback systems;
• Explore differences in collaboration patterns across cultural backgrounds;
• Establish large-scale benchmark datasets.

This project is a positive response to educational assessment in the AI era. It not only provides technical tools but also redefines the concept of learning collaboration. In the future, cultivating students' ability to collaborate efficiently with AI will become a core competency, and this framework provides an important starting point for educators to understand and develop this ability.