ElectriQ: A Benchmark for Large Language Models in Power Marketing

ElectriQ is a benchmark dataset for evaluating large language models (LLMs) in power marketing scenarios, aiming to provide professional evaluation standards for AI applications in the energy industry. Its core objectives include professional assessment, practicality verification, safety inspection, and comparability analysis, helping to solve the problem of evaluating LLM performance in the highly specialized field of power marketing.

The global energy industry is undergoing digital transformation. The development of smart grids and distributed energy has complicated power marketing scenarios, shifting from one-way power supply to two-way interactive services, and putting forward new requirements for customer service, etc. LLMs have potential in solving complex consulting and optimizing strategies, but power marketing involves multi-dimensional professional knowledge, so how to accurately evaluate LLM performance has become an urgent problem to be solved.

The sources of the ElectriQ dataset include industry standards, policies and regulations, academic literature, practical cases, and expert knowledge; the question types cover four categories: knowledge Q&A, scenario application, policy interpretation, and safety compliance; each question is equipped with a standard answer reviewed by experts, including reference answers, scoring points, common mistakes, difficulty levels, and knowledge domain labels.

The evaluation dimensions include accuracy, completeness, logic, practicality, and safety; the evaluation methods adopt automatic evaluation (keyword matching, semantic similarity, etc.), manual evaluation (expert scoring, cross-validation, etc.), and comparative evaluation (ranking analysis, difference analysis, etc.) to ensure reliable results.

The evaluation results of ElectriQ can be applied to scenarios such as model selection (helping enterprises choose suitable LLMs), model optimization (guiding fine-tuning), application design (referencing capability boundaries), and risk management (identifying potential safety hazards).

Research findings show that LLMs have problems such as unbalanced knowledge mastery (good at general knowledge but insufficient in professional details), need to improve computing ability (high error rate in numerical calculation), insufficient safety awareness (answers about grid safety are not cautious enough), and timeliness issues (lag in understanding the latest policies).

Current limitations include coverage (focusing on the Chinese market), language restrictions (mainly Chinese), slow dynamic updates, and scenario limitations (mainly text Q&A); future directions include expanding international scenarios, supporting multimodality, real-time updates, industry customization, and adversarial testing.