HITLLLMs: A Study on Consistency Between Human Experts and LLMs in Chemical Synthesis Plan Evaluation

This study focuses on the consistency of opinions between human chemistry experts and large language models (LLMs) when evaluating the quality of chemical synthesis plans, providing an empirical basis for AI-assisted decision-making in the chemical field. The HITLLLMs project provides supporting code and raw feedback materials to facilitate the research in the paper titled 'Do humans and large language models agree on the quality of synthesis plans?'.

In the field of chemical synthesis, designing high-quality synthesis routes is a core challenge in drug discovery and materials science. With the improvement of LLM capabilities, researchers are exploring the possibility of using them to assist in the evaluation of synthesis plans, but the key issue of consistency between human and machine evaluations has not been fully addressed. The HITLLLMs project focuses on this problem.

LLM Query System

LLM evaluation results are obtained by calling OpenAI and VertexAI services via llm_querying/llms_querying.py. Raw responses are stored in responses_llms, and master_paths.json contains the synthesis plans presented to experts.

Feasibility Evaluation Framework

feasibility.py defines LLM prompts to ensure the evaluation method is comparable to that of human experts.

Statistical Analysis Workflow

human_vs_llm.ipynb implements data loading and preprocessing, consistency measurement, statistical significance testing, and chart generation, which can reproduce the paper's results.

The dataset consists of three parts: 1. Professional evaluations of retrosynthetic trees by human experts; 2. Evaluation results of the same plans by multiple LLMs; 3. Comparative analysis of human and machine feedback. All raw data is integrated into expert_feedback_combined_llms.csv for easy statistical analysis and visualization.

Contributions to Cheminformatics

Provides empirical data to help understand the performance boundaries of LLMs in chemical tasks, patterns of human-machine differences, and types of synthesis plans where agreement or disagreement occurs.

Implications for AI-Assisted Design

Guides model selection, prompt engineering optimization, human-machine collaboration process design, and consistency-based quality screening mechanisms.

The project is open-sourced under the MIT license, supporting: verification of the paper's statistical results, extension to more LLM models, application to other chemical datasets, and improvement of evaluation metrics. Its method of comparing human and machine evaluations can be extended to fields such as medical diagnosis and legal analysis. Environment configuration is done via conda environment files, and API credentials need to be configured.

The HITLLLMs project is an important case of human-machine collaboration research in cheminformatics, providing insights into the capabilities and limitations of AI through rigorous analysis. With the development of LLM technology, such basic research is of great significance for ensuring that AI tools effectively assist chemistry researchers.