Human-Eval-BIA: A Large Language Model Code Generation Benchmark for Biological Image Analysis

Human-Eval-BIA is the first dedicated code generation benchmark suite for large language models (LLMs) in the field of biological image analysis. Modified based on OpenAI's HumanEval framework, it evaluates the performance of LLMs on scientific image processing tasks using over 50 professional test cases, compares the actual results of 15 mainstream LLMs, and provides objective data support for researchers to select AI programming assistants.

Large language models excel in code generation, but general-purpose benchmarks fail to reflect their performance in specific scientific fields. Biological image analysis is a core component of life sciences, involving professional tasks such as microscope image processing and cell segmentation, which have high requirements for code accuracy, efficiency, and rigor. Human-Eval-BIA fills the gap in evaluation: deeply modified based on HumanEval, it provides a standardized evaluation method, compares the performance of 15 mainstream LLMs, and offers data support for selecting AI programming assistants.

Modified based on OpenAI's HumanEval framework, it retains the core of the pass@k metric and reconstructs the test case library. The design of test cases follows the principles of scientific accuracy first, practicality orientation, verifiability, and difficulty stratification, covering typical tasks such as image filtering, segmentation, and morphological operations. Currently, it includes over 50 test cases and is continuously expanding.

It adopts the pass@k metric, calculating pass@1 (pass rate for a single generation) and pass@10 (probability of passing at least once in ten generations). Multi-dimensional analysis is conducted based on task types, difficulty levels, and image dimensions (2D/3D), helping to understand the strengths and weaknesses of models.

The tests include OpenAI GPT-4 series, Anthropic Claude series, Google Gemini series, open-source models (Llama, CodeLlama, etc.), and Blablador services. Key findings: Closed-source models have obvious advantages (pass@1 is 20-30 percentage points higher); basic operations perform well, but domain knowledge tasks vary; 3D processing is a common weakness; open-source models (CodeLlama, DeepSeek Coder) are catching up. Visualization results such as overall pass@k comparison and task-specific heatmaps are provided.

Installation and Usage: Requires Python 3.10+. Create an environment using conda/mamba, clone the repository, install dependencies, configure the corresponding model API key, then run the tests. Results are saved as JSON/CSV. Community Contributions: Submit new test cases, report issues, improve the framework, test new models. The project is open-source under the MIT license.

Current Limitations: Limited test coverage, static testing does not involve interactive debugging, and code performance is not evaluated. Future Plans: Expand the test case library, introduce performance evaluation, develop interactive test scenarios, and establish a long-term model tracking mechanism.

Human-Eval-BIA demonstrates that general-purpose code benchmarks cannot meet the needs of specific scientific fields, and domain-specific evaluation systems are crucial for AI-assisted scientific research. It provides a reference for practitioners to select models, reveals the limitations of model capabilities for AI researchers, and shows the method of building domain benchmarks for the open-source community. As LLMs penetrate deeper into scientific research, such benchmarks will play an increasingly important role.