Revealing Reproducibility Illusions in Large Language Model APIs: Same Prompt, Different Answers

A study submitted to Nature Machine Intelligence systematically reveals the reproducibility issue of mainstream Large Language Model (LLM) APIs producing inconsistent outputs under the same prompt. This problem not only affects user experience but also touches the core of scientific research and practical applications—reproducibility. The genai-reproducibility-protocol project is quantifying this overlooked "reproducibility illusion" and proposing standardized solutions.

Reproducibility is the cornerstone of scientific research. However, in the LLM field, even when controlling variables like prompts and model versions, API calls still produce different outputs, eroding the reliability of academic research. Worse still, many researchers do not fully recognize or report this issue, only presenting "representative" outputs, which may mislead judgments about model capabilities.

The genai-reproducibility-protocol project has been submitted to Nature Machine Intelligence (2026), with the core goal of establishing a standardized protocol to measure LLM API reproducibility. Key contributions include: standardized testing protocols, multi-model comparative analysis, quantification of influencing factors, and best practice recommendations. The measurement framework uses multiple calls (100+ times), with indicators covering response consistency rate, semantic similarity distribution, key information variation, confidence calibration, etc.

The roots of inconsistent LLM API outputs include: 1. Randomness mechanisms (sampling strategies introduce variation; even with temperature 0, randomness may still exist); 2. Hardware and parallel computing (GPU scheduling leads to differences in operation order, which cumulatively affect outputs); 3. API opacity (commercial APIs have black-box characteristics; users cannot know hardware/weights/parameters); 4. Model update drift (silent background weight updates are not disclosed).

Preliminary results show that the consistency rate for certain tasks (e.g., code generation, mathematical reasoning) is below 50%, meaning that the "typical" results in papers may just be random samples. More worryingly, there are systematic biases in key information variation, and models may give contradictory factual statements without a warning mechanism.

For Academia: Call for mandatory reporting of statistical results from multiple runs, open-sourcing of experimental protocols, establishment of reproducibility benchmarks, and distinction between exploratory and confirmatory research. For Industry: Recommend using output aggregation (voting from multiple calls), deterministic modes, version locking, and internal confidence assessment mechanisms to reduce business risks.

The project has initiated an important dialogue on LLM reliability, but there are still unresolved issues: How to balance creativity and determinism? How much transparency responsibility should API providers bear? Is there a technical solution to fundamentally solve reproducibility? The project team will continue to update the protocol and call on the community to participate in solving this issue.