Research on Sampling Parameters for Large Language Model Inference: An Empirical Analysis of Temperature, Top-K, and Top-P

Original Author/Maintainer: Max-0607 Source Platform: GitHub Original Project Name: masters-thesis-llm-inference Project Link: https://github.com/Max-0607/masters-thesis-llm-inference Release Time: 2026-06-12

This study conducts an empirical analysis of sampling parameters (Temperature, Top-K, Top-P) for multilingual large language model inference, delving into their impact on the stability and quality of model outputs, revealing task dependence, language differences, and interaction effects in parameter selection, and providing practical guidance and recommendations for developers on parameter tuning.

The inference process of large language models (LLMs) is not just about predicting the next word; the way of sampling from the probability distribution during generation directly affects output quality, diversity, and creativity. Temperature, Top-K, and Top-P are core parameters controlling sampling, but their optimal settings depend on specific tasks and models, and there is a lack of unified guiding principles. This study explores the impact of these parameters on the stability and quality of multilingual LLM outputs through systematic empirical analysis.

Temperature

Controls the sharpness of the probability distribution: low temperature (0.1-0.5) produces deterministic and conservative outputs; high temperature (0.8-1.5) produces diverse and creative outputs; a temperature of 0 means fully greedy decoding.

Top-K Sampling

Considers only the top K words with the highest probabilities: small K (e.g., 10) leads to focused and coherent outputs; large K (e.g.,50) leads to diverse outputs; K=1 is equivalent to greedy decoding. The drawback is that it cannot adapt to different shapes of probability distributions.

Top-P (Nucleus) Sampling

Selects the smallest set of vocabulary where the cumulative probability reaches P: small P (e.g.,0.3) produces conservative outputs; large P (e.g.,0.9) produces diverse outputs; P=1 considers all vocabulary. The advantage is adaptive adjustment of the candidate set.

Multilingual Perspective

Focuses on the sensitivity differences of sampling parameters across different languages (e.g., morphologically rich Russian and German).

Stability Evaluation

Evaluates semantic stability (consistent core meaning), format stability (compliance with expected format), and quality stability (avoiding quality fluctuations) under different parameter configurations.

Optimization Analogy

Analogizes sampling parameter tuning to an optimization problem, explores guiding concepts similar to "learning rate", and establishes a systematic parameter selection framework.

Task Dependence

• Factual Q&A: low temperature for high certainty; creative writing: higher temperature; code generation: balance creativity and grammatical correctness; summarization: conservative settings.

Language Differences

• Resource-rich languages (English, Chinese): aggressive sampling is feasible; low-resource languages: conservative settings; morphologically complex languages: high temperature easily leads to grammatical errors.

Parameter Interaction Effects

There are complex interactions between parameters: high temperature combined with small Top-K has a combined effect; the dynamic nature of Top-P compensates for the impact of Temperature; some combinations may lead to an overly small sampling space and repeated outputs.

Recommended Default Configuration

Temperature: 0.7-0.8; Top-P:0.9-0.95; Top-K:40-50 (or use Top-P alone).

Scenario-Based Tuning

1. High certainty output: Temperature 0.1-0.3 + Top-P=0.1
2. Creative divergence: Temperature1.0-1.2 + Top-P=0.95
3. Structured output (e.g., JSON): low Temperature
4. Dialogue systems: moderate parameters to balance naturalness and consistency

Evaluation Metrics

Diversity metrics (vocabulary overlap), quality metrics (manual/automatic scoring), consistency metrics (stability of multiple results).

Current Limitations

• Experiments are limited to specific model architectures and scales
• Evaluation metrics struggle to fully capture human-perceived quality
• More research is needed on cumulative effects in long text generation

Future Directions

• Adaptive sampling strategies (dynamically adjusting parameters based on context)
• Automatic search for optimal parameter combinations for specific tasks
• Research on sampling strategies in multimodal scenarios
• Reinforcement learning-guided sampling optimization

Sampling parameter tuning is a critical step in large language model applications, directly affecting user experience and output quality. This study provides theoretical basis and practical guidance for parameter selection through empirical analysis. Understanding the working principles of Temperature, Top-K, and Top-P and their impact on model behavior is an essential skill for developers of efficient LLM applications, and fine-grained sampling control will become more important as model capabilities improve.