Impact of Output Format on LLM Performance: Key Findings in Structured NLP Tasks

Recent research indicates that in structured NLP tasks like slot filling and named entity recognition (NER), the choice of output format can cause significant performance fluctuations of 2-46 F1 points for models. This finding reveals that output format, as an easily overlooked key factor, has important reference value for the actual deployment of LLMs, and format optimization should be included in system tuning processes.

In LLM application practices, researchers often focus on model selection, prompt design, and fine-tuning strategies, but overlook the impact of output format. The latest research by a French team systematically reveals that for structured tasks like slot filling and NER, differences in output format can lead to performance fluctuations of up to 46 F1 points, which is of great significance to developers deploying LLMs.

This study uses a rigorous experimental design, covering 4 SLU benchmark tests and 3 NER benchmark tests, evaluating 13 instruction-tuned open-source LLMs, and standardizing prompts and parsers to ensure comparable results. It focuses on comparing three formats:

• JSON format: Structured key-value pairs, easy to parse but with high token overhead
• XML format: Labeled hierarchical structure, good readability but complex to parse
• Inline key-value pairs: Compact text, high token efficiency but low structural clarity The multi-format, multi-model, and multi-dataset evaluation provides a solid data foundation.

The research results show that the choice of output format has a statistically significant impact on model performance, with F1 score fluctuations ranging from 2 to 46 percentage points. Its implications include:

1. Output format should be a key recorded variable in evaluation reports (most current studies do not fully disclose this);
2. The impact of format may be more significant than some model architecture differences, so it should be included in system tuning;
3. Different models have different sensitivities to formats, so targeted selection is needed instead of a one-size-fits-all approach.

The research team proposes a lightweight format selection process that only requires a small amount of development data to determine the optimal format for a specific model-dataset combination. Advantages:

1. Reduces trial-and-error costs, avoiding multiple experiments on the full dataset;
2. Enables rapid deployment, quickly determining configurations before launch;
3. Transferable, applicable to similar model-task combinations. It provides developers with a path from blind trial to evidence-based decision-making.

The research provides key implications for LLM application development:

• Evaluation Transparency: Pay attention to whether the output format is clearly reported; results lacking this information should be treated with caution;
• System Optimization: Include output format in the optimization space; adjusting the format may be more effective than changing the model;
• Task Characteristic Consideration: Although slot filling and NER both belong to information extraction, the optimal format may vary by task;
• Open Source Contribution: The team open-sourced standardized prompts and parsers to help the community unify evaluation benchmarks and reproducible research.

Against the backdrop of rapid iteration of LLM technology, this study reminds us that performance optimization often lies in details. The technical choice of output format reflects the deep mechanism of how models understand, generate, and structure information. It is an important step for developers to understand and apply these findings to improve system performance.