Automatic Classification of Customer Service Tickets: A Practical NLP Comparison from Zero-Shot to Fine-Tune

This project systematically compares three large language model application paradigms—Zero-Shot classification (BART-large-mnli), Few-Shot learning (Gemini-2.5-Flash), and Fine-Tuned adjustment (DistilBERT)—for the task of automatic customer service ticket classification, addressing the pain points of traditional manual classification being time-consuming and error-prone. The final Fine-Tuned model achieves a classification accuracy of 98.5%, covering the entire process from data preparation, model training to production deployment, and provides best practice recommendations for method selection.

Modern enterprise customer service handles a large number of free-text tickets every day; manual classification is time-consuming, labor-intensive, and error-prone. The core challenge of automatic ticket classification lies in the diverse language styles of customers (colloquialism, spelling errors, abbreviations of professional terms, etc.), requiring the model to accurately understand and categorize them into the correct business labels.

• Dataset Source: Uses Hugging Face's Bitext Customer Support Dataset (real customer service conversation records).
• Data Balance: Handles class imbalance through undersampling technology to ensure equal sample counts across all categories.
• Preprocessing Flow: LabelEncoder encodes labels, DistilBERT AutoTokenizer tokenizes (max length 128), converts to PyTorch tensors.

• Zero-Shot Classification: Uses BART-large-mnli, no labeled data needed, classifies based on natural language inference framework, accuracy 30.00%, poor domain adaptability.
• Few-Shot Learning: Leverages Gemini-2.5-Flash's contextual learning ability, builds prompt templates via LangChain, depends on example quality and context window.
• Fine-Tuned Adjustment: Based on DistilBERT-base-uncased, hyperparameters include learning rate 2e-5, 5 epochs, early stopping mechanism (patience=2), etc., fully learns domain features.

• Loss Curve: Training loss and validation loss decrease synchronously and converge, no obvious overfitting.
• Performance Comparison:

  Method	Accuracy	Characteristics
  Zero-Shot (BART)	30.00%	No labeled data needed, poor domain adaptability
  Few-Shot (Gemini)	Medium	Depends on example quality, context-limited
  Fine-Tuned (DistilBERT)	98.50%	Domain-adapted, production-ready

• Interpretability: The model can recognize professional terms, handle colloquial expressions, and understand contextual differences.

• Model Hosting: Due to model size constraints, uses Google Drive for hosting.
• Inference Deployment: Quickly implements classification via Hugging Face pipeline (code example see main text).
• Tech Stack: PyTorch, Hugging Face Transformers, Datasets, LangChain, Google GenAI, Scikit-learn, Matplotlib, etc.

• Method Selection: Use Zero-Shot for quick prototyping, Few-Shot for resource-constrained scenarios, Fine-Tune for production-level tasks.
• Data Quality: Investment in data preprocessing can improve model performance; issues like class imbalance need to be addressed.
• Model Monitoring: After deployment, need to detect performance drift, discover new categories, and retrain regularly.

This project demonstrates the full process of customer service ticket classification from data preparation to deployment. The comparison of three methods shows that Fine-Tuning is the production-level performance gold standard for domain-specific NLP tasks. The project provides a validated technical route and code implementation, which can serve as a reference benchmark for similar tasks.