Awesome Loss Functions: A Panoramic Atlas of 350+ Loss Functions and Optimization Guide for Deep Learning

In deep learning model training, loss functions are the "compass" guiding optimization directions and directly impact model performance. The Awesome Loss Functions project systematically compiles over 350 loss functions covering more than 25 domains such as classification, GANs, diffusion models, and reinforcement learning, providing practitioners with a one-stop reference for academic origins, mathematical formulas, and code implementations to solve the problem of difficult loss function selection.

Background

Maintained by AlbEris1, this project aims to address the pain point where developers over-rely on common loss functions (e.g., cross-entropy, mean squared error) while neglecting task-specific optimal choices.

Unique Value

• Comprehensiveness: Includes over 350 loss functions, covering traditional machine learning to cutting-edge deep learning technologies.
• Structured Organization: Classified by more than 25 application domains for easy on-demand retrieval.
• Academic Tracing: Each entry links to the original paper to help understand the design motivation.
• Mathematical and Code Support: Provides mathematical expressions and Python/PyTorch implementation examples.

Classification Tasks

• Cross-entropy loss: Measures differences between probability distributions, including standard, binary, and weighted variants.
• Hinge loss: Core of SVM, maximizes classification margin.
• Focal Loss: Addresses class imbalance, reduces weights of easily classified samples.
• Label smoothing: Regularization technique to prevent overconfidence in models.

GAN

• Original Minimax loss: Based on JS divergence, has gradient vanishing issues.
• Wasserstein loss: Core of WGAN, solves training instability.
• LSGAN loss: Uses least squares instead of logarithmic loss to improve generation quality.

Diffusion Models

• Denoising score matching: Reverses the noise addition process.
• Variational lower bound: Ensures optimization of the log-likelihood lower bound.
• Simplified loss: Proposed by DDPM, with better practical results.

Reinforcement Learning

• Policy gradient loss: Foundation of the REINFORCE algorithm.
• PPO clipping objective: Limits the magnitude of policy updates to improve stability.
• Actor-Critic loss: Combines policy and value functions to reduce variance.

Contrastive Learning

• InfoNCE: Used by MoCo/SimCLR, based on noise contrastive estimation.
• NT-Xent: Adopted by SimCLR, temperature parameter controls distribution smoothness.
• SupCon: Extended to supervised scenarios, uses labels to construct sample pairs.

Multi-task and Special Scenarios

• Multi-task uncertainty weighting: Balances the scale of task losses.
• DTW loss: Specialized for time-series data prediction.
• Huber loss: Highly robust, insensitive to outliers.

Task Matching Principles

• Binary classification: BCE as main, use Focal Loss or weighted BCE for imbalance.
• Multi-class classification: Cross-entropy, use hierarchical softmax for large-scale categories.
• Regression: MSE is sensitive to outliers, MAE is more robust, Huber combines both.
• Generation: GAN uses adversarial loss, diffusion models use denoising loss.

Data Characteristics Considerations

• Class imbalance: Class weighting, Focal Loss, or sampling strategies.
• Noisy labels: Label smoothing, robust loss, or Co-teaching strategies.
• Distribution shift: Domain adaptation loss or adversarial training loss.

Model Characteristics Matching

• Capacity: Large-capacity models need regularization loss, small-capacity models need aggressive loss.
• Output layer: Sigmoid with BCE, softmax with cross-entropy.
• Training phase: Pre-training uses MSE, fine-tuning uses adversarial loss; soft labels initially, hard labels later.

Adaptive Loss Functions

• AutoFocal: Automatically learns Focal Loss focus parameters.
• Adaptive label smoothing: Dynamically adjusts smoothing level.
• Meta-learning loss: Automatically discovers task-specific loss forms.

Multi-modal and Cross-modal Losses

• CLIP loss: Aligns image and text representations.
• InfoNCE multi-modal extension: Constructs sample pairs between modalities.
• Modality fusion loss: Balances contributions of modalities.

Interpretability and Fairness Losses

• Attention-guided loss: Guides models to focus on specific regions.
• Fairness constraint loss: Prevents group bias.
• Causal inference loss: Captures causal relationships instead of correlations.

Usage Guide

1. Retrieval Methods: Browse by task, sort by time, keyword search.
2. Learning Path:
   • Beginners: Start with classic loss functions (MSE, cross-entropy).
   • Intermediate: Dive into task-specific loss functions.
   • Advanced: Follow latest research, try to improve or design new losses.
3. Code Practice: Pay attention to numerical stability, gradient checking, performance optimization, and mixed-precision training.

Conclusion

This project helps developers break through the limitations of loss function selection and improve model performance through appropriate loss functions. As an active open-source project, it will continue to follow cutting-edge developments and provide a comprehensive reference for the community.