When Pointwise Metrics Fail: A New Protocol for Evaluating Multimodal Inverse Problems

This article addresses the misleading issue of traditional pointwise metrics (e.g., Mean Squared Error, MSE) in the evaluation of multimodal inverse problems and proposes a more reliable evaluation protocol. Using di-lepton top quark neutrino reconstruction as a benchmark task, the study compares the performance of various generative models including regression transformers, discrete normalizing flows, and continuous normalizing flows. Key findings: Pointwise metrics tend to favor point-estimation models, while generative models better capture the true multimodal distribution, providing critical guidance for machine learning model selection in particle physics.

Multimodal inverse problems are common in particle physics—for example, neutrino escape during top quark reconstruction leads to an underdetermined system, and the true posterior distribution is multimodal. Traditional regression methods output a single-point estimate, which is physically incomplete. Pointwise metrics (e.g., MSE) penalize all predictions deviating from the 'correct' answer, ignoring the fact that there are multiple reasonable solutions in multimodal cases, leading to misjudgment of model quality.

The di-lepton tt̄ decay is chosen as the benchmark task (two neutrinos escape, leading to inherent multiple solutions in the system). Delphes simulation data released by Raine et al. (including MadGraph event generation and detector simulation) is used, and the training-test split follows the upstream release to ensure result comparability.

Four types of models are compared: 1. Pure MSE regression transformer (point estimation, cannot capture multimodality); 2. MSE + MMD combined loss (hybrid method, encourages distribution learning); 3. Discrete normalizing flow (nu2flows, optimized for Lorentz covariance); 4. Continuous normalizing flow (CFM, cutting-edge flow model with stable training and efficient sampling).

Pointwise metrics (e.g., MSE) systematically favor point-estimation models, making them appear 'better' but masking their inability to capture multimodal structures; point-estimation models may achieve artificially high scores by memorizing training statistical features, and overfitting is hard to detect in the distribution space; good generative models should cover all multimodal solutions but are penalized by pointwise metrics.

A multidimensional evaluation framework is constructed: 1. Posterior quality assessment (visualization of single-event posterior distribution + statistical distribution matching); 2. Physical consistency check (ensuring energy-momentum conservation); 3. Uncertainty quantification (evaluating the correlation between predicted uncertainty and true error); 4. Computational efficiency comparison (sampling speed).

Experiments confirm: Pure MSE regression performs best on pointwise metrics but cannot capture multimodality; normalizing flow methods have slightly worse MSE but their posterior distributions align better with physical intuition. Implications: Model selection needs to consider the nature of the task (generative models are needed for inverse problems with multiple solutions); physical constraints are a necessary dimension of evaluation, and predictions violating conservation laws have no practical value.

Open-source contributions: The codebase uses uv for dependency management and Hydra for configuration; notebooks support self-contained synthetic experiments and chart generation to ensure reproducibility. Future directions: Extend the protocol to complex decay topologies, explore robustness under systematic uncertainties, and develop efficient sampling algorithms to meet real-time application needs.