Neural Networks and Differential Equations: A New Paradigm for Deep Learning from Infinite Layers to Continuous Modeling

This article introduces the core content of the IJCAI 2026 tutorial, explores the profound connection between neural networks and differential equations, reveals the evolutionary path of deep learning from discrete layer stacking to continuous dynamic system modeling, covers key directions such as Neural ODEs, Physics-Informed Neural Networks (PINNs), and continuous normalizing flows, and presents a new paradigm of integration between mathematics and machine learning.

Traditional deep neural networks rely on discrete layer stacking, which has limitations such as empirical depth selection, high cost of hyperparameter tuning, and poor training stability. The 2018 NeurIPS paper "Neural Ordinary Differential Equations" proposed that the layer update of Residual Networks (ResNet) can be regarded as Euler discretization of ODEs, and when the number of layers approaches infinity, it tends to a continuous dynamic system, opening up a new research direction.

The core of Neural ODEs is to parameterize the ODE derivative function using a neural network: $\frac{dh(t)}{dt} = f(h(t), t, \theta)$. Forward propagation is completed via an adaptive ODE solver. Advantages include improved memory efficiency, adaptive precision trade-offs, and efficient backpropagation via the adjoint sensitivity method. Subsequent extensions include: Stochastic Differential Equation Networks (SDE-Net) for quantifying uncertainty; Partial Differential Equation Networks (PDE-Net) for handling spatiotemporal coupling problems; and fractional differential equation networks for modeling long-range dependencies.

Physics-Informed Neural Networks (PINNs) embed physical laws as soft constraints into the loss function (data fitting + physical constraints), enabling them to learn physically consistent solutions even when data is sparse, and significantly improving the efficiency of inverse problem solving. Continuous Normalizing Flows (CNF) transform data distributions into simple prior distributions via continuous ODEs, and FFJORD avoids Jacobian computation through trace estimation, supporting high-dimensional data modeling.

Neural differential equation networks are applied in multiple domains: scientific computing accelerates traditional numerical methods such as climate simulation and molecular dynamics; time series prediction is suitable for irregular sampling scenarios (medical monitoring, financial analysis); Graph Neural ODEs model dynamic graphs and spatiotemporal graphs; and the continuous dynamic system perspective in reinforcement learning improves training stability.

The current field faces challenges: computational efficiency (unpredictable overhead from adaptive step sizes of ODE solvers); theoretical understanding (basic issues such as expressive power and generalization need to be further explored); scalability (modeling high-dimensional data); and integration with Transformers (exploring continuous variants). These directions provide innovative opportunities for future research.

The intersection of neural networks and differential equations represents the trend of mathematization in machine learning, establishing a more rigorous theoretical framework and developing efficient models. Practitioners need to improve their literacy in dynamic system analysis and differential equation solving, while researchers face unsolved mysteries and innovative opportunities. The IJCAI 2026 tutorial provides a valuable entry point for learning in this field, and deep learning is undergoing a paradigm revolution from discrete to continuous.