Physics-Informed Neural Networks (PINNs)

The Core Idea

Traditional neural networks learn purely from data — given input-output pairs, they adjust weights to minimise a prediction error. But in science, we possess far more than data alone: we have centuries of accumulated knowledge encoded in partial differential equations (PDEs), conservation laws, and symmetry principles. Physics-Informed Neural Networks (PINNs) exploit this knowledge by incorporating physical constraints directly into the loss function used during training.

The result is a framework that can solve forward problems (finding the solution of a PDE), inverse problems (inferring unknown physical parameters from observations), and data-assimilation problems (combining sparse measurements with known physics) — all within a single, unified neural network architecture. PINNs represent a paradigm shift: the network doesn't merely fit data; it learns to respect the laws of nature.

The key insight is remarkably simple. Modern deep learning frameworks provide automatic differentiation (AD), which computes exact derivatives of the network output with respect to its inputs. Since PDEs are statements about derivatives, we can evaluate how well the network's output satisfies a PDE at any point in the domain — without any mesh or discretisation — and penalise violations in the loss function.

Why PINNs Matter for Science

Setting Up the Problem

Consider a physical system governed by a PDE of the general form:

$$\mathcal{F}\bigl[u(\mathbf{x}, t)\bigr] = 0, \quad \mathbf{x} \in \Omega, \; t \in [0, T]$$

where $\mathcal{F}$ is a differential operator (possibly nonlinear), $u(\mathbf{x}, t)$ is the unknown solution field, and $\Omega$ is the spatial domain. The system is subject to boundary conditions on $\partial\Omega$ and initial conditions at $t = 0$.

We approximate $u$ by a neural network $u_\text{NN}(\mathbf{x}, t; \boldsymbol{\theta})$ with parameters $\boldsymbol{\theta}$ (weights and biases). The network takes the coordinates $(\mathbf{x}, t)$ as input and outputs the predicted field value.

The Data Loss

Given $N_d$ measurements $\{(\mathbf{x}_i, t_i, u_i)\}_{i=1}^{N_d}$, the data fidelity term penalises mismatch between the network predictions and observations:

$$\mathcal{L}_\text{data}(\boldsymbol{\theta}) = \frac{1}{N_d}\sum_{i=1}^{N_d} \bigl|u_\text{NN}(\mathbf{x}_i, t_i; \boldsymbol{\theta}) - u_i\bigr|^2$$

This is the standard mean squared error (MSE) loss familiar from supervised learning. On its own, it would produce a generic regression model with no knowledge of physics.

The Physics Loss (PDE Residual)

The defining innovation of PINNs is the physics loss. We select $N_r$ collocation points $\{(\mathbf{x}_j, t_j)\}_{j=1}^{N_r}$ in the domain and evaluate the PDE residual at each:

$$r_j = \mathcal{F}\bigl[u_\text{NN}(\mathbf{x}_j, t_j; \boldsymbol{\theta})\bigr]$$

The physics loss penalises non-zero residuals:

$$\mathcal{L}_\text{phys}(\boldsymbol{\theta}) = \frac{1}{N_r}\sum_{j=1}^{N_r} \bigl|\mathcal{F}\bigl[u_\text{NN}(\mathbf{x}_j, t_j; \boldsymbol{\theta})\bigr]\bigr|^2$$

Crucially, the derivatives inside $\mathcal{F}$ (e.g., $\partial u/\partial t$, $\partial^2 u/\partial x^2$) are computed via automatic differentiation through the network. No finite differences or meshes are needed.

Boundary and Initial Condition Losses

Boundary conditions (BCs) and initial conditions (ICs) are enforced through additional loss terms. For $N_b$ boundary points and $N_0$ initial-condition points:

$$\mathcal{L}_\text{BC} = \frac{1}{N_b}\sum_{k=1}^{N_b}\bigl|u_\text{NN}(\mathbf{x}_k, t_k; \boldsymbol{\theta}) - g(\mathbf{x}_k, t_k)\bigr|^2$$

$$\mathcal{L}_\text{IC} = \frac{1}{N_0}\sum_{m=1}^{N_0}\bigl|u_\text{NN}(\mathbf{x}_m, 0; \boldsymbol{\theta}) - u_0(\mathbf{x}_m)\bigr|^2$$

where $g$ specifies the boundary data and $u_0$ the initial profile.

The Total PINN Loss

The total loss is a weighted sum of all components:

$$\boxed{\mathcal{L}(\boldsymbol{\theta}) = \mathcal{L}_\text{data} + \lambda_\text{phys}\,\mathcal{L}_\text{phys} + \lambda_\text{BC}\,\mathcal{L}_\text{BC} + \lambda_\text{IC}\,\mathcal{L}_\text{IC}}$$

The hyperparameters $\lambda_\text{phys}$, $\lambda_\text{BC}$, $\lambda_\text{IC}$ control the relative importance of each constraint. Choosing these weights is a significant practical challenge — too small and the physics is ignored; too large and the optimiser focuses on the PDE residual at the expense of fitting data.

Gradient computation: The parameters $\boldsymbol{\theta}$ are updated by gradient descent. The gradient $\nabla_{\boldsymbol{\theta}}\mathcal{L}$ involves second-order automatic differentiation: first with respect to the inputs $(\mathbf{x}, t)$ to form the PDE residual, then with respect to the parameters $\boldsymbol{\theta}$ to obtain the training gradients. This double differentiation is the computational backbone of PINNs.

Problem Setup

As a concrete example, consider the one-dimensional heat equation on $x \in [0, L]$, $t \in [0, T]$:

$$\frac{\partial u}{\partial t} = \alpha \frac{\partial^2 u}{\partial x^2}$$

with thermal diffusivity $\alpha > 0$, boundary conditions $u(0, t) = u(L, t) = 0$, and initial condition $u(x, 0) = u_0(x)$ (e.g., $u_0(x) = \sin(\pi x / L)$).

The analytical solution for this initial condition is:

$$u_\text{exact}(x, t) = \sin\!\left(\frac{\pi x}{L}\right) e^{-\alpha (\pi/L)^2 t}$$

Network Architecture

We define a feedforward neural network with input $(x, t) \in \mathbb{R}^2$ and output $\hat{u} \in \mathbb{R}$:

$$\mathbf{h}^{(0)} = \begin{pmatrix} x \\ t \end{pmatrix}, \quad \mathbf{h}^{(l)} = \sigma\!\left(W^{(l)}\mathbf{h}^{(l-1)} + \mathbf{b}^{(l)}\right), \quad \hat{u} = W^{(L)}\mathbf{h}^{(L-1)} + b^{(L)}$$

where $\sigma$ is a smooth activation function (typically $\tanh$, since we need second derivatives to be well-defined). ReLU is generally avoided in PINNs because its second derivative is zero almost everywhere.

Computing the PDE Residual

Using automatic differentiation, we compute the required derivatives of the network output:

$$\frac{\partial \hat{u}}{\partial t} = \frac{\partial \hat{u}}{\partial \mathbf{h}^{(L-1)}} \cdot \frac{\partial \mathbf{h}^{(L-1)}}{\partial \mathbf{h}^{(L-2)}} \cdots \frac{\partial \mathbf{h}^{(1)}}{\partial \mathbf{h}^{(0)}} \cdot \frac{\partial \mathbf{h}^{(0)}}{\partial t}$$

Similarly for $\partial \hat{u}/\partial x$. The second spatial derivative $\partial^2 \hat{u}/\partial x^2$ is obtained by differentiating the first derivative again. The PDE residual at any collocation point $(x_j, t_j)$ is:

$$r(x_j, t_j) = \frac{\partial \hat{u}}{\partial t}\bigg|_{(x_j, t_j)} - \alpha \frac{\partial^2 \hat{u}}{\partial x^2}\bigg|_{(x_j, t_j)}$$

If the network perfectly satisfies the heat equation, $r = 0$ everywhere.

Backpropagation Through the Residual

For a single hidden layer with $\tanh$ activation, $\hat{u}(x, t) = \mathbf{w}_2^T \tanh(w_{1x}\, x + w_{1t}\, t + \mathbf{b}_1) + b_2$, the derivatives with respect to the inputs are:

$$\frac{\partial \hat{u}}{\partial t} = \mathbf{w}_2^T \cdot \text{diag}\!\left(\text{sech}^2(w_{1x}\, x + w_{1t}\, t + \mathbf{b}_1)\right) \cdot \mathbf{w}_{1t}$$

$$\frac{\partial \hat{u}}{\partial x} = \mathbf{w}_2^T \cdot \text{diag}\!\left(\text{sech}^2(\mathbf{z})\right) \cdot \mathbf{w}_{1x}$$

where $\mathbf{z} = w_{1x}\, x + w_{1t}\, t + \mathbf{b}_1$. For the second derivative:

$$\frac{\partial^2 \hat{u}}{\partial x^2} = \mathbf{w}_2^T \cdot \text{diag}\!\left(-2\,\text{sech}^2(\mathbf{z})\tanh(\mathbf{z})\right) \cdot \mathbf{w}_{1x}^{\circ 2}$$

where $\mathbf{w}_{1x}^{\circ 2}$ denotes element-wise squaring. The training gradient $\partial \mathcal{L}_\text{phys}/\partial \boldsymbol{\theta}$ is then computed by differentiating the residual with respect to the network weights, giving a chain of derivatives: input derivatives compose with parameter derivatives through the same computational graph.

The Inverse Problem Framework

One of the most powerful applications of PINNs is solving inverse problems: given observed data and a known form of the governing PDE, identify unknown physical parameters. Suppose the heat equation has an unknown diffusivity $\alpha$:

$$\frac{\partial u}{\partial t} = \alpha \frac{\partial^2 u}{\partial x^2}, \quad \alpha = \;?$$

We promote $\alpha$ from a fixed constant to a trainable parameter, optimised alongside the network weights $\boldsymbol{\theta}$.

Joint Optimisation

The extended parameter set is $\boldsymbol{\Theta} = \{\boldsymbol{\theta}, \alpha\}$. The total loss remains:

$$\mathcal{L}(\boldsymbol{\Theta}) = \mathcal{L}_\text{data}(\boldsymbol{\theta}) + \lambda\,\mathcal{L}_\text{phys}(\boldsymbol{\theta}, \alpha)$$

The gradient with respect to $\alpha$ involves differentiating the physics loss:

$$\frac{\partial \mathcal{L}_\text{phys}}{\partial \alpha} = \frac{2}{N_r}\sum_{j=1}^{N_r} r_j \cdot \frac{\partial r_j}{\partial \alpha} = \frac{2}{N_r}\sum_{j=1}^{N_r} r_j \cdot \left(-\frac{\partial^2 \hat{u}}{\partial x^2}\bigg|_j\right)$$

since $r_j = \partial \hat{u}/\partial t - \alpha\, \partial^2 \hat{u}/\partial x^2$ and therefore $\partial r_j/\partial \alpha = -\partial^2 \hat{u}/\partial x^2$.

The update rule for $\alpha$ follows standard gradient descent:

$$\alpha^{(n+1)} = \alpha^{(n)} - \eta \frac{\partial \mathcal{L}}{\partial \alpha}$$

Identifiability and Data Requirements

For the inverse problem to be well-posed, the data must contain sufficient information to constrain $\alpha$. Intuitively, measurements at multiple times are essential because they reveal the rate of diffusion. A snapshot at a single time point only constrains the product $\alpha \cdot t$, not $\alpha$ alone.

In practice, PINNs for inverse problems have been used to identify viscosity in fluid flows, reaction rates in chemical kinetics, and material stiffness in solid mechanics — often from surprisingly sparse datasets, thanks to the regularising effect of the physics loss.

Example: Identifying the Diffusion Coefficient

Suppose we observe the temperature field at $N_d = 50$ randomly chosen points$(x_i, t_i)$ from the true solution $u_\text{exact}(x,t) = \sin(\pi x)\,e^{-\alpha_\text{true}\pi^2 t}$ with $\alpha_\text{true} = 0.1$. We initialise the PINN with a guess $\alpha^{(0)} = 0.5$ and train jointly.

After training, the network simultaneously learns the solution field $u(x,t)$ and converges to $\alpha \approx 0.1$, recovering the true diffusion coefficient. The physics loss provides a gradient signal for $\alpha$ even at locations where no data exists, effectively propagating information from measurements throughout the entire domain.

The Problem with Soft Constraints

Standard PINNs enforce physics as soft constraints via the loss function. However, the PDE residual is never exactly zero — only approximately minimised. For conservation laws (energy, momentum, symplecticity), even small violations can accumulate over long time integrations, leading to unphysical behaviour such as energy drift.

Hard constraint approaches build conservation laws directly into the network architecture, guaranteeing their satisfaction by construction rather than by optimisation.

Hamiltonian Neural Networks (HNNs)

Consider a classical mechanical system with generalised coordinates $\mathbf{q}$ and momenta $\mathbf{p}$. Hamilton's equations state:

$$\frac{d\mathbf{q}}{dt} = \frac{\partial H}{\partial \mathbf{p}}, \qquad \frac{d\mathbf{p}}{dt} = -\frac{\partial H}{\partial \mathbf{q}}$$

where $H(\mathbf{q}, \mathbf{p})$ is the Hamiltonian (total energy). A Hamiltonian Neural Network (HNN) learns $H_{\boldsymbol{\theta}}(\mathbf{q}, \mathbf{p})$ as a neural network, then derives the equations of motion from it:

$$\frac{d\mathbf{q}}{dt}\bigg|_\text{pred} = \frac{\partial H_{\boldsymbol{\theta}}}{\partial \mathbf{p}}, \qquad \frac{d\mathbf{p}}{dt}\bigg|_\text{pred} = -\frac{\partial H_{\boldsymbol{\theta}}}{\partial \mathbf{q}}$$

The loss function compares these predicted time derivatives with observed data $(\dot{\mathbf{q}}_i, \dot{\mathbf{p}}_i)$:

$$\mathcal{L}_\text{HNN} = \sum_i \left\|\frac{\partial H_{\boldsymbol{\theta}}}{\partial \mathbf{p}}\bigg|_i - \dot{\mathbf{q}}_i\right\|^2 + \left\|\frac{\partial H_{\boldsymbol{\theta}}}{\partial \mathbf{q}}\bigg|_i + \dot{\mathbf{p}}_i\right\|^2$$

Why HNNs Preserve Energy Exactly

The key theorem is that any dynamics derived from a Hamiltonian via Hamilton's equations automatically conserves the Hamiltonian. The proof is straightforward:

$$\frac{dH}{dt} = \frac{\partial H}{\partial \mathbf{q}} \cdot \frac{d\mathbf{q}}{dt} + \frac{\partial H}{\partial \mathbf{p}} \cdot \frac{d\mathbf{p}}{dt}$$

Substituting Hamilton's equations:

$$\frac{dH}{dt} = \frac{\partial H}{\partial \mathbf{q}} \cdot \frac{\partial H}{\partial \mathbf{p}} + \frac{\partial H}{\partial \mathbf{p}} \cdot \left(-\frac{\partial H}{\partial \mathbf{q}}\right) = 0$$

This cancellation holds for any function $H$, including a neural network $H_{\boldsymbol{\theta}}$. Therefore, by construction, the learned dynamics preserves the learned energy function exactly — regardless of how well $H_{\boldsymbol{\theta}}$ approximates the true Hamiltonian. This is a structural guarantee, not an optimisation outcome.

Lagrangian Neural Networks (LNNs)

An alternative formulation uses the Lagrangian $L(\mathbf{q}, \dot{\mathbf{q}}) = T - V$ instead. A Lagrangian Neural Network learns $L_{\boldsymbol{\theta}}(\mathbf{q}, \dot{\mathbf{q}})$ and derives equations of motion via the Euler-Lagrange equation:

$$\frac{d}{dt}\frac{\partial L_{\boldsymbol{\theta}}}{\partial \dot{\mathbf{q}}} = \frac{\partial L_{\boldsymbol{\theta}}}{\partial \mathbf{q}}$$

Expanding the total time derivative on the left:

$$\frac{\partial^2 L_{\boldsymbol{\theta}}}{\partial \dot{\mathbf{q}}^2}\ddot{\mathbf{q}} + \frac{\partial^2 L_{\boldsymbol{\theta}}}{\partial \mathbf{q}\,\partial \dot{\mathbf{q}}}\dot{\mathbf{q}} = \frac{\partial L_{\boldsymbol{\theta}}}{\partial \mathbf{q}}$$

Solving for the acceleration: $\ddot{\mathbf{q}} = \left(\frac{\partial^2 L_{\boldsymbol{\theta}}}{\partial \dot{\mathbf{q}}^2}\right)^{-1}\left(\frac{\partial L_{\boldsymbol{\theta}}}{\partial \mathbf{q}} - \frac{\partial^2 L_{\boldsymbol{\theta}}}{\partial \mathbf{q}\,\partial \dot{\mathbf{q}}}\dot{\mathbf{q}}\right)$. LNNs have the advantage of working directly in generalised coordinates without requiring canonical momenta, and they naturally handle constraints via the Lagrangian formalism.

Fluid Dynamics: Navier-Stokes Equations

PINNs have been applied extensively to the incompressible Navier-Stokes equations:

$$\nabla \cdot \mathbf{v} = 0, \qquad \frac{\partial \mathbf{v}}{\partial t} + (\mathbf{v}\cdot\nabla)\mathbf{v} = -\frac{1}{\rho}\nabla p + \nu\nabla^2\mathbf{v}$$

Applications include reconstructing velocity and pressure fields from sparse particle image velocimetry (PIV) data, identifying viscosity from flow measurements, and solving forward problems in complex geometries (e.g., blood flow in patient-specific arterial geometries). The divergence-free condition $\nabla \cdot \mathbf{v} = 0$ can be enforced either as a soft constraint (additional loss term) or as a hard constraint by deriving the velocity from a stream function.

Quantum Mechanics: Schrödinger Equation

The time-dependent Schrödinger equation for a wave function $\psi(x, t)$:

$$i\hbar\frac{\partial \psi}{\partial t} = -\frac{\hbar^2}{2m}\frac{\partial^2 \psi}{\partial x^2} + V(x)\psi$$

PINNs can solve for $\psi$ by splitting into real and imaginary parts, each represented by a separate network output. The physics loss enforces both the real and imaginary components of the Schrödinger equation. This approach has been used to solve eigenvalue problems, time-dependent scattering, and multi-particle systems in reduced dimensions.

Climate Modelling

Climate models involve coupled PDEs for fluid dynamics, thermodynamics, and radiative transfer. PINNs can serve as fast surrogate models for parameterised sub-grid processes (turbulence, cloud formation, ocean mixing) while respecting conservation of energy and mass.

Recent work has applied physics-informed approaches to downscaling global climate model outputs to regional scales, assimilating satellite observations into weather forecasts, and emulating computationally expensive components of Earth system models. The physics constraints are critical here to prevent the surrogate from producing meteorologically impossible states.

Medical Imaging

In medical imaging, PINNs have been used for diffusion MRI (modelling water diffusion in brain tissue via the Bloch-Torrey equation), cardiac mechanics (enforcing incompressibility and equilibrium equations on heart wall deformations), and pharmacokinetic modelling (fitting compartmental ODE models to dynamic contrast-enhanced MRI data). The physics constraints improve image reconstruction quality from undersampled data, enabling faster scan times.

Origins and Key Milestones