The universal approximation property of neural networks states that a sufficiently large neural network with an appropriate activation function can approximate any continuous function to an arbitrary degree of accuracy on a compact domain. This property has significant implications for solving stochastic partial differential equations (SPDEs), as it suggests that neural networks can be trained to approximate the solutions to these equations under certain conditions. Extending the universal approximation theorem to SPDEs involves demonstrating that neural networks can approximate not only deterministic functions but also mappings that involve stochastic elements, capturing both the deterministic and random behavior of solutions.

1. Universal Approximation Property: Basics

• Definition: The universal approximation theorem states that for any continuous function $f: \mathbb{R}^n \rightarrow \mathbb{R}$ and any $\epsilon > 0$, there exists a feedforward neural network with a finite number of neurons and a non-linear activation function (e.g., sigmoid, ReLU) that approximates $f$ such that:

$|f(x) - u_\theta(x)| < \epsilon \quad \text{for all } x \text{ in a compact set},$

where $u_\theta(x)$ is the output of the neural network with parameters $\theta$ .

2. Extension to SPDEs

To apply the universal approximation property to SPDEs, we need to extend the concept from deterministic functions to stochastic processes and random fields. An SPDE typically takes the form:

$\frac{\partial u(t, x)}{\partial t} = \mathcal{L}u(t, x) + \sigma(u(t, x)) \dot{W}(t, x),$

where:

• $\mathcal{L}$ is a differential operator.
• $\dot{W}(t, x)$ represents a stochastic noise term.
• $\sigma(u)$ is a coefficient function modulating the noise.

3. Universal Approximation for Stochastic Processes

To prove that neural networks can approximate the solution of an SPDE, consider the following steps:

• Approximation of Deterministic Components: Neural networks with sufficient depth and width can approximate the deterministic components of the SPDE solution, such as $\mathcal{L}u(t, x)$ .
• Approximation of Stochastic Processes: The noise term $\dot{W}(t, x)$ can be represented by neural networks that model stochastic processes, such as Gaussian processes or white noise approximations.

4. Key Challenges

• Noise Representation: The neural network must approximate functions that incorporate randomness. One approach is to train the network using random samples to simulate noise, ensuring it learns the behavior of stochastic terms.
• Function Space and Regularity: The solution $u(t, x)$ of an SPDE often lives in a function space with specific regularity properties (e.g., $L^2$ , Sobolev spaces). The universal approximation result needs to be shown for mappings that may be continuous in these function spaces rather than just pointwise.

5. Approximation Result for SPDEs

The universal approximation property for SPDEs can be expressed as follows:

• Theorem (Informal): Let $u(t, x)$ be a solution to an SPDE that is continuous in $t$ and $x$ for almost every realization of the noise. For any $\epsilon > 0$, there exists a neural network $u_\theta(t, x, \omega)$ such that for all $t$ and $x$ in a compact set and for almost every realization $\omega$ of the stochastic process:

$\mathbb{E}\left[\sup_{t, x} |u(t, x, \omega) - u_\theta(t, x, \omega)|^2\right] < \epsilon.$

This indicates that neural networks can approximate the SPDE's solution within a small error bound in the mean square sense.

6. Training Neural Networks for SPDEs

• Loss Function Design: The loss function should include terms for both the PDE residual and a stochastic component. This could take the form:

$\mathcal{L}{\text{SPDE}} = \mathbb{E}\left[\left|\frac{\partial u\theta(t, x)}{\partial t} - \mathcal{L}u_\theta(t, x) - \sigma(u_\theta(t, x)) \dot{W}(t, x)\right|^2\right],$

ensuring that the network learns to minimize the difference between the predicted and actual dynamics.