Physics-Informed Neural Networks: Redefining Computational Science

5 min readDec 5, 2024

Physics-Informed Neural Networks (PINNs) represent a transformative approach in computational science, enabling the seamless integration of physical laws into the structure of neural networks. These models leverage partial differential equations (PDEs), conservation laws, or other physical constraints to guide learning, reducing the dependence on labeled data and improving generalization. By incorporating physical laws into the neural network’s training process, PINNs offer an efficient and accurate way to solve forward and inverse problems in various domains such as fluid dynamics, heat transfer, and solid mechanics. PINNs find applications in fluid dynamics, materials science, and quantum mechanics. This article explores PINNs, their mathematical formulation, implementation, and computational aspects.

Mathematical Foundation for PINNs

PINNs integrate physical laws, often expressed as PDEs, into the training objective of a neural network. Given a system governed by a PDE:

where 𝐿 is a differential operator, and 𝑢(𝑥,𝑡) is the solution to the PDE, PINNs aim to approximate this solution using a neural network. The key idea is to minimize a loss function that consists of the following terms:

Residual of the PDE: Ensures that the output of the network satisfies the PDE at each point in the domain.
Boundary/Initial Conditions: Ensures that the solution satisfies the given boundary or initial conditions of the problem.

The loss function L PINN can be written as:

where:
𝐿PDE is the residual of the PDE,
𝐿BC is the residual of boundary conditions,
𝐿IC is the residual of initial conditions,
𝜆1,𝜆2,𝜆3 are hyperparameters controlling the importance of each term.

Working Principle

The core idea of PINNs is to combine the neural network architecture with the governing equations. The network consists of an input layer representing the spatial and temporal variables (e.g., 𝑥 and 𝑡 for a heat conduction problem), hidden layers where the solution is approximated, and an output layer representing the predicted solution of the PDE.

The residuals of the PDE and boundary/initial conditions are calculated using automatic differentiation (AD), which allows for efficient computation of gradients with respect to the neural network parameters. These gradients are used to update the network weights during training using gradient-based optimization techniques (e.g., Adam optimizer).

PINN Algorithm

A simplified algorithm to train a PINN:

Initialize the neural network: Set the number of layers, nodes, and activation functions.
Define the loss function: Include terms for the PDE residual, boundary conditions, and initial conditions.
Sample points: Generate collocation points (spatial and temporal points where the solution is evaluated).
Compute the residuals: Use automatic differentiation to compute the PDE residual at each collocation point.
Minimize the loss function: Use gradient-based optimization to update the network parameters.
Monitor convergence: Track the loss function and the physical accuracy of the solution.

Example: Solving the 1D heat equation

Consider the 1D heat equation:

with the boundary conditions u(0,t)=0u(0,t) = 0u(0,t)=0, u(L, t)=0u(L, t) = 0u(L, t)=0, and initial condition u(x, 0)=sin⁡(πx)u(x, 0) = sin(pi. x)u(x,0)=sin(πx).

Step 1: Define the Neural Network

A simple feedforward neural network is defined with one hidden layer and uses the sine activation function.

Step 2: Loss Function

The total loss function consists of the PDE residual, boundary condition, and initial condition residuals:

Step 3: Initial Implementation

import tensorflow as tf
import numpy as np

# Define the Neural Network Architecture
class PINN(tf.keras.Model):
    def __init__(self, layers):
        super(PINN, self).__init__()
        self.dense1 = tf.keras.layers.Dense(layers[0], activation='tanh')
        self.dense2 = tf.keras.layers.Dense(layers[1], activation='tanh')
        self.dense3 = tf.keras.layers.Dense(1, activation=None)

    def call(self, inputs):
        x, t = inputs
        x = self.dense1(tf.concat([x, t], axis=1))
        x = self.dense2(x)
        return self.dense3(x)

# Define the PDE residual function
def pde_residual(model, x, t):
    with tf.GradientTape(persistent=True) as tape:
        tape.watch(x)
        tape.watch(t)
        u = model([x, t])
    u_t = tape.gradient(u, t)
    u_x = tape.gradient(u, x)
    u_xx = tape.gradient(u_x, x)
    residual = u_t - alpha * u_xx
    return residual

# Boundary and Initial Condition Functions
def boundary_conditions(model, x, t):
    u = model([x, t])
    return u

# Loss Function
def loss(model, x, t, x_b, t_b, x_ic, t_ic):
    # PDE residual
    pde_res = pde_residual(model, x, t)
    pde_loss = tf.reduce_mean(tf.square(pde_res))

    # Boundary Condition residual
    bc_res = boundary_conditions(model, x_b, t_b)
    bc_loss = tf.reduce_mean(tf.square(bc_res))

    # Initial Condition residual
    ic_res = boundary_conditions(model, x_ic, t_ic)
    ic_loss = tf.reduce_mean(tf.square(ic_res - np.sin(np.pi * x_ic)))

    return pde_loss + bc_loss + ic_loss

# Parameters
L = 1.0  # Length of the domain
alpha = 0.01  # Diffusivity
layers = [50, 50]  # Neural Network Architecture
epochs = 10000
learning_rate = 0.001

# Generate training data
x_train = np.linspace(0, L, 100)
t_train = np.linspace(0, 1, 100)

# Create Model
model = PINN(layers)

# Optimizer
optimizer = tf.keras.optimizers.Adam(learning_rate)

# Training Loop
for epoch in range(epochs):
    with tf.GradientTape() as tape:
        # Collocation points for PDE
        x = tf.Variable(np.random.uniform(0, L, (100, 1)), dtype=tf.float32)
        t = tf.Variable(np.random.uniform(0, 1, (100, 1)), dtype=tf.float32)
        # Boundary and initial points
        x_b = tf.Variable(np.array([0, L]), dtype=tf.float32)
        t_b = tf.Variable(np.linspace(0, 1, 100), dtype=tf.float32)
        x_ic = np.linspace(0, L, 100)
        t_ic = np.zeros_like(x_ic)

        # Compute the loss
        total_loss = loss(model, x, t, x_b, t_b, x_ic, t_ic)

    gradients = tape.gradient(total_loss, model.trainable_variables)
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))

    if epoch % 1000 == 0:
        print(f"Epoch {epoch}, Loss: {total_loss.numpy()}")

Step 4: Analysis

Training Performance: During training, the network gradually learns to minimize the loss function by adjusting its weights to satisfy the PDE and boundary conditions.
Accuracy: After sufficient training, the neural network can approximate the solution to the heat equation accurately. The predicted solution can be compared to the analytical solution (if available) or numerical solutions for validation.

Conclusion

Physics-Informed Neural Networks (PINNs) offer a powerful approach to solving complex physical problems governed by PDEs. By embedding the physical laws directly into the training process, PINNs provide an efficient and accurate alternative to traditional numerical methods. The combination of deep learning and physics allows PINNs to handle challenging problems in various fields such as fluid dynamics, material science, and medical imaging, making them an exciting tool in computational science.

My work: https://www.researchgate.net/profile/Bhaumik-Tyagi

Connect: https://www.linkedin.com/in/bhaumik-tyagi-21302a217/