Skip to content
Projects

Sparse stencil operators

Download notebookView on GitHubRun when the site was built.

mpiarray.StencilOperator represents a linear operator given by a small, fixed set of neighbour offsets — the pattern a finite-difference stencil has — without ever storing a sparse matrix: apply halo-exchanges and sums the shifted terms directly on a DistributedArray, and to_petsc_mat assembles the identical pattern as a PETSc Mat for use with PETSc’s Krylov and multigrid solvers.

Show code
import mpiarray as mpa
comm = mpa.default_comm()
rank, nranks = comm.Get_rank(), comm.Get_size()

The heat equation guide applies a five-point Laplacian by hand, with one line of slicing per neighbour. StencilOperator is the same idea made general: give it the offsets and coefficients once, and it works for any pattern and any number of axes. This guide uses a periodic grid, so every row has all five neighbours and the examples below have an exact, easily checked answer.

u = mpa.zeros((64, 48), split=(0, 1), halo=1, periodic=True)
u.local[...] = 1.0 # a constant field: the Laplacian is zero everywhere
laplacian = mpa.StencilOperator(
u.layout,
offsets=[(0, 0), (-1, 0), (1, 0), (0, -1), (0, 1)],
coefficients=[4.0, -1.0, -1.0, -1.0, -1.0],
)
lu = laplacian.apply(u) # update_halos(), then the shifted sum
lu.sum() # collective: exactly 0.0
np.float64(0.0)

apply checks that u’s layout matches the operator’s, forwards boundary= to update_halos (needed for the non-periodic walls the heat equation guide has), and raises if layout.halo is too narrow for the offsets used. A coefficient can be a single number, as above, or an array of layout.local_shape for an operator whose weight varies by grid point.

A handful of rows — Dirichlet rows, cut cells near an embedded boundary — can replace the regular stencil entirely, by global index:

pinned = mpa.StencilOperator(
u.layout,
offsets=[(0, 0), (-1, 0), (1, 0), (0, -1), (0, 1)],
coefficients=[4.0, -1.0, -1.0, -1.0, -1.0],
overrides={(0, 0): [((0, 0), 1.0)]}, # row (0, 0): identity instead of the stencil
)
pinned.apply(u).get((0, 0)) # u's own value, 1.0, not the Laplacian's 0.0
np.float64(1.0)

Each rank resolves only the overrides it owns, through DistributedArray.local_index; an index outside this rank’s block is skipped, so the mapping can be built once, from global indices, and handed to every rank unchanged.

Operators of the same layout combine with +, -, unary - and multiplication by a number, merging offsets (and overridden rows) term by term — useful for building an operator such as a shifted Laplacian, or a Jacobian that is the sum of several physical terms, out of smaller pieces:

mass = mpa.StencilOperator(u.layout, offsets=[(0, 0)], coefficients=[0.1])
shifted = laplacian + mass # (laplacian + 0.1 * identity)(u)
lu = laplacian.apply(u)
(shifted.apply(u) - lu).max() # 0.1, since u is the constant field 1.0
np.float64(0.09999999999999964)

Each side keeps its own coefficients (arrays included) and overrides; a row overridden on only one side is combined with the other side’s regular stencil at that row, resolved on whichever rank owns it.

The same pattern, without petsc4py:

Show code
from importlib.util import find_spec
petsc_available = find_spec("petsc4py") is not None
if not petsc_available:
print("Install petsc4py to run the matrix example.")
Install petsc4py to run the matrix example.
if petsc_available:
matrix = laplacian.to_petsc_mat()
vec = u.to_petsc()
result = matrix.createVecLeft()
matrix.mult(vec, result)
phi = mpa.from_petsc(result, u.layout)
print(phi.sum()) # the same 0.0 as laplacian.apply(u) above
matrix.destroy()

to_petsc_mat uses Layout.dmda for the row decomposition and mpa.petsc_numbering for the column numbers, exactly as in the PETSc guide’s hand-assembled example. Along a non-periodic axis an offset that steps outside the grid is simply omitted (an implicit Dirichlet zero); give that row its own overrides entry for any other boundary condition.