Skip to content
Projects

Parallel runs

ElectrostaticSimulation, BoundaryIntegralElectrodes and FlatQuadtree take a comm= argument, an MPI communicator. With it the ranks share the work and every rank gets the same answer as a serial run (to round-off). Without it, or under one process, nothing changes.

Install mpi4py next to an MPI library (Open MPI or MPICH). tree-d-3d imports it through maybempi, which falls back to a serial stand-in when the script is not started by mpirun, so the same script runs either way.

Each rank passes its own particles. Diagnostics such as get_total_energy are sums over every rank, so every rank must call them (they are collective):

import numpy as np
from maybempi import MPI, collectives
from tree_d_3d import ChargedBody, ElectrostaticSimulation, Quad
comm = MPI.COMM_WORLD
rng = np.random.default_rng(1)
positions = rng.uniform(-0.9, 0.9, (2, 400_000))
charges = rng.choice([-1.0, 1.0], 400_000)
mine = collectives.block(400_000, comm) # this rank's share
bodies = [
ChargedBody(*positions[:, i], charge=charges[i])
for i in range(mine.start, mine.stop)
]
sim = ElectrostaticSimulation(
bodies, Quad(0, 0, 1), k_e=1e-4, dt=1e-3, softening=1e-3, kernel="log", comm=comm
)
for _ in range(10):
sim.step()
energy = sim.get_total_energy() # on every rank
if comm.Get_rank() == 0:
print(energy)
Terminal window
mpirun -n 4 python run.py

Particles stay on the rank that created them; absorbed ones leave it. Give the ranks similar numbers of particles for an even load.

Here every rank passes the same conductors, free charges and probe points; the ranks share the probes, the right-hand side, the preconditioner blocks and the GMRES products (or the rows of the dense matrix), and gather the results:

electrodes = BoundaryIntegralElectrodes(conductors, k_e=1.0, comm=comm)
electrodes.solve(sources, charges)
phi, field = electrodes.probe(points) # the whole result on every rank

The sources are replicated and the targets distributed. Every rank gathers all the particles (MPI_Allgatherv) and builds the same tree. It then evaluates the field only at its own particles, from the whole tree. The cells’ multipole moments, the FMM’s M2L and near-field pairs and the panel solver’s sums are split evenly over the ranks, and are completed by an allreduce or a gather.

This suits a few to tens of ranks on one node or a few nodes. Each rank still holds every particle and builds the whole tree, so memory does not shrink with the ranks and the build and the gather cap the speed-up (Amdahl). A distributed tree, with particles migrating between subdomains as in a PIC code, would be the next step beyond that.

Strong scaling, one thread per rank, on one 4-core machine (mean of 2 runs; the ranks share its memory bandwidth):

Workload1 rank2 ranks4 ranksSpeed-up at 4
3 steps, 400k line charges, FMM7.63 s4.75 s3.03 s2.5x
3 steps, 400k point charges, tree14.6 s7.79 s4.14 s3.5x
3 steps, 400k line charges, tree44.1 s24.6 s13.2 s3.3x
100k probes, 2.7k panels (LU)0.31 s0.17 s0.092 s3.4x
100k probes, 21k panels0.99 s0.65 s0.34 s2.9x
GMRES set-up, 21k panels0.59 s0.37 s0.27 s2.2x
GMRES solve, 21k panels8.12 s5.37 s3.68 s2.2x
LU set-up, 2.7k panels0.66 s0.62 s0.51 s1.3x

The LU factorisation stays on every rank, so only the matrix assembly is shared.