Skip to content
Projects

Distributed arrays

Download notebookView on GitHubRun when the site was built.

A DistributedArray is a global array of which each rank stores only the block it owns, plus an optional frame of halo cells. You create one with a function named like its NumPy counterpart:

Show code
import numpy as np
import mpiarray as mpa
comm = mpa.default_comm()
rank, nranks = comm.Get_rank(), comm.Get_size()
import mpiarray as mpa
a = mpa.array([1, 2, 3, 4]) # mpiexec -n 2: rank 0 holds [1, 2], rank 1 holds [3, 4]
b = mpa.zeros((64, 48))
c = mpa.ones((64, 48), dtype=np.float32)
d = mpa.full((64, 48), 3.5)
e = mpa.empty((64, 48)) # uninitialised
f = mpa.arange(100)
g = mpa.linspace(0.0, 1.0, 101)
h = mpa.fromfunction(lambda i, j: np.sin(0.1 * i) * j, (64, 48))

Without an MPI launcher (python script.py), the same code runs as one rank holding everything, without importing mpi4py.

  • mpa.array(data) takes the global array, the same on every rank, and keeps the block each rank owns. Every rank briefly holds all of it, so use it for small arrays and tests. With MPIARRAY_DEBUG=1 the ranks check that they really passed the same data.
  • mpa.arange, mpa.linspace and mpa.fromfunction compute only the local block on each rank, so no rank ever holds the whole array. fromfunction gets the global indices of the block, as numpy.fromfunction does.
  • mpa.from_local(block) builds the array from the pieces the ranks already hold, stacked in rank order along split (default: the first axis). Pieces of any length are redistributed to the near-even split with one Alltoallv; with layout= the pieces must already match it and are used as they are (with_halos=True for storage with halos).
  • mpa.zeros / mpa.empty, then writing a.local, lets each rank fill its block from its own data.
  • mpa.load(path) reads a .npy file, each rank reading only its block, and mpa.load_hdf5(path) the datasets of an HDF5 file (see below).

zeros_like, ones_like, full_like and empty_like create an array with the layout of another one. Given a distributed array, mpa.array(a) copies it with its layout, and mpa.array(a, halo=2) or mpa.array(a, split=None) changes only the options given and redistributes it; mpa.asarray(a) returns a itself unless an option changes. a.redistribute(layout) moves an array to any other layout of the same shape, uneven ones included (see Changing the layout).

Every creation function takes the same keyword arguments:

ArgumentDefaultMeaning
split0the axis or axes split over the ranks; None: every rank holds everything
halo0halo width, for every axis or one per axis
periodicFalsewhether each axis wraps around, for every axis or one per axis
commCOMM_WORLDthe communicator
process_gridautomaticexplicit ranks per axis, instead of split
layout—an existing Layout, instead of all of the above
p = mpa.zeros((64, 48, 3), split=(0, 1), halo=(2, 2, 0), periodic=(True, False, False))

splits a vector field over its two spatial axes, with two halo layers in space and none along the component axis. Each axis is cut near-evenly: when the length does not divide, the first ranks get one element more.

A split axis needs at least as many elements as ranks, and a halo must fit in the smallest block along its axis; otherwise creating the array raises a ValueError (on every rank), instead of leaving ranks without data. Use split=None for small arrays that every rank should hold whole.

Arrays created with the same arguments have equal layouts and combine without communication. To combine arrays, create the second one with zeros_like(first) or with layout=first.layout.

Each rank stores one NumPy (or CuPy) array: its block plus halo[axis] cells on both sides of every axis. For one axis with two halo cells:

index in storage: 0 1 | 2 3 4 5 6 7 | 8 9
halo | block | halo
(left | global indices | (right
neighbour) start .. end-1 neighbour)
AttributeContents
a.localthe block, a writable view
a.local_with_halosthe whole storage, a writable view
a.shape, a.sizethe global shape and number of elements
a.layout.index_boundsthis rank’s (start, end) range along each axis
a.layout.local_shapethe shape of the block
a.layout.storage_shapethe shape of the storage

Writing a block from local data:

a = mpa.zeros((64, 48))
(x0, x1), (y0, y1) = a.layout.index_bounds
def my_function(i, j):
return np.sin(i / 10) + np.cos(j / 10)
a.local[...] = my_function(np.arange(x0, x1)[:, None], np.arange(y0, y1))

After writing the block, call update_halos() before reading the halo cells.

full = a.gather() # the global array on every rank (NumPy or CuPy)
full = a.gather(root=0) # on rank 0 only; None on the other ranks
full = a.to_numpy() # the same, as numpy.ndarray
value = a.get((3, 2)) # one element, the same on every rank
value = a[3, 2] # the same as get
part = a[:10, 5] # a new distributed array of the selected cells

All of these are collective: every rank must call them, even when only rank 0 wants the result. Use gather for output, plotting and tests, not inside a time loop on large grids. Halo cells are never included.

Printing does not communicate: repr(a) shows the layout and this rank’s block only, so if rank == 0: print(a) is safe. Print a.gather() (on every rank) to see the whole array.

copy() returns an independent array with the same layout, and astype(dtype) a converted one.

import tempfile
from pathlib import Path
workdir = comm.bcast(tempfile.mkdtemp() if rank == 0 else None, root=0)
field_path = str(Path(workdir) / "field.npy")
mpa.save(field_path, a) # an ordinary .npy file, written in parallel
b = mpa.load(
field_path, split=1, halo=2
) # any layout; the shape and dtype from the file

With several ranks both use MPI-IO: rank 0 writes the header, and every rank writes or reads only its own block, so nothing global is ever built. The files are plain NumPy files: numpy.load reads what mpa.save writes, and mpa.load reads what numpy.save writes. The file must be on a file system that all ranks see. Both calls are collective.

With h5py installed (pip install "mpiarray[hdf5]"), several arrays and attributes go into one HDF5 file:

rho, phi = a, a * 2
t, step = 0.5, 10
state_path = str(Path(workdir) / "state.h5")
mpa.save_hdf5(state_path, {"rho": rho, "phi": phi}, attrs={"time": t, "step": step})
arrays, attrs = mpa.load_hdf5(state_path, split=(0, 1), halo=1)
rho = arrays["rho"]
comm.Barrier()
if rank == 0:
import shutil
shutil.rmtree(workdir)

With an MPI-enabled h5py build (h5py.get_config().mpi), all ranks write and read their own blocks of one file at once. With an ordinary build save_hdf5 gathers each array on rank 0, which writes the file, so it must fit in rank 0’s memory; load_hdf5 lets every rank read only its block in both cases. The datasets are ordinary HDF5 datasets of the global shape, without halo cells.

The following short examples bring storage, halos and reductions together.

import numpy as np
import mpiarray as mpa
a = mpa.array([1, 2, 3, 4])
a # printing shows this rank's block and never communicates
DistributedArray(shape=(4,), dtype=int64, split=(0,), rank 0 of 4 holds [0:1]: [1])

Arrays are split along their first axis by default; split chooses other axes, and split=None gives every rank the whole array. mpa.process_grid shows how the ranks are shared for any rank count, without MPI:

for size in (4, 6, 12, 7):
print(size, "ranks ->", mpa.process_grid(size, 2, (0, 1)))
4 ranks -> (2, 2)
6 ranks -> (2, 3)
12 ranks -> (3, 4)
7 ranks -> (1, 7)

A prime number of ranks cannot be split over two axes, so it all goes to the last one.

Every array has a layout that tells each rank its place: the process grid, its neighbours and the index range it owns.

b = mpa.zeros((64, 48), split=(0, 1), periodic=(True, False))
layout = b.layout
print("process grid:", layout.process_grid)
print("neighbours (left, right) per axis:", layout.neighbours)
print("owned index ranges:", layout.index_bounds)
process grid: (2, 2)
neighbours (left, right) per axis: ((2, 2), (-2, 1))
owned index ranges: ((0, 32), (0, 24))

On one rank, the periodic axis has this rank as its own neighbour, and the wall axis has MPI.PROC_NULL on both sides.

With halo=1, each rank stores its block with one extra cell on each side. local is the block, local_with_halos the whole storage:

u = mpa.arange(8.0, halo=1, periodic=True)
print("block: ", u.local)
print("with halos: ", u.local_with_halos)
block: [0. 1.]
with halos: [0. 0. 1. 0.]

update_halos() copies the neighbours’ boundary values into the halo cells, as a finite-difference stencil needs. On this periodic axis they wrap around:

u.update_halos()
print(u.local_with_halos)
# a centred difference only needs the local block with its halos
v = u.local_with_halos
print("second difference:", v[:-2] - 2 * v[1:-1] + v[2:])
[7. 0. 1. 2.]
second difference: [8. 0.]

accumulate_halos() goes the other way: values written into halo cells, for example charge deposited by particles near the edge of a block, are added into the neighbour’s block, and the halo cells are zeroed:

rho = mpa.zeros(8, halo=1, periodic=True)
rho.local_with_halos[0] = 1.0 # deposited left of the first cell
rho.local_with_halos[-1] = 2.0 # deposited right of the last one
rho.accumulate_halos()
print(rho.local_with_halos)
[0. 2. 1. 0.]

Reductions are collective over the ranks, skip the halo cells and return the same host scalar on every rank. Arithmetic and NumPy ufuncs return new distributed arrays with the same layout:

x = mpa.linspace(0.0, 1.0, 8)
y = 2 * x + np.sin(x)
print("sum:", y.sum(), " max:", y.max(), " norm:", y.norm())
print("element 3:", y[3])
print("gathered on every rank:", y.gather())
sum: 11.633144897399504 max: 2.8414709848078967 norm: 4.887094305532312
element 3: 1.272714712135909
gathered on every rank: [0. 0.42808602 0.85327142 1.27271471 1.68369136 2.08364933
2.47026108 2.84147098]

Save the cells above as a script and run it with mpiexec -n 4 python tutorial.py: each rank then holds its own block, update_halos and accumulate_halos exchange messages with the neighbour ranks, and sum, norm, y[3] and gather communicate, so every rank must call them. On the CuPy backend (CUNUMPY_BACKEND=cupy), call xp.mpi.set_mpi_cuda_aware(...) or xp.mpi.mpi_is_cuda_aware(comm) once first.