Distributed arrays
A DistributedArray is a global array of which each rank stores only the block it owns,
plus an optional frame of halo cells. You create one with a function named like its NumPy
counterpart:
Show code
import numpy as np
import mpiarray as mpa
comm = mpa.default_comm()rank, nranks = comm.Get_rank(), comm.Get_size()import mpiarray as mpa
a = mpa.array([1, 2, 3, 4]) # mpiexec -n 2: rank 0 holds [1, 2], rank 1 holds [3, 4]b = mpa.zeros((64, 48))c = mpa.ones((64, 48), dtype=np.float32)d = mpa.full((64, 48), 3.5)e = mpa.empty((64, 48)) # uninitialisedf = mpa.arange(100)g = mpa.linspace(0.0, 1.0, 101)h = mpa.fromfunction(lambda i, j: np.sin(0.1 * i) * j, (64, 48))Without an MPI launcher (python script.py), the same code runs as one rank holding
everything, without importing mpi4py.
Where the data comes from
Section titled “Where the data comes from”mpa.array(data)takes the global array, the same on every rank, and keeps the block each rank owns. Every rank briefly holds all of it, so use it for small arrays and tests. WithMPIARRAY_DEBUG=1the ranks check that they really passed the same data.mpa.arange,mpa.linspaceandmpa.fromfunctioncompute only the local block on each rank, so no rank ever holds the whole array.fromfunctiongets the global indices of the block, asnumpy.fromfunctiondoes.mpa.from_local(block)builds the array from the pieces the ranks already hold, stacked in rank order alongsplit(default: the first axis). Pieces of any length are redistributed to the near-even split with oneAlltoallv; withlayout=the pieces must already match it and are used as they are (with_halos=Truefor storage with halos).mpa.zeros/mpa.empty, then writinga.local, lets each rank fill its block from its own data.mpa.load(path)reads a.npyfile, each rank reading only its block, andmpa.load_hdf5(path)the datasets of an HDF5 file (see below).
zeros_like, ones_like, full_like and empty_like create an array with the layout of
another one. Given a distributed array, mpa.array(a) copies it with its layout, and
mpa.array(a, halo=2) or mpa.array(a, split=None) changes only the options given and
redistributes it; mpa.asarray(a) returns a itself unless an option changes.
a.redistribute(layout) moves an array to any other layout of the same shape, uneven ones
included (see Changing the layout).
How the array is split
Section titled “How the array is split”Every creation function takes the same keyword arguments:
| Argument | Default | Meaning |
|---|---|---|
split | 0 | the axis or axes split over the ranks; None: every rank holds everything |
halo | 0 | halo width, for every axis or one per axis |
periodic | False | whether each axis wraps around, for every axis or one per axis |
comm | COMM_WORLD | the communicator |
process_grid | automatic | explicit ranks per axis, instead of split |
layout | — | an existing Layout, instead of all of the above |
p = mpa.zeros((64, 48, 3), split=(0, 1), halo=(2, 2, 0), periodic=(True, False, False))splits a vector field over its two spatial axes, with two halo layers in space and none along the component axis. Each axis is cut near-evenly: when the length does not divide, the first ranks get one element more.
A split axis needs at least as many elements as ranks, and a halo must fit in the
smallest block along its axis; otherwise creating the array raises a ValueError (on
every rank), instead of leaving ranks without data. Use split=None for small arrays that
every rank should hold whole.
Arrays created with the same arguments have equal layouts and combine without
communication. To combine arrays, create the second one with zeros_like(first) or with
layout=first.layout.
Local storage
Section titled “Local storage”Each rank stores one NumPy (or CuPy) array: its block plus halo[axis] cells on both sides
of every axis. For one axis with two halo cells:
index in storage: 0 1 | 2 3 4 5 6 7 | 8 9 halo | block | halo (left | global indices | (right neighbour) start .. end-1 neighbour)| Attribute | Contents |
|---|---|
a.local | the block, a writable view |
a.local_with_halos | the whole storage, a writable view |
a.shape, a.size | the global shape and number of elements |
a.layout.index_bounds | this rank’s (start, end) range along each axis |
a.layout.local_shape | the shape of the block |
a.layout.storage_shape | the shape of the storage |
Writing a block from local data:
a = mpa.zeros((64, 48))(x0, x1), (y0, y1) = a.layout.index_bounds
def my_function(i, j): return np.sin(i / 10) + np.cos(j / 10)
a.local[...] = my_function(np.arange(x0, x1)[:, None], np.arange(y0, y1))After writing the block, call update_halos() before
reading the halo cells.
Getting the data back
Section titled “Getting the data back”full = a.gather() # the global array on every rank (NumPy or CuPy)full = a.gather(root=0) # on rank 0 only; None on the other ranksfull = a.to_numpy() # the same, as numpy.ndarrayvalue = a.get((3, 2)) # one element, the same on every rankvalue = a[3, 2] # the same as getpart = a[:10, 5] # a new distributed array of the selected cellsAll of these are collective: every rank must call them, even when only rank 0 wants
the result. Use gather for output, plotting and tests, not inside a time loop on large
grids. Halo cells are never included.
Printing does not communicate: repr(a) shows the layout and this rank’s block only, so
if rank == 0: print(a) is safe. Print a.gather() (on every rank) to see the whole array.
copy() returns an independent array with the same layout, and astype(dtype) a converted
one.
Saving to files
Section titled “Saving to files”import tempfilefrom pathlib import Path
workdir = comm.bcast(tempfile.mkdtemp() if rank == 0 else None, root=0)field_path = str(Path(workdir) / "field.npy")mpa.save(field_path, a) # an ordinary .npy file, written in parallelb = mpa.load( field_path, split=1, halo=2) # any layout; the shape and dtype from the fileWith several ranks both use MPI-IO: rank 0 writes the header, and every rank writes or
reads only its own block, so nothing global is ever built. The files are plain NumPy files:
numpy.load reads what mpa.save writes, and mpa.load reads what numpy.save writes.
The file must be on a file system that all ranks see. Both calls are collective.
With h5py installed (pip install "mpiarray[hdf5]"), several arrays and attributes go into
one HDF5 file:
rho, phi = a, a * 2t, step = 0.5, 10state_path = str(Path(workdir) / "state.h5")mpa.save_hdf5(state_path, {"rho": rho, "phi": phi}, attrs={"time": t, "step": step})arrays, attrs = mpa.load_hdf5(state_path, split=(0, 1), halo=1)rho = arrays["rho"]comm.Barrier()if rank == 0: import shutil
shutil.rmtree(workdir)With an MPI-enabled h5py build (h5py.get_config().mpi), all ranks write and read their
own blocks of one file at once. With an ordinary build save_hdf5 gathers each array on
rank 0, which writes the file, so it must fit in rank 0’s memory; load_hdf5 lets every
rank read only its block in both cases. The datasets are ordinary HDF5 datasets of the
global shape, without halo cells.
A worked tour
Section titled “A worked tour”The following short examples bring storage, halos and reductions together.
import numpy as np
import mpiarray as mpa
a = mpa.array([1, 2, 3, 4])a # printing shows this rank's block and never communicatesDistributedArray(shape=(4,), dtype=int64, split=(0,), rank 0 of 4 holds [0:1]: [1])Splitting an array
Section titled “Splitting an array”Arrays are split along their first axis by default; split chooses other axes, and split=None gives every rank the whole array. mpa.process_grid shows how the ranks are shared for any rank count, without MPI:
for size in (4, 6, 12, 7): print(size, "ranks ->", mpa.process_grid(size, 2, (0, 1)))4 ranks -> (2, 2)6 ranks -> (2, 3)12 ranks -> (3, 4)7 ranks -> (1, 7)A prime number of ranks cannot be split over two axes, so it all goes to the last one.
Every array has a layout that tells each rank its place: the process grid, its neighbours and the index range it owns.
b = mpa.zeros((64, 48), split=(0, 1), periodic=(True, False))layout = b.layoutprint("process grid:", layout.process_grid)print("neighbours (left, right) per axis:", layout.neighbours)print("owned index ranges:", layout.index_bounds)process grid: (2, 2)neighbours (left, right) per axis: ((2, 2), (-2, 1))owned index ranges: ((0, 32), (0, 24))On one rank, the periodic axis has this rank as its own neighbour, and the wall axis has MPI.PROC_NULL on both sides.
Halo cells
Section titled “Halo cells”With halo=1, each rank stores its block with one extra cell on each side. local is the block, local_with_halos the whole storage:
u = mpa.arange(8.0, halo=1, periodic=True)print("block: ", u.local)print("with halos: ", u.local_with_halos)block: [0. 1.]with halos: [0. 0. 1. 0.]update_halos() copies the neighbours’ boundary values into the halo cells, as a finite-difference stencil needs. On this periodic axis they wrap around:
u.update_halos()print(u.local_with_halos)
# a centred difference only needs the local block with its halosv = u.local_with_halosprint("second difference:", v[:-2] - 2 * v[1:-1] + v[2:])[7. 0. 1. 2.]second difference: [8. 0.]accumulate_halos() goes the other way: values written into halo cells, for example charge deposited by particles near the edge of a block, are added into the neighbour’s block, and the halo cells are zeroed:
rho = mpa.zeros(8, halo=1, periodic=True)rho.local_with_halos[0] = 1.0 # deposited left of the first cellrho.local_with_halos[-1] = 2.0 # deposited right of the last onerho.accumulate_halos()print(rho.local_with_halos)[0. 2. 1. 0.]Reductions and operators
Section titled “Reductions and operators”Reductions are collective over the ranks, skip the halo cells and return the same host scalar on every rank. Arithmetic and NumPy ufuncs return new distributed arrays with the same layout:
x = mpa.linspace(0.0, 1.0, 8)y = 2 * x + np.sin(x)print("sum:", y.sum(), " max:", y.max(), " norm:", y.norm())print("element 3:", y[3])print("gathered on every rank:", y.gather())sum: 11.633144897399504 max: 2.8414709848078967 norm: 4.887094305532312element 3: 1.272714712135909gathered on every rank: [0. 0.42808602 0.85327142 1.27271471 1.68369136 2.08364933 2.47026108 2.84147098]On several ranks
Section titled “On several ranks”Save the cells above as a script and run it with mpiexec -n 4 python tutorial.py: each rank then holds its own block, update_halos and accumulate_halos exchange messages with the neighbour ranks, and sum, norm, y[3] and gather communicate, so every rank must call them. On the CuPy backend (CUNUMPY_BACKEND=cupy), call xp.mpi.set_mpi_cuda_aware(...) or xp.mpi.mpi_is_cuda_aware(comm) once first.