MPI support
Install for this page:
pip install "scope-profiler[mpi]", plus an MPI implementation such as Open MPI or MPICH. Parallel HDF5 output additionally requires an MPI-enabled build of h5py; serial-HDF5 merging does not.
scope-profiler is MPI-aware out of the box. When the process is launched
under an MPI launcher and mpi4py is installed, the profiler
automatically detects MPI.COMM_WORLD and handles per-rank data
collection and merging.
The notebook kernel itself is a single process, so the MPI examples on this
page are written to a scratch directory and launched as a subprocess with
mpiexec -n 2 (or -n 4); the cells then show what that run printed and
recorded. Where mpiexec or mpi4py is missing, those cells print a note
instead of running.
Installation
Section titled “Installation”pip install "scope-profiler[mpi]"How it works
Section titled “How it works”-
Launcher detection — before anything MPI-related happens, scope-profiler checks whether this process was started by
mpirun/mpiexec/srunor an equivalent launcher, by looking for the per-rank environment variables those launchers export (OMPI_COMM_WORLD_RANK,PMI_RANK,PMIX_RANK, …). If none is present,mpi4pyis never imported and no MPI call is ever made — a plainpython script.pyrun pays nothing for MPI support, not evenMPI_Init. The one exception is an application that already importedmpi4pyand initialized MPI itself; then the existing communicator is used. -
Setup —
ProfilingConfigreadsCOMM_WORLDfor rank and size. That is a local query, not a collective:setup()issues no MPI call of its own and creates no temporary directory. Nothing happens before then: importing scope-profiler touches MPI not at all, so a process that merely imports the library — a child forked from a rank, say — never joins the job. -
Recording — each rank accumulates its own timestamps in memory. Nothing is written until finalization.
-
Finalize — every rank writes its own data into the same output file; timestamp arrays are never sent to rank 0. With an MPI-enabled h5py build, ordinary region data is written through parallel HDF5. Otherwise a tiny point-to-point ownership token moves through the ranks: each rank opens the serial-HDF5 file, appends its indexed event slice, closes it, then passes the token. The last rank reports completion to rank 0, which atomically publishes the file under its final name.
A rank that entered no region contributes no rank/region index row. And because every rank must reach
finalize(), one that dies first leaves the job waiting rather than quietly dropping its data from the output.LIKWID records use the token writer even when parallel HDF5 is installed. LIKWID reads counters in a subprocess, and Open MPI does not safely support a collective MPI-IO phase after that fork. The direct writer deliberately uses point-to-point traffic only. Line-profiler records are supported by both writers.
All ranks must see
file_paththrough the same shared filesystem. Select a backend explicitly withProfileManager.setup(output_mode="direct")oroutput_mode="parallel"; the defaultautochooses the safe option.
Example
Section titled “Example”The code is identical to the serial case — no MPI-specific API calls are needed:
%%writefile mpi_example.pyfrom scope_profiler import ProfileManager
ProfileManager.setup()
@ProfileManager.profile("compute")def compute(): s = 0 for i in range(100_000): s += i return s
compute()ProfileManager.finalize()Writing mpi_example.pyRun with MPI:
mpirun -n 4 python mpi_example.pyrun_mpi(4, sys.executable, "mpi_example.py")Show the summary printed by rank 0
exit status 1--------------------------------------------------------------------------There are not enough slots available in the system to satisfy the 4slots that were requested by the application:
/home/runner/work/scope-profiler/scope-profiler/.venv/bin/python
Either request fewer slots for your application, or make more slotsavailable for use.
A "slot" is the Open MPI term for an allocatable unit where we canlaunch a process. The number of slots available are defined by theenvironment in which Open MPI processes are run:
1. Hostfile, via "slots=N" clauses (N defaults to number of processor cores if not provided) 2. The --host command line parameter, via a ":N" suffix on the hostname (N defaults to 1 if not provided) 3. Resource manager (e.g., SLURM, PBS/Torque, LSF, etc.) 4. If none of a hostfile, the --host command line parameter, or an RM is present, Open MPI defaults to the number of processor cores
In all the above cases, if you want Open MPI to default to the numberof hardware threads instead of the number of processor cores, use the--use-hwthread-cpus option.
Alternatively, you can use the --oversubscribe option to ignore thenumber of available slots when deciding the number of processes tolaunch.--------------------------------------------------------------------------The output profiling_data.h5 is one file for the whole job, holding the data
of every rank, in shared event tables indexed by rank (there is no separate
group per rank). Reading it back shows that all four ranks recorded
compute:
from scope_profiler import read_h5
if (WORKDIR / "profiling_data.h5").exists(): results = read_h5("profiling_data.h5") print(results.num_ranks, "ranks:", list(results["compute"].ranks))Profiling MPI calls in Python
Section titled “Profiling MPI calls in Python”The ordinary MPI support above profiles work on each rank. Enable MPI-call profiling when communication operations themselves should appear on the timeline.
For an unchanged script, launch it through the regular profiler command:
mpirun -n 4 scope-profiler run --mpi-calls simulation.pyMPI-call profiling is opt-in. With --mpi-calls, the runner waits for the
target to import mpi4py, then temporarily replaces mpi4py’s COMM_WORLD and
COMM_SELF globals with transparent profiling proxies and restores them at
shutdown. Communicators returned by common constructors such as Split,
Dup, Create_cart, and Create_intercomm remain wrapped automatically.
Communicators obtained through an unrecognized third-party API can still be
passed to profile_mpi_comm() explicitly. --no-mpi-calls is accepted when
an explicit config file enables the feature and the command should override it.
For a normal profiling session, enable the same behavior through
ProfilingOptions and obtain the communicator inside the session:
%%writefile options_example.pyfrom mpi4py import MPIfrom scope_profiler import ProfileManager, ProfilingOptions
options = ProfilingOptions(profile_mpi_calls=True)with ProfileManager.session(options=options, file_path="mpi-calls.h5", verbose=False): comm = MPI.COMM_WORLD comm.barrier()Writing options_example.pyrun_mpi(2, sys.executable, "options_example.py")inspect_mpi_regions("mpi-calls.h5") ╭──────────────────────────────────────────────────────────────────────────╮ │ region n │ ├──────────────────────────────────────────────────────────────────────────┤ │ mpi:barrier kind=collective bytes=0 peer=-1 root=-1 tag=-1 comm=0 2 │ ╰──────────────────────────────────────────────────────────────────────────╯(The page’s snippet is shown with verbose=False here only so that the run
prints nothing but the table below it.) The setting is also accepted in TOML:
[profiling]profile_mpi_calls = trueTo wrap an individual communicator explicitly instead:
%%writefile explicit_example.pyfrom mpi4py import MPIfrom scope_profiler import ProfileManager, profile_mpi_comm
comm = profile_mpi_comm(MPI.COMM_WORLD)
with ProfileManager.session(file_path="mpi-calls-explicit.h5", verbose=False): if comm.rank == 0: request = comm.isend({"work": 42}, dest=1, tag=7) request.wait() elif comm.rank == 1: payload = comm.recv(source=0, tag=7)
total = comm.allreduce(comm.rank) comm.barrier()Writing explicit_example.pyrun_mpi(2, sys.executable, "explicit_example.py")inspect_mpi_regions("mpi-calls-explicit.h5") ╭──────────────────────────────────────────────────────────────────────────────╮ │ region n │ ├──────────────────────────────────────────────────────────────────────────────┤ │ mpi:allreduce kind=collective bytes=-1 peer=-1 root=-1 tag=-1 comm=0 2 │ │ mpi:barrier kind=collective bytes=0 peer=-1 root=-1 tag=-1 comm=0 2 │ │ mpi:isend kind=point-to-point bytes=-1 peer=1 root=-1 tag=7 comm=0 1 │ │ mpi:recv kind=point-to-point bytes=-1 peer=0 root=-1 tag=7 comm=0 1 │ │ mpi:wait kind=wait bytes=-1 peer=1 root=-1 tag=7 comm=0 request=isend 1 │ ╰──────────────────────────────────────────────────────────────────────────────╯ProfiledMPIComm is a proxy: attributes and unwrapped methods continue to
come from the original communicator. It explicitly wraps the object-mode
send, recv, isend, irecv, bcast, reduce, allreduce, and
barrier methods, along with their uppercase buffer-mode equivalents.
Nonblocking calls return ProfiledMPIRequest; its wait() / Wait() region
retains the initiating operation’s bytes, peer, tag, and communicator.
Buffer-mode message sizes are exact when mpi4py supplies a datatype. For
object-mode values without a buffer interface, the default is bytes=-1 so
profiling never serializes an object twice or repeats its __reduce__ side
effects. Pass profile_mpi_comm(comm, pickle_object_sizes=True) to record a
protocol-5 pickle-size estimate when that extra serialization is acceptable.
A high-level receive cannot know the incoming serialized size before it
blocks, so it also records bytes=-1; use Recv/Irecv with an explicit
buffer when an exact receive capacity is important.
The module does not import mpi4py. Passing the already-created communicator is what preserves scope-profiler’s serial-process guarantee: importing scope-profiler alone never initializes MPI.
Complete runnable Python examples
Section titled “Complete runnable Python examples”The cells below execute every example with two MPI ranks. The source of each
is shown straight from the examples/ directory of the repository, so it is
the file the commands actually run rather than a copy maintained separately in
this page. The tables list the recorded MPI regions with their call counts;
the same scope-profiler inspect command also takes --columns region calls total avg to add the timings, which vary from run to run.
Explicit communicator wrapper
Section titled “Explicit communicator wrapper”This version opts in by wrapping one communicator directly.
"""Profile mpi4py communication operations and their message metadata.
Run with at least two ranks::
mpirun -n 2 python examples/ex_mpi_wrappers.py"""
from mpi4py import MPI
from scope_profiler import ProfileManager, profile_mpi_comm
def main() -> None: comm = profile_mpi_comm(MPI.COMM_WORLD)
with ProfileManager.session(file_path="mpi_calls.h5", verbose=False): if comm.rank == 0: request = comm.isend({"work": 42}, dest=1, tag=7) request.wait() elif comm.rank == 1: comm.recv(source=0, tag=7)
comm.allreduce(comm.rank) comm.barrier()
if __name__ == "__main__": main()Run it and inspect the recorded MPI regions:
mpirun -n 2 python examples/ex_mpi_wrappers.pyscope-profiler inspect mpi_calls.h5 --regions-only --include '^mpi:' \ --columns region calls total avgOutput:
if wrappers: run_mpi(2, sys.executable, wrappers) inspect_mpi_regions("mpi_calls.h5") ╭──────────────────────────────────────────────────────────────────────────────╮ │ region n │ ├──────────────────────────────────────────────────────────────────────────────┤ │ mpi:allreduce kind=collective bytes=-1 peer=-1 root=-1 tag=-1 comm=0 2 │ │ mpi:barrier kind=collective bytes=0 peer=-1 root=-1 tag=-1 comm=0 2 │ │ mpi:isend kind=point-to-point bytes=-1 peer=1 root=-1 tag=7 comm=0 1 │ │ mpi:recv kind=point-to-point bytes=-1 peer=0 root=-1 tag=7 comm=0 1 │ │ mpi:wait kind=wait bytes=-1 peer=1 root=-1 tag=7 comm=0 request=isend 1 │ ╰──────────────────────────────────────────────────────────────────────────────╯Unmodified NumPy/mpi4py program
Section titled “Unmodified NumPy/mpi4py program”This file contains no scope-profiler imports. Running it with
scope-profiler run --mpi-calls enables MPI-call instrumentation first.
"""Profile exact-size NumPy buffer operations with mpi4py.
Run with at least two ranks:
mpirun -n 2 scope-profiler run -q \ -o mpi_numpy_calls.h5 examples/ex_mpi_numpy_wrappers.py"""
import numpy as npfrom mpi4py import MPI
def main() -> None: comm = MPI.COMM_WORLD if comm.size < 2: raise RuntimeError("this example requires at least two MPI ranks")
if comm.rank == 0: values = np.arange(8, dtype=np.float64) comm.Isend([values, MPI.DOUBLE], dest=1, tag=17).Wait() elif comm.rank == 1: received = np.empty(8, dtype=np.float64) comm.Irecv([received, MPI.DOUBLE], source=0, tag=17).Wait() np.testing.assert_array_equal(received, np.arange(8, dtype=np.float64))
broadcast_value = np.array([42 if comm.rank == 0 else 0], dtype=np.int32) comm.Bcast([broadcast_value, MPI.INT], root=0) assert broadcast_value[0] == 42
local = np.array([comm.rank + 1], dtype=np.int32) total = np.zeros(1, dtype=np.int32) comm.Allreduce([local, MPI.INT], [total, MPI.INT], op=MPI.SUM) assert total[0] == comm.size * (comm.size + 1) // 2
comm.Barrier()
if __name__ == "__main__": main()Run it and inspect the recorded MPI regions:
mpirun -n 2 scope-profiler run -q --mpi-calls \ -o mpi_numpy_calls.h5 examples/ex_mpi_numpy_wrappers.pyscope-profiler inspect mpi_numpy_calls.h5 --regions-only --include '^mpi:' \ --columns region calls total avgOutput:
if numpy_wrappers: run_mpi( 2, sys.executable, "-m", "scope_profiler", "run", "-q", "--mpi-calls", "-o", "mpi_numpy_calls.h5", numpy_wrappers, ) inspect_mpi_regions("mpi_numpy_calls.h5") ╭───────────────────────────────────────────────────────────────────────────────╮ │ region n │ ├───────────────────────────────────────────────────────────────────────────────┤ │ mpi:allreduce kind=collective bytes=4 peer=-1 root=-1 tag=-1 comm=0 2 │ │ mpi:barrier kind=collective bytes=0 peer=-1 root=-1 tag=-1 comm=0 2 │ │ mpi:bcast kind=collective bytes=4 peer=-1 root=0 tag=-1 comm=0 2 │ │ mpi:irecv kind=point-to-point bytes=64 peer=0 root=-1 tag=17 comm=0 1 │ │ mpi:isend kind=point-to-point bytes=64 peer=1 root=-1 tag=17 comm=0 1 │ │ mpi:wait kind=wait bytes=64 peer=0 root=-1 tag=17 comm=0 request=irecv 1 │ │ mpi:wait kind=wait bytes=64 peer=1 root=-1 tag=17 comm=0 request=isend 1 │ ╰───────────────────────────────────────────────────────────────────────────────╯ProfilingOptions session
Section titled “ProfilingOptions session”This version activates automatic wrapping through
ProfilingOptions(profile_mpi_calls=True). It also shows that a communicator
returned by Split stays instrumented.
"""Enable automatic mpi4py instrumentation through ProfilingOptions.
Run with at least two ranks:
mpirun -n 2 python examples/ex_mpi_profiling_options.py"""
import numpy as npfrom mpi4py import MPI
from scope_profiler import ProfileManager, ProfilingOptions
def main() -> None: options = ProfilingOptions( profile_mpi_calls=True, file_path="mpi_profiling_options.h5", )
with ProfileManager.session(options=options, verbose=True): # Obtain the communicator after entering the session so this reference # receives the automatic profiling proxy. comm = MPI.COMM_WORLD if comm.size < 2: raise RuntimeError("this example requires at least two MPI ranks")
value = np.array([comm.rank], dtype=np.int32) if comm.rank == 0: comm.Send([value, MPI.INT], dest=1, tag=23) elif comm.rank == 1: comm.Recv([value, MPI.INT], source=0, tag=23) assert value[0] == 0
# Communicators derived from a profiled communicator remain profiled. local_comm = comm.Split(color=comm.rank % 2, key=comm.rank) local_total = local_comm.allreduce(comm.rank, op=MPI.SUM) assert local_total >= 0 local_comm.Free()
comm.Barrier()
if MPI.COMM_WORLD.rank == 0: print("wrote mpi_profiling_options.h5")
if __name__ == "__main__": main()Run it and inspect the recorded MPI regions:
mpirun -n 2 python examples/ex_mpi_profiling_options.pyscope-profiler inspect mpi_profiling_options.h5 --regions-only \ --include '^mpi:' --columns region calls total avgOutput:
if options_example: run_mpi(2, sys.executable, options_example) inspect_mpi_regions("mpi_profiling_options.h5")Show the output
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ region % session total [s] │ ├────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ scope_profiler.session (2x) 100.00% 0.001277 │ │ └─ (own) 50.30% 0.000643 │ │ └─ mpi:recv kind=point-to-point bytes=4 peer=0 root=-1 tag=23 comm=0 30.90% 0.000395 │ │ └─ mpi:send kind=point-to-point bytes=4 peer=1 root=-1 tag=23 comm=0 2.89% 0.000037 │ │ └─ mpi:allreduce kind=collective bytes=-1 peer=-1 root=-1 tag=-1 comm=3 (2x) 11.34% 0.000145 │ │ └─ mpi:barrier kind=collective bytes=0 peer=-1 root=-1 tag=-1 comm=0 (2x) 4.58% 0.000058 │ ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Info ──────────────────────────────────────────────────────────────────────────╮ │ Summary: mpi_profiling_options.h5 (2 ranks) │ │ │ │ Explore: │ │ Inspect: scope-profiler inspect mpi_profiling_options.h5 │ │ TUI: scope-profiler tui mpi_profiling_options.h5 │ │ │ │ Visualize and export: │ │ Plot: scope-profiler plot default mpi_profiling_options.h5 -o plots --show │ │ Report: scope-profiler report mpi_profiling_options.h5 -o report.html │ │ Export: scope-profiler export plot-data mpi_profiling_options.h5 -o data │ │ Lines: scope-profiler line-profile mpi_profiling_options.h5 │ │ │ │ Compare runs: │ │ Diff: scope-profiler diff BASE.h5 CANDIDATE.h5 │ │ Check: scope-profiler check BASE.h5 CANDIDATE.h5 │ │ │ │ Durations are in seconds. │ │ Regions may nest, so the summed total can exceed the wall-clock time. │ │ % session uses wall-clock coverage; overlapping recursive calls count once. │ ╰─────────────────────────────────────────────────────────────────────────────────╯
wrote mpi_profiling_options.h5 ╭─────────────────────────────────────────────────────────────────────────────╮ │ region n │ ├─────────────────────────────────────────────────────────────────────────────┤ │ mpi:allreduce kind=collective bytes=-1 peer=-1 root=-1 tag=-1 comm=3 2 │ │ mpi:barrier kind=collective bytes=0 peer=-1 root=-1 tag=-1 comm=0 2 │ │ mpi:recv kind=point-to-point bytes=4 peer=0 root=-1 tag=23 comm=0 1 │ │ mpi:send kind=point-to-point bytes=4 peer=1 root=-1 tag=23 comm=0 1 │ ╰─────────────────────────────────────────────────────────────────────────────╯Profiling MPI calls in C++
Section titled “Profiling MPI calls in C++”scope_profiler_mpi.hpp provides C++11 wrappers for MPI_Send, MPI_Recv,
MPI_Isend, MPI_Irecv, MPI_Wait, MPI_Barrier, MPI_Bcast, MPI_Reduce,
and MPI_Allreduce:
#include <scope_profiler_mpi.hpp>
sp_init("mpi-calls", rank);
sp::mpi::Request request = sp::mpi::isend( values, count, MPI_DOUBLE, peer, 7, MPI_COMM_WORLD);sp::mpi::wait(request);
sp::mpi::allreduce( local, global, count, MPI_DOUBLE, MPI_SUM, MPI_COMM_WORLD);
sp_finalize();The wrappers return the underlying MPI status code. isend and irecv
return a move-only request whose error() reports initiation status,
native() exposes the underlying MPI_Request, and active() shows whether
MPI still owns it. The application remains responsible for completing or
freeing every request, exactly as with native MPI.
With CMake, configure scope-profiler with
-DSCOPE_PROFILER_ENABLE_MPI=ON, then link the MPI add-on beside either native
backend:
target_link_libraries(simulation PRIVATE scope-profiler::cpp scope-profiler::mpi)scope-profiler::header-only can replace scope-profiler::cpp. Defining
SP_DISABLE_PROFILING removes metadata construction, datatype-size queries,
region strings, and profiler references while the wrappers continue to call
MPI normally.
(The C++ snippets are not compiled by the documentation build, which only executes Python; see the C/C++ guide for building and running native programs.)
MPI metadata representation
Section titled “MPI metadata representation”Both languages use the same canonical region labels:
mpi:send kind=point-to-point bytes=32 peer=2 root=-1 tag=9 comm=7mpi:allreduce kind=collective bytes=32 peer=-1 root=-1 tag=-1 comm=7mpi:wait kind=wait bytes=32 peer=2 root=-1 tag=9 comm=7 request=isendThis makes mixed Python/C++ communication directly comparable and keeps the
data compatible with all existing plots and exporters. -1 means “not
applicable” or “unknown.” Python regions also persist each field as an
mpi.* tag. Communicator IDs come from MPI_Comm_c2f in C++ and Comm.py2f
in Python. Receive labels describe the requested source and buffer capacity;
with wildcard receives, inspect the returned MPI status for the actual source.
Pass any such label to parse_mpi_region() to obtain an
MPIRegionMetadata object, including labels loaded from a native C++ trace.
from scope_profiler import parse_mpi_region
label = "mpi:wait kind=wait bytes=32 peer=2 root=-1 tag=9 comm=7 request=isend"parse_mpi_region(label)MPIRegionMetadata(operation='wait', kind='wait', bytes=32, peer=2, root=-1, tag=9, communicator=7, request='isend')Visualizing MPI results
Section titled “Visualizing MPI results”The Gantt chart CLI and Python API support rank selection:
# Show all ranksscope-profiler plot default profiling_data.h5 --show
# Show only ranks 0 and 2scope-profiler plot default profiling_data.h5 --show --ranks 0 2
# Range syntaxscope-profiler plot default profiling_data.h5 --show --ranks 0-3From Python, using the four-rank profiling_data.h5 written by the example
above:
from scope_profiler import read_h5
if (WORKDIR / "profiling_data.h5").exists(): results = read_h5("profiling_data.h5") region = results["compute"]
# Aggregated over every rank (durations in seconds) print(region.num_calls, region.total_duration, region.average_duration)
# Per-rank breakdowns, for spotting load imbalance print(region.average_durations()) # {rank: seconds} print(region.max_durations()) print(region.num_calls_per_rank())
# Or the raw per-rank data for rank_id in region.ranks: print(f"Rank {rank_id}: avg = {region[rank_id].average_duration:.6f} s")Without MPI
Section titled “Without MPI”If the run was not started by an MPI launcher, or mpi4py is not
installed, scope-profiler silently falls back to single-rank mode. No
code changes are needed — the API is identical. Running the very same
mpi_example.py as a plain Python process records one rank:
plain = subprocess.run( [sys.executable, "mpi_example.py"], check=False, capture_output=True, text=True, cwd=WORKDIR,)print("exit status", plain.returncode)print(read_h5("profiling_data.h5").num_ranks, "rank")exit status 01 rankOverriding the detection
Section titled “Overriding the detection”If a launcher is not recognized (or you want to profile an MPI-enabled build as if it were serial), force the decision from the environment:
SCOPE_PROFILER_MPI=1 ./my_launcher python my_script.py # always use MPI.COMM_WORLDSCOPE_PROFILER_MPI=0 mpirun -n 4 python my_script.py # never touch MPIThis is deliberately not a setup() parameter: the choice belongs to how the
job is launched, not to the code being profiled. With SCOPE_PROFILER_MPI=1
and no mpi4py installed, the run falls back to single-rank mode.
The cell below launches two processes with SCOPE_PROFILER_MPI=0: each one
profiles itself as a serial run into a file of its own, instead of the two
joining one job:
%%writefile override_example.pyimport sys
from scope_profiler import ProfileManager
with ProfileManager.session(file_path=sys.argv[1], verbose=False): with ProfileManager.region("work"): sum(range(1000))Writing override_example.pyrun_mpi( 2, "sh", "-c", f'"{sys.executable}" override_example.py serial_rank_$OMPI_COMM_WORLD_RANK.h5', environment={"SCOPE_PROFILER_MPI": "0"},)for name in sorted(path.name for path in WORKDIR.glob("serial_rank_*.h5")): print(name, read_h5(name).num_ranks, "rank")serial_rank_0.h5 1 rankserial_rank_1.h5 1 rank