MPI support#
Install for this page:
pip install "scope-profiler[mpi]", plus an MPI implementation such as Open MPI or MPICH. Parallel HDF5 output additionally requires an MPI-enabled build of h5py; serial-HDF5 merging does not.
scope-profiler is MPI-aware out of the box. When the process is launched
under an MPI launcher and mpi4py is installed, the profiler
automatically detects MPI.COMM_WORLD and handles per-rank data
collection and merging.
Installation#
pip install "scope-profiler[mpi]"
How it works#
Launcher detection — before anything MPI-related happens, scope-profiler checks whether this process was started by
mpirun/mpiexec/srunor an equivalent launcher, by looking for the per-rank environment variables those launchers export (OMPI_COMM_WORLD_RANK,PMI_RANK,PMIX_RANK, …). If none is present,mpi4pyis never imported and no MPI call is ever made — a plainpython script.pyrun pays nothing for MPI support, not evenMPI_Init. The one exception is an application that already importedmpi4pyand initialized MPI itself; then the existing communicator is used.Setup —
ProfilingConfigreadsCOMM_WORLDfor rank and size. That is a local query, not a collective:setup()issues no MPI call of its own and creates no temporary directory. Nothing happens before then: importing scope-profiler touches MPI not at all, so a process that merely imports the library — a child forked from a rank, say — never joins the job.Recording — each rank accumulates its own timestamps in memory. Nothing is written until finalization.
Finalize — every rank writes its own data into the same output file; timestamp arrays are never sent to rank 0. With an MPI-enabled h5py build, ordinary region data is written through parallel HDF5. Otherwise a tiny point-to-point ownership token moves through the ranks: each rank opens the serial-HDF5 file, appends its indexed event slice, closes it, then passes the token. The last rank reports completion to rank 0, which atomically publishes the file under its final name.
A rank that entered no region contributes no rank/region index row. And because every rank must reach
finalize(), one that dies first leaves the job waiting rather than quietly dropping its data from the output.LIKWID records use the token writer even when parallel HDF5 is installed. LIKWID reads counters in a subprocess, and Open MPI does not safely support a collective MPI-IO phase after that fork. The direct writer deliberately uses point-to-point traffic only. Line-profiler records are supported by both writers.
All ranks must see
file_paththrough the same shared filesystem. Select a backend explicitly withProfileManager.setup(output_mode="direct")oroutput_mode="parallel"; the defaultautochooses the safe option.
Example#
The code is identical to the serial case — no MPI-specific API calls are needed:
# mpi_example.py
from scope_profiler import ProfileManager
ProfileManager.setup()
@ProfileManager.profile("compute")
def compute():
s = 0
for i in range(100_000):
s += i
return s
compute()
ProfileManager.finalize()
Run with MPI:
mpirun -n 4 python mpi_example.py
The output profiling_data.h5 will contain groups rank0 through
rank3, each with their own timing data.
Profiling MPI calls in Python#
The ordinary MPI support above profiles work on each rank. Enable MPI-call profiling when communication operations themselves should appear on the timeline.
For an unchanged script, launch it through the regular profiler command:
mpirun -n 4 scope-profiler run --mpi-calls simulation.py
MPI-call profiling is opt-in. With --mpi-calls, the runner waits for
the target to import mpi4py, then temporarily replaces mpi4py’s
COMM_WORLD and COMM_SELF globals with transparent profiling proxies
and restores them at shutdown. Communicators returned by common
constructors such as Split, Dup, Create_cart, and
Create_intercomm remain wrapped automatically. Communicators obtained
through an unrecognized third-party API can still be passed to
profile_mpi_comm() explicitly. --no-mpi-calls is accepted when an
explicit config file enables the feature and the command should override
it.
For a normal profiling session, enable the same behavior through
ProfilingOptions and obtain the communicator inside the session:
from mpi4py import MPI
from scope_profiler import ProfileManager, ProfilingOptions
options = ProfilingOptions(profile_mpi_calls=True)
with ProfileManager.session(options=options, file_path="mpi-calls.h5"):
comm = MPI.COMM_WORLD
comm.barrier()
The setting is also accepted in TOML:
[profiling]
profile_mpi_calls = true
To wrap an individual communicator explicitly instead:
from mpi4py import MPI
from scope_profiler import ProfileManager, profile_mpi_comm
comm = profile_mpi_comm(MPI.COMM_WORLD)
with ProfileManager.session(file_path="mpi-calls.h5"):
if comm.rank == 0:
request = comm.isend({"work": 42}, dest=1, tag=7)
request.wait()
elif comm.rank == 1:
payload = comm.recv(source=0, tag=7)
total = comm.allreduce(comm.rank)
comm.barrier()
ProfiledMPIComm is a proxy: attributes and unwrapped methods continue
to come from the original communicator. It explicitly wraps the
object-mode send, recv, isend, irecv, bcast, reduce,
allreduce, and barrier methods, along with their uppercase
buffer-mode equivalents. Nonblocking calls return ProfiledMPIRequest;
its wait() / Wait() region retains the initiating operation’s bytes,
peer, tag, and communicator.
Buffer-mode message sizes are exact when mpi4py supplies a datatype. For
object-mode values without a buffer interface, the default is bytes=-1
so profiling never serializes an object twice or repeats its
__reduce__ side effects. Pass
profile_mpi_comm(comm, pickle_object_sizes=True) to record a
protocol-5 pickle-size estimate when that extra serialization is
acceptable. A high-level receive cannot know the incoming serialized
size before it blocks, so it also records bytes=-1; use Recv/Irecv
with an explicit buffer when an exact receive capacity is important.
The module does not import mpi4py. Passing the already-created communicator is what preserves scope-profiler’s serial-process guarantee: importing scope-profiler alone never initializes MPI.
Complete runnable Python examples#
The documentation build executes every example below with two MPI ranks
and captures the displayed output from the resulting profile. The source
blocks are included directly from examples/, so they are the files the
commands actually run rather than copies maintained separately in this
page.
Explicit communicator wrapper#
This version opts in by wrapping one communicator directly.
"""Profile mpi4py communication operations and their message metadata.
Run with at least two ranks::
mpirun -n 2 python examples/ex_mpi_wrappers.py
"""
from mpi4py import MPI
from scope_profiler import ProfileManager, profile_mpi_comm
def main() -> None:
comm = profile_mpi_comm(MPI.COMM_WORLD)
with ProfileManager.session(file_path="mpi_calls.h5", verbose=False):
if comm.rank == 0:
request = comm.isend({"work": 42}, dest=1, tag=7)
request.wait()
elif comm.rank == 1:
comm.recv(source=0, tag=7)
comm.allreduce(comm.rank)
comm.barrier()
if __name__ == "__main__":
main()
Run it and inspect the recorded MPI regions:
mpirun -n 2 python examples/ex_mpi_wrappers.py
scope-profiler inspect mpi_calls.h5 --regions-only --include '^mpi:' \
--columns region calls total avg
Output:
==============================================================================
mpi_calls.h5
2 rank(s), 6 region(s), 0.02 MiB, 0.000616945 s wall clock, 0.00179448 s total (setup to finalize)
==============================================================================
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ region n total [s] avg/call [s] │
├───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ mpi:recv kind=point-to-point bytes=-1 peer=0 root=-1 tag=7 comm=0 1 0.000051 5.131e-05 │
│ mpi:isend kind=point-to-point bytes=-1 peer=1 root=-1 tag=7 comm=0 1 0.000029 2.867e-05 │
│ mpi:wait kind=wait bytes=-1 peer=1 root=-1 tag=7 comm=0 request=isend 1 0.000007 6.522e-06 │
│ mpi:allreduce kind=collective bytes=-1 peer=-1 root=-1 tag=-1 comm=0 2 0.000273 1.365e-04 │
│ mpi:barrier kind=collective bytes=0 peer=-1 root=-1 tag=-1 comm=0 2 0.000014 6.893e-06 │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Info ────────────────────────────────────────────────────────────────╮
│ Summary: Regions (5) │
│ │
│ Explore: │
│ Inspect: scope-profiler inspect mpi_calls.h5 │
│ TUI: scope-profiler tui mpi_calls.h5 │
│ │
│ Visualize and export: │
│ Plot: scope-profiler plot default mpi_calls.h5 -o plots --show │
│ Report: scope-profiler report mpi_calls.h5 -o report.html │
│ Export: scope-profiler export plot-data mpi_calls.h5 -o data │
│ Lines: scope-profiler line-profile mpi_calls.h5 │
│ │
│ Compare runs: │
│ Diff: scope-profiler diff BASE.h5 CANDIDATE.h5 │
│ Check: scope-profiler check BASE.h5 CANDIDATE.h5 │
│ │
│ Durations are in seconds. │
│ Regions may nest, so the summed total can exceed the wall-clock time. │
╰───────────────────────────────────────────────────────────────────────╯
Unmodified NumPy/mpi4py program#
This file contains no scope-profiler imports. Running it with
scope-profiler run --mpi-calls enables MPI-call instrumentation first.
"""Profile exact-size NumPy buffer operations with mpi4py.
Run with at least two ranks:
mpirun -n 2 scope-profiler run -q \
-o mpi_numpy_calls.h5 examples/ex_mpi_numpy_wrappers.py
"""
import numpy as np
from mpi4py import MPI
def main() -> None:
comm = MPI.COMM_WORLD
if comm.size < 2:
raise RuntimeError("this example requires at least two MPI ranks")
if comm.rank == 0:
values = np.arange(8, dtype=np.float64)
comm.Isend([values, MPI.DOUBLE], dest=1, tag=17).Wait()
elif comm.rank == 1:
received = np.empty(8, dtype=np.float64)
comm.Irecv([received, MPI.DOUBLE], source=0, tag=17).Wait()
np.testing.assert_array_equal(received, np.arange(8, dtype=np.float64))
broadcast_value = np.array([42 if comm.rank == 0 else 0], dtype=np.int32)
comm.Bcast([broadcast_value, MPI.INT], root=0)
assert broadcast_value[0] == 42
local = np.array([comm.rank + 1], dtype=np.int32)
total = np.zeros(1, dtype=np.int32)
comm.Allreduce([local, MPI.INT], [total, MPI.INT], op=MPI.SUM)
assert total[0] == comm.size * (comm.size + 1) // 2
comm.Barrier()
if __name__ == "__main__":
main()
Run it and inspect the recorded MPI regions:
mpirun -n 2 scope-profiler run -q \
-o mpi_numpy_calls.h5 examples/ex_mpi_numpy_wrappers.py
scope-profiler inspect mpi_numpy_calls.h5 --regions-only --include '^mpi:' \
--columns region calls total avg
Output:
==============================================================================
mpi_numpy_calls.h5
0 rank(s), 0 region(s), 0.01 MiB, 0.262329 s total (setup to finalize)
==============================================================================
Regions (0)
(no regions recorded)
ProfilingOptions session#
This version activates automatic wrapping through
ProfilingOptions(profile_mpi_calls=True). It also shows that a
communicator returned by Split stays instrumented.
"""Enable automatic mpi4py instrumentation through ProfilingOptions.
Run with at least two ranks:
mpirun -n 2 python examples/ex_mpi_profiling_options.py
"""
import numpy as np
from mpi4py import MPI
from scope_profiler import ProfileManager, ProfilingOptions
def main() -> None:
options = ProfilingOptions(
profile_mpi_calls=True,
file_path="mpi_profiling_options.h5",
)
with ProfileManager.session(options=options, verbose=True):
# Obtain the communicator after entering the session so this reference
# receives the automatic profiling proxy.
comm = MPI.COMM_WORLD
if comm.size < 2:
raise RuntimeError("this example requires at least two MPI ranks")
value = np.array([comm.rank], dtype=np.int32)
if comm.rank == 0:
comm.Send([value, MPI.INT], dest=1, tag=23)
elif comm.rank == 1:
comm.Recv([value, MPI.INT], source=0, tag=23)
assert value[0] == 0
# Communicators derived from a profiled communicator remain profiled.
local_comm = comm.Split(color=comm.rank % 2, key=comm.rank)
local_total = local_comm.allreduce(comm.rank, op=MPI.SUM)
assert local_total >= 0
local_comm.Free()
comm.Barrier()
if MPI.COMM_WORLD.rank == 0:
print("wrote mpi_profiling_options.h5")
if __name__ == "__main__":
main()
Run it and inspect the recorded MPI regions:
mpirun -n 2 python examples/ex_mpi_profiling_options.py
scope-profiler inspect mpi_profiling_options.h5 --regions-only \
--include '^mpi:' --columns region calls total avg
Output:
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ region % session total [s] │
├────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ scope_profiler.session (2x) 100.00% 0.001680 │
│ └─ (own) 58.56% 0.000984 │
│ └─ mpi:recv kind=point-to-point bytes=4 peer=0 root=-1 tag=23 comm=0 33.42% 0.000562 │
│ └─ mpi:send kind=point-to-point bytes=4 peer=1 root=-1 tag=23 comm=0 1.79% 0.000030 │
│ └─ mpi:allreduce kind=collective bytes=-1 peer=-1 root=-1 tag=-1 comm=3 (2x) 5.34% 0.000090 │
│ └─ mpi:barrier kind=collective bytes=0 peer=-1 root=-1 tag=-1 comm=0 (2x) 0.89% 0.000015 │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Info ──────────────────────────────────────────────────────────────────────────╮
│ Summary: mpi_profiling_options.h5 (2 ranks) │
│ │
│ Explore: │
│ Inspect: scope-profiler inspect mpi_profiling_options.h5 │
│ TUI: scope-profiler tui mpi_profiling_options.h5 │
│ │
│ Visualize and export: │
│ Plot: scope-profiler plot default mpi_profiling_options.h5 -o plots --show │
│ Report: scope-profiler report mpi_profiling_options.h5 -o report.html │
│ Export: scope-profiler export plot-data mpi_profiling_options.h5 -o data │
│ Lines: scope-profiler line-profile mpi_profiling_options.h5 │
│ │
│ Compare runs: │
│ Diff: scope-profiler diff BASE.h5 CANDIDATE.h5 │
│ Check: scope-profiler check BASE.h5 CANDIDATE.h5 │
│ │
│ Durations are in seconds. │
│ Regions may nest, so the summed total can exceed the wall-clock time. │
│ % session uses wall-clock coverage; overlapping recursive calls count once. │
╰─────────────────────────────────────────────────────────────────────────────────╯
wrote mpi_profiling_options.h5
==============================================================================
mpi_profiling_options.h5
2 rank(s), 5 region(s), 0.02 MiB, 0.00110771 s wall clock, 0.00183263 s total (setup to finalize)
==============================================================================
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ region n total [s] avg/call [s] │
├──────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ mpi:recv kind=point-to-point bytes=4 peer=0 root=-1 tag=23 comm=0 1 0.000562 5.615e-04 │
│ mpi:send kind=point-to-point bytes=4 peer=1 root=-1 tag=23 comm=0 1 0.000030 3.005e-05 │
│ mpi:allreduce kind=collective bytes=-1 peer=-1 root=-1 tag=-1 comm=3 2 0.000090 4.486e-05 │
│ mpi:barrier kind=collective bytes=0 peer=-1 root=-1 tag=-1 comm=0 2 0.000015 7.474e-06 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Info ──────────────────────────────────────────────────────────────────────────╮
│ Summary: Regions (4) │
│ │
│ Explore: │
│ Inspect: scope-profiler inspect mpi_profiling_options.h5 │
│ TUI: scope-profiler tui mpi_profiling_options.h5 │
│ │
│ Visualize and export: │
│ Plot: scope-profiler plot default mpi_profiling_options.h5 -o plots --show │
│ Report: scope-profiler report mpi_profiling_options.h5 -o report.html │
│ Export: scope-profiler export plot-data mpi_profiling_options.h5 -o data │
│ Lines: scope-profiler line-profile mpi_profiling_options.h5 │
│ │
│ Compare runs: │
│ Diff: scope-profiler diff BASE.h5 CANDIDATE.h5 │
│ Check: scope-profiler check BASE.h5 CANDIDATE.h5 │
│ │
│ Durations are in seconds. │
│ Regions may nest, so the summed total can exceed the wall-clock time. │
╰─────────────────────────────────────────────────────────────────────────────────╯
Profiling MPI calls in C++#
scope_profiler_mpi.hpp provides C++11 wrappers for MPI_Send,
MPI_Recv, MPI_Isend, MPI_Irecv, MPI_Wait, MPI_Barrier,
MPI_Bcast, MPI_Reduce, and MPI_Allreduce:
#include <scope_profiler_mpi.hpp>
sp_init("mpi-calls", rank);
sp::mpi::Request request = sp::mpi::isend(
values, count, MPI_DOUBLE, peer, 7, MPI_COMM_WORLD);
sp::mpi::wait(request);
sp::mpi::allreduce(
local, global, count, MPI_DOUBLE, MPI_SUM, MPI_COMM_WORLD);
sp_finalize();
The wrappers return the underlying MPI status code. isend and irecv
return a move-only request whose error() reports initiation status,
native() exposes the underlying MPI_Request, and active() shows
whether MPI still owns it. The application remains responsible for
completing or freeing every request, exactly as with native MPI.
With CMake, configure scope-profiler with
-DSCOPE_PROFILER_ENABLE_MPI=ON, then link the MPI add-on beside either
native backend:
target_link_libraries(simulation PRIVATE
scope-profiler::cpp
scope-profiler::mpi
)
scope-profiler::header-only can replace scope-profiler::cpp.
Defining SP_DISABLE_PROFILING removes metadata construction,
datatype-size queries, region strings, and profiler references while the
wrappers continue to call MPI normally.
MPI metadata representation#
Both languages use the same canonical region labels:
mpi:send kind=point-to-point bytes=32 peer=2 root=-1 tag=9 comm=7
mpi:allreduce kind=collective bytes=32 peer=-1 root=-1 tag=-1 comm=7
mpi:wait kind=wait bytes=32 peer=2 root=-1 tag=9 comm=7 request=isend
This makes mixed Python/C++ communication directly comparable and keeps
the data compatible with all existing plots and exporters. -1 means
“not applicable” or “unknown.” Python regions also persist each field as
an mpi.* tag. Communicator IDs come from MPI_Comm_c2f in C++ and
Comm.py2f in Python. Receive labels describe the requested source and
buffer capacity; with wildcard receives, inspect the returned MPI status
for the actual source. Pass any such label to parse_mpi_region() to
obtain an MPIRegionMetadata object, including labels loaded from a
native C++ trace.
Visualizing MPI results#
The Gantt chart CLI and Python API support rank selection:
# Show all ranks
scope-profiler plot default profiling_data.h5 --show
# Show only ranks 0 and 2
scope-profiler plot default profiling_data.h5 --show --ranks 0 2
# Range syntax
scope-profiler plot default profiling_data.h5 --show --ranks 0-3
From Python:
from scope_profiler import read_h5
results = read_h5("profiling_data.h5")
region = results["compute"]
# Aggregated over every rank (durations in seconds)
print(region.num_calls, region.total_duration, region.average_duration)
# Per-rank breakdowns, for spotting load imbalance
print(region.average_durations()) # {rank: seconds}
print(region.max_durations())
print(region.num_calls_per_rank())
# Or the raw per-rank data
for rank_id in region.ranks:
print(f"Rank {rank_id}: avg = {region[rank_id].average_duration:.6f} s")
Without MPI#
If the run was not started by an MPI launcher, or mpi4py is not
installed, scope-profiler silently falls back to single-rank mode. No
code changes are needed — the API is identical.
Overriding the detection#
If a launcher is not recognized (or you want to profile an MPI-enabled build as if it were serial), force the decision from the environment:
SCOPE_PROFILER_MPI=1 ./my_launcher python my_script.py # always use MPI.COMM_WORLD
SCOPE_PROFILER_MPI=0 mpirun -n 4 python my_script.py # never touch MPI
This is deliberately not a setup() parameter: the choice belongs to
how the job is launched, not to the code being profiled. With
SCOPE_PROFILER_MPI=1 and no mpi4py installed, the run falls back to
single-rank mode.