MPI support

MPI support#

scope-profiler is MPI-aware out of the box. When the process is launched under an MPI launcher and mpi4py is installed, the profiler automatically detects MPI.COMM_WORLD and handles per-rank data collection and merging.

Installation#

pip install "scope-profiler[mpi]"

How it works#

  1. Launcher detection — before anything MPI-related happens, scope-profiler checks whether this process was started by mpirun/mpiexec/srun or an equivalent launcher, by looking for the per-rank environment variables those launchers export (OMPI_COMM_WORLD_RANK, PMI_RANK, PMIX_RANK, …). If none is present, mpi4py is never imported and no MPI call is ever made — a plain python script.py run pays nothing for MPI support, not even MPI_Init. The one exception is an application that already imported mpi4py and initialized MPI itself; then the existing communicator is used.

  2. Setup — ProfilingConfig reads COMM_WORLD for rank and size. That is a local query, not a collective: setup() issues no MPI call of its own and creates no temporary directory. Nothing happens before then: importing scope-profiler touches MPI not at all, so a process that merely imports the library — a child forked from a rank, say — never joins the job.

  3. Recording — each rank accumulates its own timestamps in memory. Nothing is written until finalization.

  4. Finalize — every rank writes its own data into the same output file; timestamp arrays are never sent to rank 0. With an MPI-enabled h5py build, ordinary region data is written through parallel HDF5. Otherwise a tiny point-to-point ownership token moves through the ranks: each rank opens the serial-HDF5 file, appends its indexed event slice, closes it, then passes the token. The last rank reports completion to rank 0, which atomically publishes the file under its final name.

    A rank that entered no region contributes no rank/region index row. And because every rank must reach finalize(), one that dies first leaves the job waiting rather than quietly dropping its data from the output.

    LIKWID records use the token writer even when parallel HDF5 is installed. LIKWID reads counters in a subprocess, and Open MPI does not safely support a collective MPI-IO phase after that fork. The direct writer deliberately uses point-to-point traffic only. Line-profiler records are supported by both writers.

    All ranks must see file_path through the same shared filesystem. Select a backend explicitly with ProfileManager.setup(output_mode="direct") or output_mode="parallel"; the default auto chooses the safe option.

Example#

The code is identical to the serial case — no MPI-specific API calls are needed:

# mpi_example.py
from scope_profiler import ProfileManager

ProfileManager.setup()

@ProfileManager.profile("compute")
def compute():
    s = 0
    for i in range(100_000):
        s += i
    return s

compute()
ProfileManager.finalize()

Run with MPI:

mpirun -n 4 python mpi_example.py

The output profiling_data.h5 will contain groups rank0 through rank3, each with their own timing data.

Visualizing MPI results#

The Gantt chart CLI and Python API support rank selection:

# Show all ranks
scope-profiler plot default profiling_data.h5 --show

# Show only ranks 0 and 2
scope-profiler plot default profiling_data.h5 --show --ranks 0 2

# Range syntax
scope-profiler plot default profiling_data.h5 --show --ranks 0-3

From Python:

from scope_profiler import read_h5

results = read_h5("profiling_data.h5")
region = results["compute"]

# Aggregated over every rank (durations in seconds)
print(region.num_calls, region.total_duration, region.average_duration)

# Per-rank breakdowns, for spotting load imbalance
print(region.average_durations())   # {rank: seconds}
print(region.max_durations())
print(region.num_calls_per_rank())

# Or the raw per-rank data
for rank_id in region.ranks:
    print(f"Rank {rank_id}: avg = {region[rank_id].average_duration:.6f} s")

Without MPI#

If the run was not started by an MPI launcher, or mpi4py is not installed, scope-profiler silently falls back to single-rank mode. No code changes are needed — the API is identical.

Overriding the detection#

If a launcher is not recognized (or you want to profile an MPI-enabled build as if it were serial), force the decision from the environment:

SCOPE_PROFILER_MPI=1 ./my_launcher python my_script.py   # always use MPI.COMM_WORLD
SCOPE_PROFILER_MPI=0 mpirun -n 4 python my_script.py     # never touch MPI

This is deliberately not a setup() parameter: the choice belongs to how the job is launched, not to the code being profiled. With SCOPE_PROFILER_MPI=1 and no mpi4py installed, the run falls back to single-rank mode.

Contents