2. Post-processing profiling data

Download notebookView on GitHubRun when the site was built.

Install for this tutorial: pip install "scope-profiler[pproc]"; pandas is used for the DataFrame examples.

This notebook is a tour of the analysis API: ProfilingResults and the MPIRegion / Region objects it hands out.

The one thing worth memorising: every duration and timestamp in this API is in seconds, converted from the nanoseconds stored on disk.

import time

from scope_profiler import ProfileManager, read_h5

ProfileManager.setup(file_path="profiling_data.h5", label="tutorial 2")


@ProfileManager.profile("assemble")
def assemble(scale):
    time.sleep(0.004 * scale)


with ProfileManager.region("setup"):
    time.sleep(0.02)

for step in range(5):
    with ProfileManager.region("timestep"):
        assemble(scale=1 + step % 3)
        with ProfileManager.region("solve"):
            time.sleep(0.003)

ProfileManager.finalize(verbose=False)
results = read_h5("profiling_data.h5")
results

<ProfilingResults 'profiling_data.h5': 4 region(s), 1 rank(s)>

ProfilingResults is a mapping of regions

It behaves like an ordered mapping from region name to region, so the usual Python idioms work.

print("regions:", results.region_names)
print("count:", len(results))
print("'solve' recorded?", "solve" in results)
print("ranks in file:", results.num_ranks)

for region in results:
    print(f"  {region.name:<10} {region.num_calls:>3} calls")

regions: ['assemble', 'setup', 'timestep', 'solve']
count: 4
'solve' recorded? True
ranks in file: 1
  assemble     5 calls
  setup        1 calls
  timestep     5 calls
  solve        5 calls

Asking for something that is not there tells you what is there:

try:
    results["sovle"]  # typo
except KeyError as exc:
    print(exc)

"No region named 'sovle' in profiling_data.h5. Available regions: ['assemble', 'setup', 'timestep', 'solve']"

Two levels: MPIRegion and Region

results["solve"] returns an MPIRegion — the region across every rank that recorded it. Indexing it by rank gives a Region, the timings from that one rank.

For a serial run there is only rank 0, but the two levels are the same API you use for an MPI run.

timestep = results["timestep"]

print("aggregated over ranks")
print("  ranks       :", timestep.ranks)
print("  num_calls   :", timestep.num_calls)
print("  total   [s] :", timestep.total_duration)
print("  average [s] :", timestep.average_duration)
print("  min/max [s] :", timestep.min_duration, timestep.max_duration)
print("  std     [s] :", timestep.std_duration)

rank0 = timestep[0]
print("\nrank 0 only")
print("  durations [s]:", rank0.durations)
# Timestamps are on a monotonic clock with an arbitrary origin; measure them
# from the start of the run, as events() does (see tutorial 5).
print("  start times [s]:", (rank0.start_times - results.time_origin).round(3))
print("  summary:", rank0.get_summary())

aggregated over ranks
  ranks       : [0]
  num_calls   : 5
  total   [s] : 0.051660574
  average [s] : 0.010332114799999998
  min/max [s] : 0.007122461 0.015124428
  std     [s] : 0.0029847773632117624

rank 0 only
  durations [s]: [0.00716621 0.01112677 0.01512443 0.00712246 0.01112071]
  start times [s]: [0.023 0.03  0.041 0.056 0.063]
  summary: {'num_calls': 5, 'total_duration': 0.051660574, 'inclusive_duration': 0.051660574, 'exclusive_duration': 5.2648e-05, 'average_duration': 0.0103321148, 'min_duration': 0.007122461, 'max_duration': 0.015124428, 'first_duration': 0.007166206, 'last_duration': 0.01112071, 'std_duration': 0.002984777363211763}

Per-rank breakdowns are available as dictionaries keyed by rank, which is where load imbalance shows up in an MPI run:

print("calls per rank :", timestep.num_calls_per_rank())
print("total per rank :", timestep.total_durations())
print("avg per rank   :", timestep.average_durations())
print("max per rank   :", timestep.max_durations())

calls per rank : {0: 5}
total per rank : {0: 0.051660574}
avg per rank   : {0: 0.0103321148}
max per rank   : {0: 0.015124428}

Summaries and DataFrames

summary() returns one dict per region, aggregated over ranks. print_summary() formats the same data as a table — nested regions are indented under their parent, with an (own) row for the time a parent spends outside its children. It takes sort=, columns= and the usual include / exclude filters; suppress_notes=True drops the footer of follow-up commands.

results.print_summary(suppress_notes=True)

  ╭─────────────────────────────────╮
  │ region              total [s]   │
  ├─────────────────────────────────┤
  │ setup               0.020075    │
  │ timestep (5x)       0.051661    │
  │ └─ (own)               0.000053 │
  │ └─ assemble (5x)       0.036307 │
  │ └─ solve (5x)          0.015301 │
  ╰─────────────────────────────────╯

Each summary row carries the totals, the per-call statistics (average, min, max, std, and the percentiles p50, p95, p99), the exclusive time that excludes nested regions, and the load imbalance across ranks. Here are a few of the fields:

fields = (
    "name",
    "num_calls",
    "total_duration",
    "exclusive_duration",
    "p95",
    "imbalance",
)
for row in results.summary():
    print({key: row[key] for key in fields})

{'name': 'setup', 'num_calls': 1, 'total_duration': 0.020074563, 'exclusive_duration': 0.020074563, 'p95': 0.020074563, 'imbalance': 0.0}
{'name': 'timestep', 'num_calls': 5, 'total_duration': 0.051660574, 'exclusive_duration': 5.2648e-05, 'p95': 0.0143248962, 'imbalance': 0.0}
{'name': 'assemble', 'num_calls': 5, 'total_duration': 0.036307122, 'exclusive_duration': 0.036307122, 'p95': 0.011261087999999999, 'imbalance': 0.0}
{'name': 'solve', 'num_calls': 5, 'total_duration': 0.015300804, 'exclusive_duration': 0.015300804, 'p95': 0.0030610358, 'imbalance': 0.0}

With pandas installed (it comes with the pproc extra), to_dataframe() gives you the same data ready for sorting, filtering and plotting.

frame = results.to_dataframe()
columns = ["name", "num_calls", "total_duration", "exclusive_duration", "p95_duration"]
frame[columns].sort_values("total_duration", ascending=False)

	name	num_calls	total_duration	exclusive_duration	p95_duration
1	timestep	5	0.051661	0.000053	0.014325
2	assemble	5	0.036307	0.036307	0.011261
0	setup	1	0.020075	0.020075	0.020075
3	solve	5	0.015301	0.015301	0.003061

per_rank=True emits one row per (region, rank) instead — the shape you want for load-balance analysis.

per_rank = results.to_dataframe(per_rank=True)
per_rank[["name", "rank", "num_calls", "total_duration", "average_duration"]]

	name	rank	num_calls	total_duration	average_duration
0	setup	0	1	0.020075	0.020075
1	timestep	0	5	0.051661	0.010332
2	assemble	0	5	0.036307	0.007261
3	solve	0	5	0.015301	0.003060

# Share of total recorded time per region.
frame = frame.assign(share=lambda df: df.total_duration / df.total_duration.sum())
frame[["name", "num_calls", "total_duration", "share"]].sort_values(
    "share", ascending=False
)

	name	num_calls	total_duration	share
1	timestep	5	0.051661	0.418836
2	assemble	5	0.036307	0.294359
0	setup	1	0.020075	0.162754
3	solve	5	0.015301	0.124051

Selecting regions

get_regions(), summary(), to_dataframe() and the plotting functions all take include / exclude, which are regular expressions matched against the region name (via re.match, so they anchor at the start).

print([region.name for region in results.get_regions(include="s")])
print([region.name for region in results.get_regions(exclude=["setup", "assemble"])])
print([row["name"] for row in results.summary(include=["solve", "timestep"])])

['setup', 'solve']
['timestep', 'solve']
['timestep', 'solve']

Run metadata

Each file carries a description of the environment it was recorded in, which is what tells two otherwise identical runs apart months later. By default that is a minimal record: versions, platform, CPU model, rank and thread counts, the start time and the run’s label.

for key, value in sorted(results.metadata.items()):
    print(f"{key:>24}: {value}")

        chip_information: AMD EPYC 9V45 96-Core Processor
        finalize_time_ns: 183186334551
                   label: tutorial 2
                mpi_size: 1
               num_nodes: 1
         omp_num_threads: 4
                platform: Linux-7.0.0-1017-azure-x86_64-with-glibc2.39
          python_version: 3.12.3
  scope_profiler_version: 0.7.5
           start_time_ns: 183111860933
               timestamp: 2026-10-22T06:38:32.031523+00:00
             total_cores: 4

setup(metadata_detail="full") adds what is more personal or more bulky: the user and host name, the working directory, loaded environment modules, selected environment variables such as PATH, and any SLURM_* job variables. It is off by default so a profile can be shared without leaking that. Capturing the region source text is a separate switch, capture_region_source=True.

ProfileManager.setup(file_path="full.h5", metadata_detail="full", label="full metadata")
with ProfileManager.region("work"):
    pass
full = ProfileManager.finalize(verbose=False, return_results=True)

print("extra fields:", sorted(set(full.metadata) - set(results.metadata)))

extra fields: ['PATH', 'hostname', 'modules', 'uname', 'user', 'working_directory']

scope-profiler inspect prints the metadata grouped and readable — together with per-region statistics — straight from the command line. Long values such as PATH are clipped unless you pass --full:

scope-profiler inspect profiling_data.h5
scope-profiler inspect profiling_data.h5 --metadata-only
scope-profiler inspect profiling_data.h5 --regions-only --sort total

write_metadata_json() exports the metadata instead, with one entry per file and no clipping of long values:

from scope_profiler.inspection import write_metadata_json

payload = write_metadata_json("profiling_data.h5", "metadata.json")
print(sorted(payload["files"][0]["metadata"])[:8])

['chip_information', 'finalize_time_ns', 'label', 'mpi_size', 'num_nodes', 'omp_num_threads', 'platform', 'python_version']

Exporting statistics

collect_region_statistics() builds a JSON-ready dict (durations in seconds, with a units field spelling that out), and write_region_statistics_json() writes it to a file. Both accept several runs at once for run comparisons.

from scope_profiler import collect_region_statistics, write_region_statistics_json

stats = collect_region_statistics(results, include="solve")
print(stats["units"])
print(stats["files"][0]["region_statistics"]["solve"])

write_region_statistics_json(results, "region_statistics.json")
print("wrote region_statistics.json")

{'durations': 'seconds'}
{'count': 5, 'average_duration_seconds': 0.0030601608, 'min_duration_seconds': 0.003058693, 'max_duration_seconds': 0.003061126, 'first_duration_seconds': 0.003061126, 'last_duration_seconds': 0.003058693, 'std_duration_seconds': 8.227170595046889e-07, 'total_duration_seconds': 0.015300804, 'per_rank': {'0': {'count': 5, 'average_duration_seconds': 0.0030601608, 'min_duration_seconds': 0.003058693, 'max_duration_seconds': 0.003061126, 'first_duration_seconds': 0.003061126, 'last_duration_seconds': 0.003058693, 'std_duration_seconds': 8.227170595046889e-07, 'total_duration_seconds': 0.015300804}}}
wrote region_statistics.json

Next

3. Visualizing results turns these numbers into Gantt, flame and duration charts, 5. Custom analysis goes below the summaries to the individual calls, and 8. Reports and comparison compares whole runs.