API reference

API reference#

ProfileManager#

The main entry point for all profiling operations. Class methods provide the process-wide default manager, so the common case does not require an instance. Create ProfileManager() instances when independent profiling sessions need to coexist; each instance owns its configuration, regions, decorators, and output.

class scope_profiler.profile_manager.ProfileManager#

Manage and track a set of profiling regions.

The class methods remain the process-wide default API. Instantiating the class creates an independent manager with its own configuration, regions, decorators, and call-id space, allowing multiple profiling sessions to be active at the same time:

cpu = ProfileManager()
io = ProfileManager()

with cpu.session(file_path="cpu.h5"):
    with io.session(file_path="io.h5"):
        with cpu.profile_region("compute"), io.profile_region("write"):
            work()

classmethod profile_region(region_name, functions=None, tags=None)#

Get an existing ProfileRegion by name, or create a new one if it doesn’t exist.

Parameters:

region_name (str) – The name of the profiling region.
functions (list of callable, optional) –
Functions to register for line-by-line profiling. Only has an effect when use_line_profiler=True. Useful when using the context manager form, since the decorator form (wrap) registers functions automatically:
```
with ProfileManager.profile_region("my_region", functions=[my_func]):
    my_func()
```
tags (iterable of str, optional) – User-defined labels persisted with the region. Reusing a region name with a different non-None tag set raises ValueError.

Returns:

ProfileRegion

Return type:

The ProfileRegion instance.

classmethod profile(region_name=None, recursive=None)#

Decorator factory for profiling a function.

Parameters:

region_name (str, optional) – Name for the profiling region. If not provided, uses the decorated function’s name. Supports being used with or without parentheses.
recursive (bool, optional) – If True, also profiles Python function calls made by the decorated function (excluding scope-profiler internals). If None, falls back to ProfileManager.setup(recursive_profile=...).

Returns:

Decorated function wrapped with profiling instrumentation.

Return type:

Callable

Notes

The decorated function is registered so that calling ProfileManager.setup() after decoration re-binds the wrapper to the new region class at zero per-call cost. This means @ProfileManager.profile can be applied at class-definition time even when setup() is called later.

classmethod finalize(verbose=True, return_results=False, native_traces=None, verbose_line_profiler=False)#

Finalize profiling and write the run’s data to a single output file.

Copies each region’s buffered timestamps out, moves every rank’s copy to rank 0, and has rank 0 write them into one HDF5 file. Nothing is staged on the filesystem, so no shared $TMPDIR is needed. Optionally prints profiling statistics for each region.

Under MPI this is collective: every rank must call it, with the same arguments, or the job hangs – rank 0 waits for a payload from every other rank. A rank that dies before reaching it therefore leaves the job waiting rather than silently dropping that rank’s data.

With use_likwid=True this is also where the LIKWID markers are closed and every marker region of the run is read back and stored in the output file under rank<r>/likwid/regions/<tag>; see get_likwid_regions() for reading it back.

Parameters:

verbose (bool, optional) – If True, prints the concise profiling summary (default: True).
verbose_line_profiler (bool, optional) – If True, prints detailed line-profiler tables when line profiling is enabled (default: False).
return_results (bool, optional) –
If True, return the run’s data as a ProfilingResults - the same post-processing API read_h5() gives back, built straight from the in-memory buffers instead of by reading the output file back:
```
results = ProfileManager.finalize(return_results=True)
results.print_summary()
df = results.to_dataframe()
```
This works with deactivate_file_output=True, where no file is written at all. Under MPI the per-rank data is gathered on rank 0, which is collective: every rank must pass the same value.
native_traces (path or sequence of paths, optional) –
Trace files (or directories of them) written by the Fortran region API in this same process, to fold into this run’s output. Each rank picks up the trace matching its own rank, so a mixed-language MPI run still produces one file:
```
kernels.stop_profiling()            # Fortran sp_finalize()
ProfileManager.finalize(native_traces=".")
```
Call the Fortran side’s sp_finalize() first: its trace has to exist by the time this reads it. A region name recorded on both sides raises, rather than silently double-counting.

Returns:

The run’s profiling data when return_results=True, and None otherwise. Under MPI rank 0 gets the whole run, like the merged output file; the other ranks get an empty result set for which print_summary(), the plot_* functions and the exporters do nothing, so the script above needs no rank guard. See is_root.

Return type:

ProfilingResults or None

classmethod read_results()#

Open the merged profiling file this run wrote, for post-processing.

Convenience for analysing results in the same script that produced them:

ProfileManager.finalize()
results = ProfileManager.read_results()
results.print_summary()

Returns:: The data in the file at config.file_path.
Return type:: ProfilingResults
Raises:: FileNotFoundError – If the merged file does not exist yet. It is written by finalize(), and only on rank 0 - guard the call with if ProfileManager.get_config()._rank == 0 under MPI.

classmethod get_region(region_name)#

Get a registered ProfileRegion by name.

Parameters:: region_name (str) – The name of the profiling region.
Returns:: ProfileRegion or None
Return type:: The registered ProfileRegion instance or None if not found.

classmethod get_all_regions()#

Get all registered ProfileRegion instances.

Returns:: dict
Return type:: Dictionary of all registered ProfileRegion instances.

classmethod setup(options=None, *, file_path=None, label=None, use_likwid=None, use_line_profiler=None, deactivate_profiling=None, use_nvtx=None, use_gpu_timing=None, gpu_timing_backend=None, deactivate_file_output=None, recursive_profile=None, aggregation_mode=None, capture_region_source=None, buffer_limit=None, output_mode=None, hdf5_compression=None, hdf5_compression_level=None, hdf5_chunk_size=None, config_path=None)#

Initialize and configure the profiling system.

Parameters:

options (ProfilingOptions, optional) –
A ProfilingOptions bag holding any of the settings below, for reuse across calls or construction away from the call site:
```
options = ProfilingOptions(use_likwid=True, file_path="run.h5")
ProfileManager.setup(options=options)
```
An explicit keyword argument passed alongside options wins over the same field on options, which in turn wins over config_path and the defaults below.
file_path (str, optional) – Path to the output profiling data file (default: “profiling_data.h5”).
label (str or None, optional) –
Short name for this run (default: None, i.e. the output file’s stem). Post-processing uses it wherever a run has to be named – chart legends, the summary heading, scope-profiler inspect, the JSON statistics – which is what makes several runs distinguishable when they are compared:
```
ProfileManager.setup(file_path="run_a.h5", label="128 ranks")
```
It is stored in the output file as the label metadata field, so it survives into every later post-processing step.
use_likwid (bool, optional) – Enable LIKWID hardware counter collection (default: False).
use_line_profiler (bool, optional) – Enable line-by-line profiling via line_profiler (default: False).
deactivate_profiling (bool, optional) – Turn profiling off entirely (default: False). Every region becomes a no-op, so the instrumentation can stay in the code at near-zero cost instead of being removed.
use_nvtx (bool, optional) – Add NVTX ranges to profiled regions for NVIDIA Nsight tools (default: False). Requires scope-profiler[nvtx].
use_gpu_timing (bool, optional) – Record CUDA-event elapsed device time for each profiled region (default: False). CPU timestamps are still recorded, so the normal timeline remains enqueue-side timing.
gpu_timing_backend (str or object, optional) – CUDA-event backend for use_gpu_timing: "auto", "torch", "cupy", or a custom object implementing record_event() and elapsed_time_ns(start_event, end_event).
deactivate_file_output (bool, optional) –
Write no HDF5 file at all (default: False), not even the run metadata. Use it with finalize(return_results=True) to analyse a run entirely in memory:
```
ProfileManager.setup(deactivate_file_output=True)
...
results = ProfileManager.finalize(return_results=True)
```
recursive_profile (bool, optional) – Enable recursive profiling for all decorated functions by default (default: False). This can be overridden per decorator with @ProfileManager.profile(..., recursive=...).
aggregation_mode (bool, optional) – Record only count, inclusive total, minimum, maximum, and exclusive total per region. Timeline events are unavailable in this mode; it cannot be combined with line, GPU, NVTX, or LIKWID profiling.
capture_region_source (bool, optional) –
Record where each region is defined – the with block or the decorated function – once per distinct source file, the first time any of its regions is created (default: False). See source_text. Off by default because the cost, while cheap for a typical file, is not always: it is one ast.parse + tree walk of that file, so it tracks the file’s total size, not the size or number of the regions in it – under a millisecond for a typical few-hundred- line file, but tenths of a second per rank for one containing thousands of lines across many regions. Every rank pays that independently, so it can compound to whole seconds under contention on a job with more ranks than idle cores (measured: ~0.3s/rank at 8 ranks, ~2.9s/rank at 64, for a single ~10,000-line file, on a shared/oversubscribed node):
```
ProfileManager.setup(capture_region_source=True)
```
buffer_limit (int, optional) – Initial number of profiling events preallocated per region (default: 1024). Buffers grow on demand, so this is a starting size rather than a limit; raise it for very hot regions to avoid repeated reallocation.
output_mode ({"auto", "direct", "parallel"}, optional) – MPI file writer. auto prefers MPI-enabled h5py when compatible with the active instrumentation and otherwise lets ranks append directly to one serial-HDF5 file in token order.
hdf5_compression ({"gzip", "lzf", "zstd"} or None, optional) – Compression filter for timestamp and GPU-duration datasets.
hdf5_compression_level (int or None, optional) – GZIP level 0–9 or Zstandard level 1–22.
hdf5_chunk_size (int or None, optional) – Maximum events per dataset chunk. Enables chunked partial reads even without compression.
config_path (str or os.PathLike, optional) – TOML file containing a [profiling] table with these settings. Values passed directly to setup() take precedence.

Notes

The run’s start time is the moment setup() is called; it is stored as the start_time_ns metadata field and is the origin of the relative timeline in post-processing. MPI is not configurable either: collectives are used exactly when the process was started by an MPI launcher, so a plain python script.py never imports mpi4py. See scope_profiler.mpi_launch for the detection and its SCOPE_PROFILER_MPI override.

classmethod set_config(config)#

Set a new profiling configuration and update the region class.

Parameters:: config (ProfilingConfig) – The new profiling configuration to apply.
Return type:: None

classmethod get_config()#

Get the current profiling configuration, creating a default one if setup() has not been called.

This is the only place a configuration comes into being outside setup(), and it is deliberately lazy: constructing one resolves the MPI communicator, which imports mpi4py and therefore calls MPI_Init in any process the launcher marked as a rank. Doing that at import time would mean import scope_profiler silently joins the MPI job – fatal in a process forked from a rank, which is exactly what the LIKWID counter read-back does.

Returns:: The current profiling configuration.
Return type:: ProfilingConfig

ProfilingConfig#

Holds one manager’s profiling configuration. Normally you interact with it through ProfileManager.setup(), but you can also construct one directly for advanced use cases.

class scope_profiler.profile_config.ProfilingConfig(file_path='profiling_data.h5', label=None, use_likwid=False, use_line_profiler=False, deactivate_profiling=False, use_nvtx=False, use_gpu_timing=False, gpu_timing_backend='auto', deactivate_file_output=False, recursive_profile=False, aggregation_mode=False, capture_region_source=False, buffer_limit=1024, output_mode='auto', hdf5_compression=None, hdf5_compression_level=None, hdf5_chunk_size=None)#

Configuration for one profiling manager.

This class centralizes configuration for LIKWID performance counters, buffer limits, and file paths. Each manager owns a separate instance, so independently configured profiling sessions can coexist. Constructing it is purely local: it reads the communicator for rank and size but issues no MPI call of its own, so setup() does not have to be collective.

Parameters:

file_path (str)
label (str | None)
use_likwid (bool)
use_line_profiler (bool)
deactivate_profiling (bool)
use_nvtx (bool)
use_gpu_timing (bool)
deactivate_file_output (bool)
recursive_profile (bool)
aggregation_mode (bool)
capture_region_source (bool)
buffer_limit (int)
output_mode (str)
hdf5_compression (str | None)
hdf5_compression_level (int | None)
hdf5_chunk_size (int | None)

classmethod reset()#

Compatibility no-op retained for callers of older releases.

Configurations are ordinary per-manager objects now, so there is no process-wide instance to reset.

pylikwid_markerinit()#: Initialize LIKWID markers if LIKWID is enabled.

pylikwid_markerclose()#

Close LIKWID markers to finalize measurement regions.

Idempotent: repeated calls (a second finalize(), say) do nothing.

collect_likwid_results(region_names)#

Close the LIKWID markers and return the run’s counter results.

Tries three sources, richest first, so a host where LIKWID cannot do the fancy parts still ends up with real numbers in the HDF5 file:

the perfmon read-back, run in a subprocess because it can crash the interpreter outright on hosts that cannot really count;
LIKWID’s marker file, parsed directly — real values, but no event names or derived metrics;
a marker-API snapshot taken before the markers were closed, for when there is no marker file at all.

See scope_profiler.likwid_data for why the first one is fenced off.

Parameters:: region_names (iterable of str) – Region names to snapshot via the marker API.
Returns:: Empty when LIKWID is disabled, the process was not started under likwid-perfctr -m / likwid-mpirun -marker, or the markers were already collected.
Return type:: list of scope_profiler.likwid_data.LikwidRegionResult

Notes

A process can only do this once. Closing the markers tears the marker API down, and querying it afterwards crashes the interpreter rather than raising, so a second finalize() returns nothing instead of reading again. The counters of the whole process therefore end up in the file written by the first finalize().

likwid_environment()#

Return the LIKWID_* environment variables of this process.

Return type:: dict

property comm: Intercomm | None#: MPI communicator or None if MPI is unavailable.

property deactivate_profiling: bool#: Return whether profiling is globally turned off.

property buffer_limit: int#: Initial per-region buffer capacity; buffers grow beyond it as needed.

property file_path: str#: Global output file path for combined profiling data.

property output_mode: str#

auto, direct, or parallel.

Type:: MPI HDF5 output strategy

property hdf5_compression: str | None#: Compression filter used for timestamp datasets.

property hdf5_compression_level: int | None#: Configured GZIP or Zstandard compression level.

property hdf5_chunk_size: int | None#: Maximum number of events per timestamp chunk.

property use_likwid: bool#: Return whether LIKWID profiling is enabled.

property use_line_profiler: bool#: Return whether line_profiler profiling is enabled.

property use_nvtx: bool#: Return whether NVTX annotations are enabled.

property use_gpu_timing: bool#: Return whether CUDA-event GPU timing is enabled.

property gpu_timing_backend#: CUDA-event timing backend selector or backend object.

property deactivate_file_output: bool#: Return whether the run writes no HDF5 file at all.

property recursive_profile: bool#: Return whether recursive decorator profiling is enabled by default.

property capture_region_source: bool#: Return whether a region’s defining source is captured at creation.

property aggregation_mode: bool#: Whether regions retain aggregates instead of individual events.

property paused: bool#: Whether runtime timing collection is temporarily suspended.

property start_time_ns: int#

The run’s start time (ns, perf_counter_ns clock).

The moment the configuration was created. Persisted as metadata and used as the timeline origin when reading the results back.

property label: str | None#: Short name for this run, or None if none was given to setup().

property metadata: dict#: Environment metadata collected on this rank (hostname, OpenMP threads, …).

Region classes#

BaseProfileRegion#

class scope_profiler.region_profiler.BaseProfileRegion(region_name, config, tags=())#

Base class providing shared profiling logic.

Handles start/end time buffering and call counting. The buffers grow on demand and are copied out once, at the end of the run, by ProfileManager.finalize() – regions never touch HDF5 themselves.

Parameters:

region_name (str)
config (ProfilingConfig)

region_name#

config#

tags#

ptr#

buffer_limit#

capacity#

start_times#

end_times#

source_file#

source_lineno#

source_text#

set_source(filename, lineno, text)#

Record where this region is defined, the first time it is called.

First writer wins: a region name reused at more than one call site keeps only the first location, matching how their timings are already pooled together under one name (see issue #161).

Return type:: None

property has_source: bool#: Whether this region’s call-site source was captured.

wrap(func)#

Wrap a function for profiling.

Subclasses must override this method to implement the appropriate profiling behavior.

append(start, end)#

Append a start/end time pair to the buffer, growing it if needed.

Parameters:

start (float)
end (float)

Return type:

None

property num_calls: int#

Times this region was entered, for the lifetime of the process.

Derived rather than counted: the slots in use (ptr) plus whatever earlier finalize() calls already copied out. Keeping it out of __enter__ removes an attribute write from every recorded call.

get_durations_numpy()#

Return durations (end - start) for buffered entries as a NumPy array.

Return type:: ndarray

open_slots()#

Buffer slots whose call is still running, in ascending order.

A slot is open until its end time is written, which is what distinguishes a call in flight from one that has returned. Both region forms are covered: the context manager writes its start on entry, the decorator writes nothing until the call returns.

Return type:: ndarray

closed_slots()#

Mask of buffered slots this finalize() should copy out.

A slot qualifies once its end time is written and it has not already gone out with an earlier finalize(). None – the usual case – means every slot qualifies and the caller can skip the mask entirely.

Return type:: ndarray | None

mark_written()#

Record that everything buffered so far has been handed to finalize().

Called by finalize() once the data has been copied out, so that a second run in the same process reports only its own events instead of re-reporting the first run’s. The timestamp buffer rewinds (the arrays are reused; anything past ptr is unread scratch), while num_calls keeps counting for the lifetime of the process — it is the in-memory view of the region, which callers inspect after finalize().

A call still running has a slot reserved that finalize() did not copy out, so it must survive the rewind. It is moved to the front of the buffer rather than pinning everything behind it: leaving the whole buffer in place would make the next finalize() re-report every completed call sitting below it, with a second set of call ids.

The one case that still cannot rewind is a call entered through the decorator form, which keeps its slot index in the wrapper’s own frame where nothing can remap it. Recognisable because such a slot is open without appearing in _scope_ptr_stack. There the buffer stays put and the slots already copied out are recorded instead, so the next finalize() skips them rather than reporting those calls again.

Return type:: None

get_end_times_numpy()#

Return end times offset by the run’s start time.

Return type:: ndarray

get_start_times_numpy()#

Return start times offset by the run’s start time.

Return type:: ndarray

add_function(func)#

Return type:: None

DisabledProfileRegion#

class scope_profiler.region_profiler.DisabledProfileRegion(region_name, config, tags=())#

Profiling region that performs no measurements.

Used when profiling is disabled but code paths must remain valid.

Parameters:

region_name (str)
config (ProfilingConfig)

wrap(func)#: Return the original function unchanged — no wrapper, no overhead.

append(start, end)#: Ignored: no data recorded.

get_durations_numpy()#: Return an empty array since nothing is recorded.

buffer_limit#

capacity#

config#

end_times#

ptr#

region_name#

source_file#

source_lineno#

source_text#

start_times#

tags#

TimeOnlyProfileRegion#

class scope_profiler.region_profiler.TimeOnlyProfileRegion(region_name, config, tags=())#

Region that records timing, collected once at the end of the run.

Parameters:

region_name (str)
config (ProfilingConfig)

wrap(func)#: Wrap a function to measure its execution time.

buffer_limit#

capacity#

config#

end_times#

ptr#

region_name#

source_file#

source_lineno#

source_text#

start_times#

tags#

FullProfileRegion#

class scope_profiler.region_profiler.FullProfileRegion(region_name, config, tags=())#

Region that records both timing and LIKWID metrics, and writes to HDF5.

This is the most complete profiling mode: users obtain LIKWID markers and nanosecond-resolution timing.

Parameters:

region_name (str)
config (ProfilingConfig)

likwid_marker_start#

likwid_marker_stop#

wrap(func)#: Wrap a function to measure time and collect LIKWID metrics.

LineProfilerRegion#

class scope_profiler.region_profiler.LineProfilerRegion(region_name, config, tags=())#

Region that records timing and line-by-line profiling via line_profiler.

Uses line_profiler to collect per-line execution statistics for decorated functions. Also records nanosecond timestamps.

Line-by-line profiling is most useful with the decorator (wrap) path, which automatically registers the function with the line profiler. When used as a context manager, the profiler is enabled/disabled around the block - any functions previously added via the decorator path will be profiled while the context is active.

Parameters:

region_name (str)
config (ProfilingConfig)

wrap(func)#: Wrap a function to measure execution time and collect line-by-line stats.

enter_frame(frame)#: Enter this region while registering frame with line_profiler.

enter_timing_only()#: Enter this region without registering or enabling line_profiler.

record_line_timing(frame, lineno, duration_ns)#

Record one line timing sample from recursive CLI tracing.

Parameters:

lineno (int)
duration_ns (int)

Return type:

None

manual_line_records(unit=1e-09)#

Return manually traced line timings in the persisted record shape.

Parameters:: unit (float)
Return type:: list

add_function(func)#

Return type:: None

print_stats()#: Print line-by-line profiling statistics.

get_stats()#: Return the line_profiler stats object.

Post-processing#

Everything in this section is importable from the package root:

from scope_profiler import (
    build_call_arrays,
    build_call_stack,
    read_h5,
    read_h5_summary,
    plot_flame_chart,
    plot_gantt,
    write_region_statistics_json,
)

All durations and timestamps exposed here are in seconds; the HDF5 file stores nanoseconds.

ProfilingResults#

The post-processing API itself, and the only type the analysis layer has. ProfilingResults.from_h5() builds one from a merged file, and ProfileManager.finalize(return_results=True) builds the same thing straight from memory.

class scope_profiler.results.ProfilingResults(regions, metadata=None, num_ranks=None, likwid=None, line_profile=None, file_path='', is_root=True, exclusive_totals=None, event_data_available=True)#

The profiling data of one run, as an ordered mapping of region name to MPIRegion:

results = ProfileManager.finalize(return_results=True)
for region in results:
    print(region.name, region.total_duration)

solve = results["solve"]        # same as results.get_region("solve")
solve[0].average_duration       # rank 0, in seconds

The same object comes back from a merged output file, via from_h5() (or its module-level twin read_h5()):

results = ProfilingResults.from_h5("profiling_data.h5")

All durations are reported in seconds.

Parameters:

regions (dict[str, MPIRegion])
metadata (dict | None)
num_ranks (int | None)
likwid (dict[int, dict[str, LikwidRegionResult]] | None)
line_profile (dict[int, list] | None)
file_path (str | Path)
is_root (bool)
exclusive_totals (dict[str, dict[int, int]] | None)
event_data_available (bool)

classmethod from_h5(file_path, verbose=False)#

Load a merged profiling file written by ProfileManager.finalize:

results = ProfilingResults.from_h5("profiling_data.h5")
results.print_summary()

scope_profiler.read_h5() is the same thing under a shorter name.

Parameters:

file_path (str | Path) – Path to the merged HDF5 file containing profiling data.
verbose (bool, optional) – Print each rank group as it is read (default: False).

Returns:

The run’s profiling data, of whichever class this was called on.

Return type:

ProfilingResults

Raises:

FileNotFoundError – If the specified HDF5 file does not exist.

get_region(region_name)#

Retrieve profiling data for a specific region.

Parameters:: region_name (str) – Name of the region to retrieve.
Returns:: Region object containing profiling data for all ranks.
Return type:: Region
Raises:: KeyError – If the specified region name does not exist.

property region_names: list[str]#: Names of all regions, in order of appearance.

summary(include=None, exclude=None)#

Summarize every region, aggregated over ranks.

Parameters:

include (list of str or str, optional) – Regex patterns selecting which regions to summarize, matched as in get_regions().
exclude (list of str or str, optional) – Regex patterns selecting which regions to summarize, matched as in get_regions().

Returns:

One dict per region (see get_summary()), ordered by first start time. inclusive_duration includes nested regions; exclusive_duration excludes them. Durations are in seconds.

Return type:

List[dict]

to_dataframe(include=None, exclude=None, per_rank=False)#

Return the region summaries as a pandas DataFrame.

Parameters:

include (list of str or str, optional) – Regex patterns selecting which regions to include, matched as in get_regions().
exclude (list of str or str, optional) – Regex patterns selecting which regions to include, matched as in get_regions().
per_rank (bool, optional) – If True, emit one row per (region, rank) with a rank column instead of one aggregated row per region (default: False).

Returns:

Region statistics, with durations in seconds.

Return type:

pandas.DataFrame

Raises:

ImportError – If pandas is not installed.

events(include=None, exclude=None, ranks=None, relative=True, origin=None)#

Return one dict per recorded call, across all regions and ranks.

This is the long-form (“tidy”) view to build custom plots from: one row per call rather than one per region.

Parameters:

include (list of str or str, optional) – Regex patterns selecting which regions to include, matched as in get_regions().
exclude (list of str or str, optional) – Regex patterns selecting which regions to include, matched as in get_regions().
ranks (list of int or int, optional) – Restrict to these ranks (default: all).
relative (bool, optional) – If True (default), timestamps are measured from time_origin — the start time registered by ProfileManager.setup(), or the first region entry for runs without one — so the timeline starts at zero. If False, the raw monotonic-clock timestamps are returned; those are only comparable within a single run.
origin (float, optional) – Measure from this timestamp instead, in seconds on the recording clock. Overrides relative; pass origin=results.minimum_start_time to zero the timeline on the first region entry regardless of what the run registered.

Returns:

Entries with keys name, rank, call_index, start, end and duration, in seconds, ordered by region (as in get_regions()) then rank then call order. Entries also carry gpu_duration when CUDA-event timing was enabled for that region.

Return type:

List[dict]

Examples

>>> for event in results.events(include="solve"):  
...     print(event["rank"], event["start"], event["duration"])

to_events_dataframe(include=None, exclude=None, ranks=None, relative=True, origin=None)#

Return every recorded call as a pandas DataFrame (one row per call).

Parameters:

include (list[str] | str | None) – As in events().
exclude (list[str] | str | None) – As in events().
ranks (list[int] | int | None) – As in events().
relative (bool) – As in events().
origin (float | None) – As in events().

Returns:

Columns name, rank, call_index, start, end and duration, with times in seconds. A gpu_duration column is included when any selected event has CUDA-event timing.

Return type:

pandas.DataFrame

Raises:

ImportError – If pandas is not installed.

call_stack(rank=0, include=None, exclude=None, relative=True, origin=None)#

Reconstruct the nested call stack for one rank.

Regions record no call graph, so nesting is recovered from timestamp containment - the same reconstruction the flame chart, the .prof export and the speedscope export use.

Parameters:

rank (int, optional) – Rank whose calls to reconstruct (default: 0).
include (list of str or str, optional) – Regex patterns selecting which regions to include, matched as in get_regions().
exclude (list of str or str, optional) – Regex patterns selecting which regions to include, matched as in get_regions().
relative (bool, optional) – If True (default), timestamps start at zero; see events().
origin (float, optional) – Measure from this timestamp instead; see events().

Returns:

One entry per call, parents before children, with keys call_id, name, start, end, duration, depth and parent (the enclosing call’s id, or None). See build_call_stack().

Return type:

List[dict]

call_graph(rank=0, include=None, exclude=None)#

Return call relationships without timestamps or durations.

New profiles persist call_id and parent_id for every Python event. Legacy profiles fall back to the timestamp-based call stack. The returned nodes contain only call_id, parent_id, name, call_index and depth.

Filtering renests: a call whose parent was excluded is reported under its nearest surviving ancestor, with the depth that implies, exactly as call_stack() does. Leaving the excluded id in parent_id would hand back a graph with edges pointing at nodes that are not in it.

call_id is unique within this rank, not across the file - each rank numbers its own calls.

Parameters:: rank (int)
Return type:: list[dict]

print_summary(include=None, exclude=None, ranks=None, sort='start', title=None, stream=None, suppress_notes=False, columns=None, percentage_mode='coverage')#

Print a region summary table, aggregated over ranks.

Renders the same table as scope-profiler inspect and the summary printed by ProfileManager.finalize().

Parameters:

include (list of str or str, optional) – Regex patterns selecting which regions to print, matched as in get_regions().
exclude (list of str or str, optional) – Regex patterns selecting which regions to print, matched as in get_regions().
ranks (list of int, optional) – Restrict the statistics to these ranks (default: all).
sort (str, optional) – Column to order by: start (default), total, calls, avg, min, max, std or name.
title (str, optional) – Heading above the table (default: the file path and rank count).
stream (file-like, optional) – Where to write (default: stdout).
columns (list of str or str, optional) – Region summary columns to print. Defaults to region, calls, percent, total and avg. The percentage is relative to scope_profiler.session. Use region for the region-name column.
percentage_mode ({"coverage", "exclusive"}, optional) – Quantity used for % session. Defaults to wall-clock coverage; use exclusive to attribute time after nested regions.
suppress_notes (bool)

Return type:

None

Notes

Does nothing on a non-root rank (see is_root), so a parallel script can call it unguarded and print the table once.

default_title()#

Heading naming this run, for a summary table.

The label leads when the run has one, since that is the name the user chose; the file path follows either way, because when several runs are being compared it is what tells them apart on disk.

Returns:: e.g. "results/run_a.h5 (128 ranks)".
Return type:: str

property file_path: Path#

The run’s output file.

Returns:: The file path as a pathlib.Path object. For results taken straight from memory this is the path setup() was configured with, which need not exist on disk.
Return type:: Path

property label: str | None#

The run’s label, as given to ProfileManager.setup(label=...).

Returns:: The label, or None for a run that was not given one. Use display_label to name the run in output regardless.
Return type:: str or None

property display_label: str#

What to call this run in charts, tables and exports.

The label when the run has one, and otherwise the stem of its output file — which is what post-processing named runs by before labels existed, and remains the default.

Returns:: A non-empty name for the run.
Return type:: str

property is_root: bool#

Whether this rank holds the run’s data.

Always True for a file that was read back, and for serial runs. Under MPI, finalize(return_results=True) gathers everything on rank 0, so only rank 0’s results are the root ones; the others come back empty and with this False.

Everything that produces output - print_summary(), the plot_* functions, the exporters - does nothing for non-root results. That is what lets a parallel script call them unguarded and still write each figure exactly once, from rank 0. Read it when a script needs to make the same distinction for output of its own.

Returns:: True unless this is a non-root rank’s share of an MPI run.
Return type:: bool

property has_event_data: bool#: Whether per-call timestamps are available to event-based APIs.

property metadata: dict#

Get environment metadata for the run (gathered from rank 0).

Returns:: Metadata dict (hostname, OpenMP thread count, platform, versions, etc.), or an empty dict if the run recorded none.
Return type:: dict

property line_profile: dict[int, list]#

Persisted line-profiler records keyed by rank.

Each record contains region, filename, function, first_lineno, line_numbers, hits, times and unit. The elapsed seconds for a line are times * unit.

property num_ranks: int#

Get the number of ranks recorded in the profiling data.

Returns:: Number of ranks.
Return type:: int

property has_likwid: bool#: Whether the run recorded LIKWID hardware counter results.

property likwid_ranks: list[int]#: Ranks that recorded LIKWID results, in ascending order.

get_likwid_regions(rank=None)#

Get the LIKWID marker results of the run.

Parameters:: rank (int, optional) – Return only this rank’s regions. By default every rank is included, keyed by rank.
Returns:: With rank given, a mapping of region tag to LikwidRegionResult; otherwise a mapping of rank to such a dict. Empty when the run did not use LIKWID.
Return type:: dict

Examples

results = read_h5("profiling_data.h5")
for rank, regions in results.get_likwid_regions().items():
    for tag, result in regions.items():
        for name, values in zip(result.metric_names, result.metrics):
            print(rank, tag, name, values)

get_likwid_region(tag, rank=0)#

Get one region’s LIKWID results for a single rank.

Parameters:

tag (str) – LIKWID marker region tag (the profiled region’s name).
rank (int, optional) – Rank to read from (default: 0).

Returns:

The region’s counters, event names and derived metrics.

Return type:

LikwidRegionResult

Raises:

KeyError – If the rank recorded no LIKWID data or has no such region.

likwid_to_dataframe()#

Return every LIKWID event and metric as a tidy pandas DataFrame.

One row per (rank, region, hardware thread), with a column per event and per derived metric, plus the region’s LIKWID runtime and call count. Regions measured with different event groups simply leave the other group’s columns empty.

Returns:: Empty if there is no LIKWID data.
Return type:: pandas.DataFrame
Raises:: ImportError – If pandas is not installed.

print_likwid_summary(stream=None)#

Print every LIKWID region’s events and derived metrics.

Parameters:: stream (file-like, optional) – Destination (default: stdout).
Return type:: None

property minimum_start_time: float#

Get the minimum start time across all regions and ranks.

This is the origin of the timeline: subtract it from any timestamp to get seconds since the first region entry. Regions with no recorded calls are ignored; the result is 0.0 if no region recorded any.

Returns:: Minimum start time in seconds.
Return type:: float

property run_start_time: float | None#

When the run started, in seconds on the recording clock.

This is the start_time_ns metadata field, written by ProfileManager.setup() — by default the moment setup() was called, or an earlier instant if one was passed to it.

Returns:: The registered start time, or None for runs without one (older files, or runs that never called setup()). Use startup_time for the elapsed time before the first region, which stays defined either way.
Return type:: float or None

property time_origin: float#

Zero point of the relative timeline, in seconds.

The registered run_start_time when there is one, and otherwise the first region entry (minimum_start_time). This is what events() and call_stack() measure from.

Note the plot_* functions instead frame their x axis on the first region entry, so that a long gap between setup() and the first region does not fill a chart with empty space. Pass origin=results.minimum_start_time to events() to reproduce the numbers on a chart’s axis.

Returns:: Origin timestamp on the recording clock.
Return type:: float

property startup_time: float#

Seconds between the start of the run and the first profiled region.

Time the instrumentation never saw: imports, reading input, building a mesh. Zero when there is no registered start time (run_start_time is None), since the run is then only known from its first region onwards.

Returns:: Elapsed time before the first region entry, in seconds.
Return type:: float

property maximum_end_time: float#

Get the maximum end time across all regions and ranks.

Returns:: Maximum end time in seconds, or 0.0 if no region recorded timing.
Return type:: float

property time_span: float#

Wall-clock seconds between the first region entry and the last exit.

Returns:: Duration of the profiled window in seconds, or 0.0 if no region recorded timing.
Return type:: float

property finalize_time: float | None#

When finalize() was called, in seconds on the recording clock.

This is the finalize_time_ns metadata field, read as the first thing ProfileManager.finalize() does – before it spends any time collecting or writing the run’s data, so it marks the moment finalize() was reached rather than the moment it returned.

Returns:: The registered finalize time, or None for runs without one (older files, or a run that never reached finalize()).
Return type:: float or None

property total_time: float | None#

Wall-clock seconds from setup() to finalize().

Unlike time_span (first region entry to last exit), this covers the whole instrumented program: startup work before the first region, gaps between regions, and any teardown after the last one but before finalize() is called – the number to report as “how long did the run take” alongside the region breakdown.

Returns:: finalize_time minus run_start_time, or None if either is missing – an older file, or a run that set up profiling (or called finalize) some other way than ProfileManager.setup()/finalize().
Return type:: float or None

get_regions(include=None, exclude=None)#

Get a list of all regions in order of appearance.

Returns:

List of Region objects.

Return type:

List[Region]

Parameters:

include (list[str] | str | None)
exclude (list[str] | str | None)

read_h5#

The module-level spelling of ProfilingResults.from_h5(); the two are interchangeable.

scope_profiler.h5reader.read_h5(file_path, verbose=False)#

Load a merged profiling file for post-processing.

The discoverable spelling of ProfilingResults.from_h5; the two are interchangeable:

from scope_profiler import read_h5

results = read_h5("profiling_data.h5")
results.print_summary()

solve = results["solve"]        # same as results.get_region("solve")
solve[0].average_duration       # rank 0, in seconds

Parameters:

file_path (str | Path) – Path to the merged HDF5 file containing profiling data.
verbose (bool, optional) – Print each rank group as it is read (default: False).

Returns:

The run’s profiling data. All durations are reported in seconds.

Return type:

ProfilingResults

Raises:

FileNotFoundError – If the specified HDF5 file does not exist.

read_h5_summary#

Load fixed-size scalar statistics without reading per-call event columns.

scope_profiler.h5reader.read_h5_summary(file_path, verbose=False, *, fallback=True, include_likwid=True, include_line_profile=True, regions=None, ranks=None)#

Load fixed-size statistics, optionally falling back for older files.

The returned object supports metadata and scalar region statistics. Event, timeline, call-stack and percentile APIs intentionally have no per-call data; use read_h5() when those are required.

Parameters:

file_path (str | Path)
verbose (bool)
fallback (bool)
include_likwid (bool)
include_line_profile (bool)
regions (str | list[str] | tuple[str, ...] | None)
ranks (int | list[int] | tuple[int, ...] | None)

Return type:

ProfilingResults

Native (C and Fortran) traces#

Reading what the C and Fortran region APIs write – they share one format; see Profiling C and C++ code and Profiling Fortran code.

scope_profiler.native_trace.load_traces(inputs, label=None)#

Read Fortran traces into the standard post-processing API.

Parameters:

inputs (path or sequence of paths) – Trace files and/or directories containing them (see find_traces()).
label (str, optional) – Name for the run in summaries, charts and exports.

Returns:

The same object a Python run produces, so every summary, plot and exporter works on it unchanged.

Return type:

ProfilingResults

Raises:

TraceFormatError – If two trace files claim the same rank.

scope_profiler.native_trace.convert_traces(inputs, output_path, label=None)#

Convert Fortran traces into a standard scope-profiler HDF5 file.

The result is indistinguishable from one a Python run wrote, so scope-profiler plot / inspect and read_h5() work on it directly.

Parameters:

inputs (path or sequence of paths) – Trace files and/or directories containing them.
output_path (str or Path) – HDF5 file to write.
label (str, optional) – Name for the run; defaults to the output file’s stem.

Returns:

The file that was written.

Return type:

Path

scope_profiler.native_trace.read_trace(path)#

Read one rank’s trace file.

Parameters:: path (str or Path) – A .spt file written by sp_finalize().
Returns:: (rank, regions), where regions maps a region name to (start_times, end_times) int64 arrays in nanoseconds – exactly the shape RankPayload carries.
Return type:: tuple
Raises:: TraceFormatError – If the file is not a trace, is truncated, or was written by a newer format version.

scope_profiler.native_trace.find_traces(inputs)#

Collect trace files from paths, directories, or a mix of both.

Parameters:: inputs (path or sequence of paths) – Files to read, and/or directories to search (non-recursively) for *.spt.
Returns:: The trace files, sorted, with duplicates removed.
Return type:: list of Path
Raises:: FileNotFoundError – If an input does not exist, or a directory holds no trace files.

scope_profiler.native_trace.write_results(results, output_path)#

Write any ProfilingResults out as a standard HDF5 file.

Goes through ProfilingWriter, so the result has exactly the layout a Python run produces – which is what lets an imported (or merged) run be read back by read_h5() and fed to every plot and exporter.

Parameters:

results (ProfilingResults) – The run to write.
output_path (str or Path) – HDF5 file to create.

Returns:

The file that was written.

Return type:

Path

scope_profiler.native_trace.fortran_source_path()#

Path to scope_profiler.f90, the module to compile into your program.

It ships with the package, so this works from an installed wheel:

gfortran -c $(python -c             "import scope_profiler.native_trace as t; print(t.fortran_source_path())")

Returns:: The Fortran module source.
Return type:: Path

scope_profiler.native_trace.c_source_path()#

Path to scope_profiler.c, the implementation to compile in.

Its header sits next to it; c_include_dir() is what to put on the compiler’s include path:

cc -c $(python -c             "import scope_profiler.native_trace as t; print(t.c_source_path())")            -I$(python -c             "import scope_profiler.native_trace as t; print(t.c_include_dir())")

Returns:: The C source file.
Return type:: Path

scope_profiler.native_trace.c_include_dir()#

Directory holding scope_profiler.h, for the compiler’s -I.

Returns:: The include directory.
Return type:: Path

Combining runs#

scope_profiler.results.merge_results(*result_sets, label=None, file_path=None)#

Combine several result sets into one.

The case this exists for is a mixed-language run: a Python driver records its own regions while the Fortran (or other) code it calls records theirs, and what the user wants at the end is one profile covering both. Ranks line up by number, so rank 3’s Python regions and rank 3’s Fortran regions end up side by side.

Parameters:

*result_sets (ProfilingResults) – The sets to combine. Non-root sets (the empty ones every rank but 0 gets back under MPI) are ignored, so this can be called unguarded in a parallel script.
label (str, optional) – Name for the combined run. Defaults to the first set’s label.
file_path (str or Path, optional) – Output path to attribute the combined set to. Defaults to the first set’s.

Returns:

One result set holding every region of every input.

Return type:

ProfilingResults

Raises:

ValueError – If no result sets were given, or if a region name appears in more than one of them. Merging same-named regions would silently double-count a Python wrapper and the native region inside it, so the collision has to be resolved by the caller – name the regions apart, for instance with a "fortran:" prefix.

Region#

class scope_profiler.region.Region(start_times, end_times, gpu_durations=None, call_ids=None, parent_ids=None, source_file=None, source_lineno=None, source_text=None, tags=(), aggregate=None, event_data_available=True)#

Timing data for one region on one rank.

All duration and timestamp properties are reported in seconds; the underlying HDF5 data is stored in nanoseconds.

Parameters:

start_times (ndarray)
end_times (ndarray)
gpu_durations (ndarray | None)
call_ids (ndarray | None)
parent_ids (ndarray | None)
source_file (str | None)
source_lineno (int | None)
source_text (str | None)
aggregate (dict | None)
event_data_available (bool)

get_summary()#

Return a summary of the region’s statistics as a dictionary.

Returns:: Dictionary containing statistics: num_calls, total_duration, average_duration, min_duration, max_duration, first_duration, last_duration, and std_duration. Durations are in seconds.
Return type:: Dict[str, Any]

events(origin=0.0)#

Return one dict per recorded call.

Parameters:: origin (float, optional) – Seconds subtracted from every timestamp, so passing results.minimum_start_time yields a timeline starting at zero (default: 0.0, i.e. raw timestamps).
Returns:: One entry per call with keys call_index, start, end and duration, in seconds and in recorded order.
Return type:: list of dict

property call_ids#: Explicit call ids, or None for legacy profiles.

property parent_ids#: Explicit parent ids, or None for legacy profiles.

property has_timing: bool#: Whether this region recorded any calls at all.

property has_event_data: bool#: Whether per-call timestamps were loaded for this region.

property has_source: bool#: Whether this region’s call-site source was captured.

property has_gpu_timing: bool#: Whether this region has CUDA-event elapsed timings.

property stored_summary: dict | None#: Fixed-size statistics used by aggregate and summary-only results.

property source_file: str | None#: Path of the file this region is defined in, or None if not captured.

property source_lineno: int | None#: Line the region’s call site starts at, or None if not captured.

property source_text: str | None#

Source text of the region’s with block or decorated function.

None if it was not captured – either the file it came from is no longer readable, or the file predates this being recorded.

property tags: tuple[str, ...]#: User-defined tags attached to this region.

property start_times_ns: ndarray#: Start times of all calls in nanoseconds, exactly as stored.

property end_times_ns: ndarray#: End times of all calls in nanoseconds, exactly as stored.

property durations_ns: ndarray#: Duration of all calls in nanoseconds, as integers.

property gpu_durations_ns: ndarray | None#: CUDA-event elapsed device times in nanoseconds, or None if absent.

property inclusive_durations_ns: ndarray#: Inclusive duration of every call in nanoseconds.

property exclusive_durations_ns: ndarray#: Exclusive duration of every call in nanoseconds.

property start_times: ndarray#: Start times of all calls in seconds.

property first_start_time: float#: First start time in seconds.

property last_end_time: float#: Last end time in seconds.

property end_times: ndarray#: End times of all calls in seconds.

property durations: ndarray#: Duration of all calls in seconds.

property gpu_durations: ndarray | None#: CUDA-event elapsed device times in seconds, or None if absent.

property inclusive_durations: ndarray#: Inclusive duration of every call in seconds.

property exclusive_durations: ndarray#: Exclusive duration of every call in seconds.

property num_calls: int#: Number of recorded calls.

property total_duration: float#: Total time spent in this region in seconds (sum of all durations).

property gpu_total_duration: float | None#: Total CUDA-event elapsed device time in seconds, or None if absent.

property inclusive_duration: float#: Total inclusive time, including nested regions, in seconds.

property total_exclusive_duration: float#: Total time excluding nested regions, in seconds.

property exclusive_duration: float#: Alias for total_exclusive_duration.

property average_duration: float#: Average duration per call in seconds.

property gpu_average_duration: float | None#: Average CUDA-event elapsed device time in seconds, or None if absent.

property min_duration: float#: Minimum duration among all calls in seconds.

property max_duration: float#: Maximum duration among all calls in seconds.

property first_duration: float#: Duration of the first recorded call, in seconds.

property last_duration: float#: Duration of the last recorded call, in seconds.

property std_duration: float#: Standard deviation of durations in seconds.

percentile_duration(percentile)#

Return a duration percentile in seconds.

percentile follows numpy.percentile() and must be between 0 and 100. Empty regions return 0.0 for consistency with the other duration statistics.

Parameters:: percentile (float)
Return type:: float | None

property p50_duration: float | None#: Median duration in seconds.

property p95_duration: float | None#: 95th-percentile duration in seconds.

property p99_duration: float | None#: 99th-percentile duration in seconds.

MPIRegion#

class scope_profiler.mpi_region.MPIRegion(name, regions)#

One named region across all ranks that recorded it.

Indexing gives the per-rank Region (region[0]), while the properties on this class aggregate over every rank. All durations are in seconds.

Parameters:

name (str)
regions (dict[int, Region])

property name: str#: Name of the region.

property regions: dict[int, Region]#: Dictionary of rank IDs to their corresponding Region objects.

property ranks: list[int]#: Sorted list of ranks that recorded this region.

property has_timing: bool#: Whether any rank recorded timestamps for this region.

property has_event_data: bool#: Whether every represented rank has per-call timestamps loaded.

property has_source: bool#: Whether any rank captured this region’s call-site source.

property has_gpu_timing: bool#: Whether any rank recorded CUDA-event timings for this region.

property source_file: str | None#: Path of the file this region is defined in, or None if not captured.

property source_lineno: int | None#: Line the region’s call site starts at, or None if not captured.

property source_text: str | None#: Source text of the region’s with block or decorated function.

property tags: tuple[str, ...]#: User-defined tags attached to this region.

get_summary()#

Return statistics aggregated over every rank.

Returns:: Dictionary with name, num_ranks, num_calls (summed over ranks) and duration statistics in seconds, pooled over all calls on all ranks.
Return type:: Dict[str, Any]

events(ranks=None, origin=0.0)#

Return one dict per recorded call, on every rank.

This is the long-form view custom plots and dataframes want: each entry is a single call rather than a per-region aggregate.

Parameters:

ranks (list of int or int, optional) – Restrict to these ranks (default: all ranks that recorded the region). Ranks without data for this region are skipped.
origin (float, optional) – Seconds subtracted from every timestamp, so passing results.minimum_start_time yields a timeline starting at zero (default: 0.0, i.e. raw timestamps).

Returns:

Entries with keys name, rank, call_index, start, end and duration, in seconds, ordered by rank and then by call order.

Return type:

list of dict

property durations: ndarray#

Get every recorded call duration on every rank, in seconds.

Returns:: Pooled durations. Empty if no rank recorded timing.
Return type:: np.ndarray

property gpu_durations: ndarray | None#: CUDA-event elapsed device times pooled across ranks, or None if absent.

property inclusive_durations: ndarray#: Inclusive durations pooled across ranks, in seconds.

property exclusive_durations: ndarray#: Exclusive durations pooled across ranks, in seconds.

property num_calls: int#

Total number of calls summed over all ranks.

Returns:: Summed call count.
Return type:: int

num_calls_per_rank()#

Get the call count for each rank.

Returns:: Dictionary mapping rank IDs to their call counts.
Return type:: Dict[int, int]

average_durations()#

Get the average duration for each rank.

Returns:: Dictionary mapping rank IDs to their average durations.
Return type:: Dict[int, float]

min_durations()#

Get the minimum duration for each rank.

Returns:: Dictionary mapping rank IDs to their minimum durations.
Return type:: Dict[int, float]

max_durations()#

Get the maximum duration for each rank.

Returns:: Dictionary mapping rank IDs to their maximum durations.
Return type:: Dict[int, float]

total_durations()#

Get the total duration for each rank.

Returns:: Dictionary mapping rank IDs to their total durations.
Return type:: Dict[int, float]

property total_duration: float#

Get the total duration summed over all ranks and calls.

Returns:: Total duration in seconds.
Return type:: float

property gpu_total_duration: float | None#: Total CUDA-event elapsed device time across ranks, or None if absent.

property inclusive_duration: float#: Total inclusive time across ranks, in seconds.

property total_exclusive_duration: float#: Total time excluding nested regions, across ranks, in seconds.

property exclusive_duration: float#: Alias for total_exclusive_duration.

property average_duration: float#

Get the mean duration over every call on every rank.

Note this pools all calls, so ranks with more calls weigh more heavily; use average_durations() for the per-rank breakdown.

Returns:: Average duration in seconds, or 0.0 if no timing was recorded.
Return type:: float

property gpu_average_duration: float | None#: Average CUDA-event elapsed device time across ranks, or None if absent.

property std_duration: float#

Get the standard deviation over every call on every rank.

Returns:: Standard deviation in seconds, or 0.0 if no timing was recorded.
Return type:: float

percentile_duration(percentile)#

Return a pooled duration percentile across all ranks.

Parameters:: percentile (float)
Return type:: float | None

property p50_duration: float | None#: Median duration across all ranks, in seconds.

property p95_duration: float | None#: 95th-percentile duration across all ranks, in seconds.

property p99_duration: float | None#: 99th-percentile duration across all ranks, in seconds.

property rank_imbalance: float#

Maximum per-rank total divided by the mean per-rank total.

A value of 1.0 means perfectly balanced ranks. Values are 0.0 when no calls were recorded or only one rank has timing data.

property rank_imbalance_pct: float#: Excess of the slowest rank over the mean, as a percentage.

property min_duration: float#

Get the minimum duration across all ranks.

Returns:: The minimum duration among all ranks, in seconds.
Return type:: float

property max_duration: float#

Get the maximum duration across all ranks.

Returns:: The maximum duration among all ranks, in seconds.
Return type:: float

property first_duration: float#

Get the duration of the call that started earliest across all ranks.

Returns:: Duration in seconds of the chronologically first call, or 0.0 if no rank recorded timing.
Return type:: float

property last_duration: float#

Get the duration of the call that ended latest across all ranks.

Returns:: Duration in seconds of the chronologically last call, or 0.0 if no rank recorded timing.
Return type:: float

property first_start_time: float#

Get the earliest start time across all ranks.

Returns:: The earliest start time among all ranks, in seconds, or 0.0 if no rank recorded timing.
Return type:: float

property last_end_time: float#

Get the latest end time across all ranks.

Returns:: The latest end time among all ranks, in seconds, or 0.0 if no rank recorded timing.
Return type:: float

Call stack reconstruction#

Regions record no call graph, so nesting is recovered from timestamp containment. ProfilingResults.call_stack() is the usual entry point; the functions below operate on its result and let you walk the reconstructed tree when building your own nested visualisation.

scope_profiler.call_stack.build_call_stack(regions, rank, origin=0.0)#

Reconstruct per-call nesting for one rank, one dict per call.

A convenience wrapper over build_call_arrays() for callers that want to iterate calls rather than columns. It allocates a dict per call (~700 bytes), so prefer the arrays for anything that has to scale with a long run.

Parameters:

regions (iterable of MPIRegion) – Regions to include, e.g. results.get_regions().
rank (int) – Rank whose calls to reconstruct. Regions with no data for this rank are skipped.
origin (float, optional) – Seconds subtracted from every timestamp, so passing results.minimum_start_time yields a timeline starting at zero (default: 0.0, i.e. raw timestamps).

Returns:

One dict per call, ordered by start time (parents before children), with keys call_id, name, call_path, start, end, duration (the inclusive duration), inclusive_duration, exclusive_duration (seconds), depth (0 for a top-level call) and parent (the call_id of the enclosing call in this same list, or None). call_path joins a call and each of its ancestors with " > ", keeping same-named regions at different call sites distinct. A color key carries whatever the plotting code assigned to the region. call_id is stable for the returned list and is unique within this rank/stack reconstruction.

Return type:

list of dict

Raises:

NestingError – If the intervals are not properly nested.

Notes

Calls are indexed by position in the returned list rather than by name, because a region called more than once - or recursively - contributes several calls under one name.

scope_profiler.call_stack.call_stack_roots(calls)#

Indices of the top-level calls in a build_call_stack() result.

Parameters:: calls (list[dict])
Return type:: list[int]

scope_profiler.call_stack.call_stack_children(calls)#

Child indices per call, for walking a build_call_stack() result.

Returns a list parallel to calls: entry i holds the indices of the calls directly nested inside call i, in start-time order.

Parameters:: calls (list[dict])
Return type:: list[list[int]]

Intervals must be properly nested#

Two calls recorded on one rank either nest completely or do not overlap at all. A call that starts inside another and ends after it raises NestingError rather than being reconstructed under a guessed parent — there is no call stack that describes it, and every consumer (flame chart, .prof, speedscope, exclusive time) would otherwise have to invent one.

A stack of with blocks or decorated functions always satisfies this. The realistic way to violate it is a mismatched sp_begin/sp_end pair in C or Fortran; see the C and Fortran guides.

finalize() is deliberately more forgiving than the readers: a rank whose intervals do not nest keeps its timings and loses only the call graph, with a warning, rather than throwing away a run that has already finished computing.

exception scope_profiler.call_stack.NestingError#

Raised when recorded intervals are not properly nested.

A rank’s calls must form a forest: any two intervals are either disjoint or one contains the other. Partial overlap has no call stack to reconstruct, and every consumer of this module (flame chart, .prof, speedscope, exclusive time) would have to invent one.

In practice this means a mismatched sp_begin/sp_end pair in C or Fortran, or region objects driven from several threads, which the region buffers do not support anyway.

Reconstruction at scale#

build_call_stack allocates a dict per call (~700 bytes), which is fine for inspecting a run and not fine for a long one. build_call_arrays returns the same reconstruction as numpy columns and is what the exporters and finalize() use — a run with ten million events per rank reconstructs in a couple of seconds instead of a couple of minutes.

scope_profiler.call_stack.build_call_arrays(regions, rank)#

Reconstruct one rank’s nesting as numpy arrays.

The workhorse behind build_call_stack(), and what anything processing a whole run should call: it never builds a dict per call, so it stays usable on the tens of millions of events a long simulation produces.

Parameters:

regions (iterable of MPIRegion) – Regions to include, e.g. results.get_regions(). Regions with no data for rank are skipped.
rank (int) – Rank whose calls to reconstruct.

Return type:

CallArrays

Raises:

NestingError – If any interval ends before it starts, or two intervals overlap without one containing the other.

class scope_profiler.call_stack.CallArrays(names, source_files, source_lines, region_index, call_index, start_ns, end_ns, depth, parent, exclusive_ns)#

One rank’s reconstructed nesting, in start-sorted (call_id) order.

Every array is indexed by call_id, so parent indexes straight back into the same arrays. region_index and call_index point at where a call came from: region names[region_index[i]], call number call_index[i] within that region’s buffers.

Parameters:

names (list[str])
source_files (list[str | None])
source_lines (list[int | None])
region_index (ndarray)
call_index (ndarray)
start_ns (ndarray)
end_ns (ndarray)
depth (ndarray)
parent (ndarray)
exclusive_ns (ndarray)

names: list[str]#: Alias for field number 0

source_files: list[str | None]#: Alias for field number 1

source_lines: list[int | None]#: Alias for field number 2

region_index: ndarray#: Alias for field number 3

call_index: ndarray#: Alias for field number 4

start_ns: ndarray#: Alias for field number 5

end_ns: ndarray#: Alias for field number 6

depth: ndarray#: Alias for field number 7

parent: ndarray#: Alias for field number 8

exclusive_ns: ndarray#: Alias for field number 9

Plotting#

scope_profiler.plotting_scripts.plot_gantt(profiling_data, ranks=None, include=None, exclude=None, filepath=None, show=False, verbose=True, cmap='tab20', data_filepath=None, data_format='csv', backend='matplotlib', return_fig=False, min_duration=0.0, start_time=None, end_time=None, aggregate_calls=1, collapse_depth=None)#

Plot a Gantt chart of all (or selected) regions with per-rank lanes using maxplotlib.

Parameters:

backend (str) – Backend to use for rendering: “matplotlib” (default) or “plotly”.
return_fig (bool) – Return the rendered figure instead of the default None. Matplotlib returns (fig, axes); Plotly returns its figure object.
profiling_data (ProfilingResults | Sequence[ProfilingResults])
ranks (list[int] | int | None)
include (list[str] | str | None)
exclude (list[str] | str | None)
filepath (str | None)
show (bool)
verbose (bool)
cmap (str)
data_filepath (str | Path | None)
data_format (str)
min_duration (float)
start_time (float | None)
end_time (float | None)
aggregate_calls (int)
collapse_depth (int | None)

Return type:

object | None

scope_profiler.plotting_scripts.plot_flame_chart(profiling_data, ranks=None, include=None, exclude=None, filepath=None, show=False, verbose=True, cmap='inferno', data_filepath=None, data_format='csv', backend='matplotlib', return_fig=False)#

Plot a flame chart reconstructing the call stack from region timings.

Parameters:

backend (str) – Backend to use for rendering: “matplotlib” (default) or “plotly”.
return_fig (bool) – Return the rendered figure instead of the default None. Matplotlib returns (fig, axes); Plotly returns its figure object.
profiling_data (ProfilingResults | Sequence[ProfilingResults])
ranks (list[int] | int | None)
include (list[str] | str | None)
exclude (list[str] | str | None)
filepath (str | None)
show (bool)
verbose (bool)
cmap (str)
data_filepath (str | Path | None)
data_format (str)

Return type:

object | None

scope_profiler.plotting_scripts.plot_flame_graph(profiling_data, ranks=None, include=None, exclude=None, filepath=None, show=False, verbose=True, cmap='inferno', data_filepath=None, data_format='csv', backend='matplotlib', return_fig=False)#

Plot an aggregated flame graph whose x-axis represents total time.

Parameters:

profiling_data (ProfilingResults | Sequence[ProfilingResults])
ranks (list[int] | int | None)
include (list[str] | str | None)
exclude (list[str] | str | None)
filepath (str | None)
show (bool)
verbose (bool)
cmap (str)
data_filepath (str | Path | None)
data_format (str)
backend (str)
return_fig (bool)

Return type:

object | None

scope_profiler.plotting_scripts.plot_durations(profiling_data, ranks=None, include=None, exclude=None, labels=None, metric='total', sort_by=None, top_n=None, combine_regions=None, stack_children=False, filepath=None, show=False, verbose=True, cmap='tab20', log_scale=False, data_filepath=None, data_format='csv', backend='matplotlib', return_fig=False)#

Plot duration bar charts for one or more profiling files using maxplotlib.

Parameters:

combine_regions (dict[str, list[str] | str], optional) – Merge several regions into a single bar, e.g. {"setup": ["setup: .*"]} combines every setup: ... region into one bar named “setup”, pooling their calls the same way sort_by and the other duration statistics pool a single region’s calls. Each value is one or more regex patterns (matched like include); a region matching several groups is claimed by whichever group is listed first.
metric (str) – Duration metric to render (avg, min, max or total).
stack_children (bool) – Split each bar into the region’s own (exclusive) time plus one segment per region called directly from it, stacked on top of each other, so a bar shows where its time went instead of only how much there was. The nesting comes from timestamp containment, the same reconstruction the flame chart uses, over every region in the run – a child that is filtered out of the bars still gets its own segment. Only total and avg decompose this way; min/max are rejected. Colors then identify segments rather than runs, so with several runs the bars are still grouped per region in run order but the legend names the segments.
backend (str) – Backend to use for rendering: “matplotlib” (default) or “plotly”.
return_fig (bool) – Return the rendered figure instead of the saved filepath list.
profiling_data (ProfilingResults | Sequence[ProfilingResults])
ranks (list[int] | int | None)
include (list[str] | str | None)
exclude (list[str] | str | None)
labels (Sequence[str] | None)
sort_by (str | None)
top_n (int | None)
filepath (str | None)
show (bool)
verbose (bool)
cmap (str)
log_scale (bool)
data_filepath (str | Path | None)
data_format (str)

Returns:

List of filepaths that were written (empty if filepath is None).

Return type:

list[str]

scope_profiler.plotting_scripts.plot_duration_timeseries(profiling_data, ranks=None, include=None, exclude=None, filepath=None, show=False, verbose=True, cmap='tab20', log_scale=False, data_filepath=None, data_format='csv', backend='matplotlib', return_fig=False)#

Plot each region’s call duration over wall-clock time, with a min-max band.

One line per region tracks the mean duration over the ranks that recorded each call, shaded between the minimum and maximum duration seen across those ranks, so rank imbalance shows up as a widening band.

Parameters:

backend (str) – Backend to use for rendering: “matplotlib” (default) or “plotly”.
return_fig (bool) – Return the rendered figure instead of the default None. Matplotlib returns (fig, axes); Plotly returns its figure object.
profiling_data (ProfilingResults | Sequence[ProfilingResults])
ranks (list[int] | int | None)
include (list[str] | str | None)
exclude (list[str] | str | None)
filepath (str | None)
show (bool)
verbose (bool)
cmap (str)
log_scale (bool)
data_filepath (str | Path | None)
data_format (str)

Return type:

None

scope_profiler.plotting_scripts.plot_speedup(profiling_data, x_field='num_ranks', ranks=None, include=None, exclude=None, filepath=None, show=False, verbose=True, cmap='tab20', data_filepath=None, data_format='csv', backend='matplotlib', return_fig=False)#

Plot scope speedup versus a chosen parallelism/metadata field using maxplotlib.

Parameters:

backend (str) – Backend to use for rendering: “matplotlib” (default) or “plotly”.
profiling_data (ProfilingResults | Sequence[ProfilingResults])
x_field (str)
ranks (list[int] | int | None)
include (list[str] | str | None)
exclude (list[str] | str | None)
filepath (str | None)
show (bool)
verbose (bool)
cmap (str)
data_filepath (str | Path | None)
data_format (str)
return_fig (bool)

Return type:

object | None

Statistics export#

scope_profiler.plotting_scripts.collect_region_statistics(profiling_data, ranks=None, include=None, exclude=None, labels=None)#

Collect aggregate region-duration statistics for one or more profiling files.

Parameters:

profiling_data (ProfilingResults | Sequence[ProfilingResults])
ranks (list[int] | int | None)
include (list[str] | str | None)
exclude (list[str] | str | None)
labels (Sequence[str] | None)

Return type:

dict

scope_profiler.plotting_scripts.write_region_statistics_json(profiling_data, filepath, ranks=None, include=None, exclude=None, labels=None)#

Write aggregate region-duration statistics to a JSON file.

Parameters:

profiling_data (ProfilingResults | Sequence[ProfilingResults])
filepath (str | Path)
ranks (list[int] | int | None)
include (list[str] | str | None)
exclude (list[str] | str | None)
labels (Sequence[str] | None)

Return type:

dict

Exporting to other tools#

scope_profiler.speedscope_export.export_speedscope(profiling_data, filepath, ranks=None, include=None, exclude=None, verbose=True)#

Write speedscope JSON files for the selected ranks.

One file is written per input HDF5 file, holding one profile per rank: unlike .prof, the format carries several profiles per file, and speedscope switches between them from its profile selector.

Parameters:

profiling_data (ProfilingResults | Sequence[ProfilingResults]) – The run(s) to export: file runs, in-memory results from ProfileManager.finalize(return_results=True), or a mix.
filepath (str | Path) – Base output path, e.g. figures/profile.speedscope.json. The input file’s stem is appended when more than one file is exported.
ranks (list[int] | int, optional) – Ranks to export (default: rank 0 only).
include (list[str] | str, optional) – Region name filters, as for the plotting functions.
exclude (list[str] | str, optional) – Region name filters, as for the plotting functions.
verbose (bool)

Returns:

The files written, in the order they were written.

Return type:

list[Path]

scope_profiler.prof_export.export_prof(profiling_data, filepath, ranks=None, include=None, exclude=None, call_paths=True, verbose=True)#

Write per-rank .prof files readable by pstats/snakeviz.

Parameters:

profiling_data (ProfilingResults | Sequence[ProfilingResults]) – The run(s) to export: file runs, in-memory results from ProfileManager.finalize(return_results=True), or a mix.
filepath (str | Path) – Base output path, e.g. figures/profile.prof. A _rank<N> suffix is appended per rank (and the input file’s stem too, when more than one file is exported), since .prof has no notion of ranks or runs.
ranks (list[int] | int, optional) – Ranks to export (default: rank 0 only).
include (list[str] | str, optional) – Region name filters, as for the plotting functions.
exclude (list[str] | str, optional) – Region name filters, as for the plotting functions.
call_paths (bool, optional) – Preserve each reconstructed parent/child path as a separate node in the exported pstats tree. This makes SnakeViz distinguish same-named regions called below different parents. Set to False for a compact, name-aggregated export.
verbose (bool)

Returns:

The files written, in the order they were written.

Return type:

list[Path]

LIKWID#

See LIKWID hardware counters for the workflow and the HDF5 layout.

LikwidRegionResult#

class scope_profiler.likwid_data.LikwidRegionResult(tag, group_id=-1, group_name='', cpus=<factory>, times=<factory>, call_counts=<factory>, event_names=<factory>, counter_names=<factory>, events=<factory>, metric_names=<factory>, metrics=<factory>, source='full_api')

Counter results for a single LIKWID marker region.

The per-thread arrays are all indexed by the same thread axis, so events[e, t] and metrics[m, t] refer to the thread whose runtime is times[t] and whose CPU is cpus[t].

Parameters:

tag (str)
group_id (int)
group_name (str)
cpus (list[int])
times (ndarray)
call_counts (ndarray)
event_names (list[str])
counter_names (list[str])
events (ndarray)
metric_names (list[str])
metrics (ndarray)
source (str)

tag

Region name as passed to markerstartregion.

Type:: str

group_id

Index of the LIKWID event group this region was measured with.

Type:: int

group_name

Name of that group (e.g. "CLOCK"), empty if unknown.

Type:: str

cpus

Hardware threads that took part in the region.

Type:: list of int

times

Per-thread accumulated runtime in seconds, shape (nthreads,).

Type:: numpy.ndarray

call_counts

Per-thread number of times the region was entered, shape (nthreads,).

Type:: numpy.ndarray

event_names

Names of the raw hardware events, length nevents. Not unique: a group may program one event on several counters (see event_labels).

Type:: list of str

counter_names

Hardware counter register each event was programmed on (FIXC0, PMC1, MBOX3C0, …), length nevents. Empty for files written before counter names were recorded.

Type:: list of str

events

Raw counter values, shape (nevents, nthreads).

Type:: numpy.ndarray

metric_names

Names of LIKWID’s derived metrics, length nmetrics.

Type:: list of str

metrics

Derived metric values, shape (nmetrics, nthreads).

Type:: numpy.ndarray

source

Which collection path produced this result: "full_api" (perfmon read-back, the only one with real event names and metrics), "marker_file" (LIKWID’s marker file parsed directly — real values, placeholder event names, no metrics) or "marker_api" (a markergetregion snapshot of the calling thread only).

Type:: str

tag: str

group_id: int = -1

group_name: str = ''

cpus: list[int]

times: ndarray

call_counts: ndarray

event_names: list[str]

counter_names: list[str]

events: ndarray

metric_names: list[str]

metrics: ndarray

source: str = 'full_api'

property event_labels: list[str]

Unique per-event labels, safe to use as dict or column keys.

event_names alone is not unique. A group such as MEM_DP programs the same event on one counter per memory channel, so CAS_COUNT_RD legitimately appears eight times on a socket with eight channels (the ones reading zero are unpopulated). Keying anything by the bare name silently keeps only the last channel.

Names that occur once are returned unchanged; repeated ones get the hardware counter appended — CAS_COUNT_RD:MBOX0C0, CAS_COUNT_RD:MBOX1C0, … — or a positional suffix if the file predates counter names being recorded.

as_dict()

Return the region’s results as a plain dictionary.

Convenience for callers that want to build a DataFrame or dump the numbers without touching the array layout.

Return type:: dict

Summary tables#

scope_profiler.summary.likwid_tables(results, include=None, exclude=None, ranks=None)#

Build one LIKWID counter table per (rank, event group).

Regions become columns and counters become rows: a run typically has a handful of regions but a few dozen counters, so this orientation stays readable where the transpose would not. Splitting per event group keeps every column of a table comparable, since a group fixes which events and metrics exist.

Columns are ordered by descending LIKWID runtime (ties alphabetically), so the costliest regions lead, as in the region table.

Parameters:

results (ProfilingResults) – Source of the LIKWID results.
include (list of str or str, optional) – Regex patterns selecting which regions to report, matched as for the region table.
exclude (list of str or str, optional) – Regex patterns selecting which regions to report, matched as for the region table.
ranks (list of int, optional) – Restrict to these ranks (default: all).

Returns:

One entry per table, each with rank, group, columns (the region labels) and sections ((title, rows) pairs, where a row is (name, values)). Empty when the file holds no LIKWID data.

Return type:

list of dict

scope_profiler.summary.print_likwid_table(table, title=None, stream=None)#

Print one LIKWID counter table from likwid_tables().

Return type:: None

scope_profiler.summary.print_likwid_tables(results, include=None, exclude=None, ranks=None, stream=None)#

Print every LIKWID counter table a result set exposes.

A no-op for files recorded without LIKWID, so callers can invoke it unconditionally.

Collection and storage#

scope_profiler.likwid_data.collect_marker_results_isolated(timeout=120.0)#

Run the perfmon read-back in a child process and return its results.

collect_marker_results() has to re-initialize LIKWID’s perfmon module, and on hosts where the counters are not really usable that can abort the process outright instead of raising — which at finalize() time would destroy a completed run’s output. Running it behind a process boundary turns that worst case into a missing enrichment.

Parameters:: timeout (float, optional) – Seconds to wait for the child before giving up (default: 120).
Returns:: None when the child could not deliver results (crashed, timed out, or LIKWID refused to re-open the counters), which tells the caller to fall back to parse_marker_file().
Return type:: list of LikwidRegionResult or None

scope_profiler.likwid_data.parse_marker_file(path=None)#

Read LIKWID’s marker file directly, without calling into LIKWID.

The file markerclose() writes is plain text:

<nthreads> <nregions> <ngroups>
<region_id>:<tag>-<group_id>          (one line per region)
<region_id> <group_id> <cpu> <call_count> <time> <nevents> <values...>

Parsing it here is the crash-proof path: it touches no counters and calls no LIKWID function, so it cannot take the interpreter down the way re-initializing perfmon can. The cost is that event names, counter registers and derived metrics are not in the file — those exist only inside LIKWID’s group definitions — so events come back positionally named and the metric list is empty.

Parameters:: path (str, optional) – Marker file to read (default: $LIKWID_FILEPATH).
Returns:: One entry per region, ordered by region id. Empty if the file is missing or malformed.
Return type:: list of LikwidRegionResult

scope_profiler.likwid_data.collect_marker_results(pylikwid)#

Read every region of the run back from LIKWID’s marker file.

Must be called after markerclose(), which is what writes the file. Because markerclose() also tears the perfmon module down, the event sets named in LIKWID_EVENTS are re-registered first; that is what makes event and metric names available for the values in the file.

Parameters:: pylikwid (module) – The imported pylikwid module.
Returns:: One entry per region recorded by LIKWID, in file order. Empty when the process is not running under the marker API, or when the performance counters cannot be re-opened.
Return type:: list of LikwidRegionResult

scope_profiler.likwid_data.collect_region_snapshots(pylikwid, region_names)#

Read the current counter values for each named region.

Uses the marker API’s markergetregion, so it must be called before markerclose() and reports only the calling thread. Regions that LIKWID does not know about (never entered, or measured under a different name) are skipped.

Parameters:

pylikwid (module) – The imported pylikwid module.
region_names (iterable of str) – Region tags to query.

Returns:

One entry per region that returned data, with keys tag, nevents, events, time and count.

Return type:

list of dict

scope_profiler.likwid_data.snapshots_to_results(snapshots)#

Convert marker-API snapshots into the common result structure.

Used as the fallback when the full API cannot re-open the counters, so the snapshots taken before markerclose() can still be written to HDF5.

Parameters:: snapshots (Iterable[dict])
Return type:: list[LikwidRegionResult]

scope_profiler.likwid_data.write_likwid_results(h5file, results, environment=None)#

Write collected LIKWID results into an open HDF5 file.

Creates (replacing any previous copy) a likwid group holding one subgroup per region under likwid/regions/<tag>.

Parameters:

h5file (h5py.File or h5py.Group) – Destination, typically the per-rank profiling file.
results (iterable of LikwidRegionResult) – Regions to store.
environment (dict, optional) – LIKWID environment variables to record as attributes on the group, so the event set a file was measured with stays with the data.

Return type:

None

scope_profiler.likwid_data.markers_available()#

Whether this process runs under the LIKWID marker API.

markerinit() silently degrades to a no-op when LIKWID’s environment is absent, so this is what distinguishes “there will be counter data” from “the script was started as a plain python script.py”.

Return type:: bool

scope_profiler.likwid_data.likwid_environment()#

Return the LIKWID environment of the current process.

Returns:: The LIKWID_* variables set by the launcher. Empty when the process was not started under likwid-perfctr -m.
Return type:: dict

API reference

Contents

API reference#

ProfileManager#

ProfilingConfig#

Region classes#

BaseProfileRegion#

DisabledProfileRegion#

TimeOnlyProfileRegion#

FullProfileRegion#

LineProfilerRegion#

Post-processing#

ProfilingResults#

read_h5#

read_h5_summary#

Native (C and Fortran) traces#

Combining runs#

Region#

MPIRegion#

Call stack reconstruction#

Intervals must be properly nested#

Reconstruction at scale#

Plotting#

Statistics export#

Exporting to other tools#

LIKWID#

LikwidRegionResult#

Summary tables#

Collection and storage#