API reference

Install for this page: pip install "scope-profiler[pproc]" covers the complete reference. The core profiling and file APIs need only pip install scope-profiler; plotting/DataFrames need pproc, and LIKWID collection needs scope-profiler[likwid] plus a system LIKWID installation.

This page walks through the public API by task. Every module, with all its members, is documented in the API reference.

Module-level shortcuts

The everyday lifecycle, instrumentation, and state-query calls are importable from the package root, so the common case needs no class name:

import scope_profiler as sp

with sp.session(file_path="run.h5", return_results=True) as run:
    with sp.region("solve"):
        solve()

results = run.results

sp.setup, sp.finalize, sp.session, sp.region, sp.profile, sp.is_active, sp.is_configured, sp.registered_regions, and sp.recorded_regions are the corresponding ProfileManager class methods rather than wrappers around them, and act on the same process-wide default manager. Use ProfileManager directly for independent manager instances.

Library instrumentation that is off by default

A library registers an opt-in tag once with sp.register_opt_in_tag("mylib") and passes tags=("mylib",) to region, profile, define_region or instrument. Those regions stay disabled until a run enables the tag with setup(enable_tags=["mylib"]), the enable_tags TOML setting, SCOPE_PROFILER_ENABLE_TAGS=mylib or scope-profiler run --enable-tag mylib. The registry is process-wide and shared by every manager.

Test suites reset the profiler between tests with sp.reset(), which keeps the instrumentation (decorated functions and define_region definitions) so that it records again after the next setup().

ProfileManager

The main entry point for all profiling operations. Class methods provide the process-wide default manager, so the common case does not require an instance. Create ProfileManager() instances when independent profiling sessions need to coexist; each instance owns its configuration, regions, decorators, and output.

ProfileManagerclass#

View source

class ProfileManager

Manage and track a set of profiling regions.

The class methods remain the process-wide default API. Instantiating the class creates an independent manager with its own configuration, regions, decorators, and call-id space, allowing multiple profiling sessions to be active at the same time::

cpu = ProfileManager()
io = ProfileManager()

with cpu.session(file_path="cpu.h5"):
    with io.session(file_path="io.h5"):
        with cpu.profile_region("compute"), io.profile_region("write"):
            work()

profile_regionattributeclass attributeinstance attribute#

View source

profile_region = region

define_regionmethodclassmethod#

View source

def define_region(region_name, functions=None, tags=None) -> RegionHandle

Define a reusable region handle that survives later setup calls.

finalizemethodclassmethod#

View source

def finalize(verbose: bool = True, return_results: bool = False, native_traces=None, verbose_line_profiler: bool = False)

Finalize profiling and write the run’s data to a single output file.

Copies each region’s buffered timestamps out, moves every rank’s copy to rank 0, and has rank 0 write them into one HDF5 file. Nothing is staged on the filesystem, so no shared $TMPDIR is needed. Optionally prints profiling statistics for each region.

Under MPI this is collective: every rank must call it, with the same arguments, or the job hangs - rank 0 waits for a payload from every other rank. A rank that dies before reaching it therefore leaves the job waiting rather than silently dropping that rank’s data.

With use_likwid=True this is also where the LIKWID markers are closed and every marker region of the run is read back and stored in the output file under rank<r>/likwid/regions/<tag>; see :meth:~scope_profiler.results.ProfilingResults.get_likwid_regions for reading it back.

Parameters

verbose : bool, optional If True, prints the concise profiling summary (default: True). verbose_line_profiler : bool, optional If True, prints detailed line-profiler tables when line profiling is enabled (default: False). return_results : bool, optional If True, return the run’s data as a :class:~scope_profiler.results.ProfilingResults - the same post-processing API :func:~scope_profiler.h5reader.read_h5 gives back, built straight from the in-memory buffers instead of by reading the output file back::

    results = ProfileManager.finalize(return_results=True)
    results.print_summary()
    df = results.to_dataframe()

This works with ``deactivate_file_output=True``, where no file is
written at all. Under MPI the per-rank data is gathered on rank 0,
which is collective: every rank must pass the same value.

path or sequence of paths, optional

Files (or directories of them) written by the C or Fortran region API in this same process, to fold into this run’s output - either native format, a .spt trace or the .h5 an SP_USE_HDF5 C build writes. Each rank picks up the file matching its own rank, so a mixed-language MPI run still produces one file::

kernels.stop_profiling()            # native sp_finalize()
ProfileManager.finalize(native_traces=".")

Call the native side’s sp_finalize() first: its output has to exist by the time this reads it. A region name recorded on both sides raises, rather than silently double-counting.

Returns

ProfilingResults or None The run’s profiling data when return_results=True, and None otherwise. Under MPI rank 0 gets the whole run, like the merged output file; the other ranks get an empty result set for which print_summary(), the plot_* functions and the exporters do nothing, so the script above needs no rank guard. See :attr:~scope_profiler.results.ProfilingResults.is_root.

get_all_regionsmethodclassmethod#

View source

def get_all_regions() -> dict[str, BaseProfileRegion]

Get all registered ProfileRegion instances.

Returns

dict: Dictionary of all registered ProfileRegion instances.

get_configmethodclassmethod#

View source

def get_config() -> ProfilingConfig

Get the current profiling configuration, creating a default one if setup() has not been called.

This is the only place a configuration comes into being outside setup(), and it is deliberately lazy: constructing one resolves the MPI communicator, which imports mpi4py and therefore calls MPI_Init in any process the launcher marked as a rank. Doing that at import time would mean import scope_profiler silently joins the MPI job - fatal in a process forked from a rank, which is exactly what the LIKWID counter read-back does.

Resolving defaults is not itself setup. Regions remain disabled until :meth:setup (directly or through :meth:session) installs a config with :meth:set_config. This lets applications declare decorators at import time without accidentally starting a profiling run.

Returns

ProfilingConfig The current profiling configuration.

get_regionmethodclassmethod#

View source

def get_region(region_name) -> BaseProfileRegion

Get a registered ProfileRegion by name.

Parameters

region_name: str The name of the profiling region.

Returns

ProfileRegion or None: The registered ProfileRegion instance or None if not found.

instrumentmethodclassmethod#

View source

def instrument(module, *, include=('*'), exclude=(), prefix=None, tags=None)

Decorate matching module-level functions without editing source.

tags is passed to every :meth:profile decorator, so a library can put a whole module behind one opt-in tag (see :meth:register_opt_in_tag).

is_activemethodclassmethod#

View source

def is_active() -> bool

Whether instrumentation currently records calls.

is_configuredmethodclassmethod#

View source

def is_configured() -> bool

Whether explicit setup has configured this manager.

last_resultsmethodclassmethod#

View source

def last_results()

Most recent in-memory results requested from this manager, if any.

metadatamethodclassmethod#

View source

def metadata(**values)

Attach key/value metadata to calls completed inside this scope.

pausemethodclassmethod#

View source

def pause() -> None

Temporarily stop recording new profiling scopes.

Pause/resume is intended for boundaries between simulation phases. An open scope cannot be split safely without changing its call identity, so pausing while a region is active raises a clear error.

profilemethodclassmethod#

View source

def profile(region_name=None, recursive=None, metadata=None, record_outcome=False, tags=None)

Decorator factory for profiling a function.

Parameters

region_name : str, optional Name for the profiling region. If not provided, uses the decorated function’s name. Supports being used with or without parentheses. recursive : bool, optional If True, also profiles Python function calls made by the decorated function (excluding scope-profiler internals). If None, falls back to ProfileManager.setup(recursive_profile=...). metadata : callable or dict, optional Metadata to attach to each call. A callable receives the decorated function’s (*args, **kwargs) and must return a mapping. record_outcome : bool, optional Add status and, for failures, exception metadata to each decorated call (default: False). tags : iterable of str, optional Labels persisted with the region, as for :meth:region. An opt-in tag (see :meth:register_opt_in_tag) leaves the function unprofiled unless the run enables it.

Returns

Callable Decorated function wrapped with profiling instrumentation.

Notes

The decorated function is registered so that calling ProfileManager.setup() after decoration re-binds the wrapper to the new region class at zero per-call cost. This means @ProfileManager.profile can be applied at class-definition time even when setup() is called later. While its region is disabled (before setup(), or behind an opt-in tag that is off) the wrapper calls the function directly, skipping metadata and recursive tracing.

profile_commandmethodclassmethod#

View source

def profile_command(command, *, output=None, recursive=False, all=False, mpi_calls=False)

Run a command under scope-profiler run and return its result.

profile_scriptmethodclassmethod#

View source

def profile_script(script_path, *, script_args=None, output=None, **kwargs)

Profile a script from Python using the CLI runner semantics.

read_resultsmethodclassmethod#

View source

def read_results() -> ProfilingResults

Open the merged profiling file this run wrote, for post-processing.

Convenience for analysing results in the same script that produced them::

ProfileManager.finalize()
results = ProfileManager.read_results()
results.print_summary()

Returns

ProfilingResults The data in the file at config.file_path.

Raises

FileNotFoundError If the merged file does not exist yet. It is written by :meth:finalize, and only on rank 0 - guard the call with if ProfileManager.get_config()._rank == 0 under MPI.

recorded_regionsmethodclassmethod#

View source

def recorded_regions() -> tuple[str, ...]

Names of regions that have recorded at least one call.

regionmethodclassmethod#

View source

def region(region_name, functions=None, tags=None) -> BaseProfileRegion

Get the profiling region named region_name, creating it if needed.

The returned region is a context manager, which is how a block of code is timed::

with ProfileManager.region("solve"):
    solve()

Parameters

region_name: str The name of the profiling region. functions : list of callable, optional Functions to register for line-by-line profiling. Only has an effect when use_line_profiler=True. Useful when using the context manager form, since the decorator form (wrap) registers functions automatically::

    with ProfileManager.region("my_region", functions=[my_func]):
        my_func()

iterable of str, optional

User-defined labels persisted with the region. Reusing a region name with a different non-None tag set raises ValueError. A region carrying a tag registered with :meth:register_opt_in_tag is disabled unless the run enables that tag.

Returns

ProfileRegion : The ProfileRegion instance.

Notes

profile_region is the original name for this method and remains available as an alias; the two are the same object.

region_factorymethodclassmethod#

View source

def region_factory(region_name, *, tags=None) -> RegionFactory

Create reusable regions whose calls may carry dynamic metadata.

register_opt_in_tagmethodclassmethod#

View source

def register_opt_in_tag(*tags: str) -> None

Mark tags as opt-in: their regions are off unless enabled.

Meant for libraries that ship their own instrumentation. Register the tag once, at import time, and tag the library’s regions with it::

mylib/__init__.py

import scope_profiler as sp

sp.register_opt_in_tag("mylib")

@sp.profile("mylib.solve", tags=("mylib",))
def solve(): ...

Applications that profile themselves then don’t record mylib’s regions until they ask for them, without editing mylib: with setup(enable_tags=["mylib"]) (also a ProfilingOptions field and a TOML setting), the SCOPE_PROFILER_ENABLE_TAGS=mylib environment variable, or scope-profiler run --enable-tag mylib. "*" enables every opt-in tag. Listing the tag in region_tags enables it too.

Regions and decorated functions that already exist pick the change up, so the call may come after the library’s own imports.

Instrumentation that is off costs one extra Python call (roughly 0.1-0.3 µs per call), so keep it to operations that take microseconds or more.

registered_regionsmethodclassmethod#

View source

def registered_regions() -> tuple[str, ...]

Names of all instrumentation points known to this manager.

resetmethodclassmethod#

View source

def reset() -> None

Forget the configuration and everything recorded; keep the instrumentation.

Leaves the manager as importing the instrumented code does: not configured, every region disabled until the next :meth:setup. Decorated functions and :meth:define_region definitions stay registered, and decorated functions call straight through until then. Meant for test suites that profile in more than one test::

@pytest.fixture(autouse=True)
def _fresh_profiler():
    sp.reset()
    yield
    sp.reset()

resumemethodclassmethod#

View source

def resume() -> None

Resume recording after :meth:pause.

Calling resume before pause is harmless, which makes it safe to place in conditional simulation control paths.

runmethodclassmethod#

View source

def run(func, *args, output=None, **kwargs)

Run a callable inside a managed profiling session.

run_scriptmethodclassmethod#

View source

def run_script(script_path: str, script_args: list | None = None, region_name: str | None = None, only_user_code: bool = True, recursive: bool = True, include_patterns=(), exclude_patterns=(), entrypoint: str | None = None) -> None

Run a script under recursive profiling, similar to python -m cProfile.

Instruments every Python function call made while the script runs and records each as its own region, without requiring any decorators or context managers in the script itself. Intended to be called after ProfileManager.setup() and followed by ProfileManager.finalize(); see python -m scope_profiler for the CLI wrapper around this.

Parameters

script_path : str Path to the script to execute. script_args : list of str, optional Arguments exposed to the script as sys.argv[1:]. region_name : str, optional Name for the region wrapping the whole script’s execution (default: the script’s basename). only_user_code : bool, optional If True (default), skip instrumenting standard-library and installed-package frames, tracing only the script’s own code. This keeps overhead low and the output focused. Set to False to trace everything, including third-party and stdlib calls. recursive : bool, optional Trace Python function calls and add a region around the script (default: True). When False, run only explicitly instrumented profile decorators and region blocks. include_patterns, exclude_patterns : iterable of str, optional Glob filters for recursively traced function names. entrypoint : str, optional Function name to use as the root region when recursively tracing.

sample_everymethodclassmethod#

View source

def sample_every(every: int, start: int = 0)

Profile selected timesteps in a loop.

The yielded callable is a context manager. Pass it the timestep number at the boundary of each iteration::

with ProfileManager.sample_every(10) as profile_step:
    for timestep in range(num_steps):
        with profile_step(timestep):
            advance_simulation()

Timestep start and then every every-th timestep are profiled; all other iterations run while profiling is paused. The timestep context must surround the profiled work and must not be entered while another profiled scope is active. On exit, the previous pause state is restored.

sessionmethodclassmethod#

View source

def session(options: ProfilingOptions | None = None, *, verbose: bool = True, verbose_line_profiler: bool = False, return_results: bool = False, native_traces=None, output=_UNSET, replace: bool = False, config_path: str | os.PathLike[str] | None = None, **overrides: Unpack[SetupOptions])

Return a context manager that sets up and finalizes profiling.

Every argument other than verbose, verbose_line_profiler, return_results and native_traces is passed to :meth:setup, with the same meaning and precedence there - options (a :class:~scope_profiler.profile_config.ProfilingOptions), config_path, and any setting as a keyword::

with ProfileManager.session(options=options) as run:
    ...

with ProfileManager.session(file_path="run.h5") as run:
    ...

Finalization runs even when the profiled block raises; the original exception is preserved.

native_traces is handed to :meth:finalize, which folds the output of a C or Fortran library profiled in this same process into this run’s file - in either native format. See its documentation for the two rules that mixing languages imposes.

When return_results=True, the context object exposes the finalized :class:~scope_profiler.results.ProfilingResults as results::

with ProfileManager.session(return_results=True, verbose=False) as run:
    with ProfileManager.profile_region("solve"):
        solve()
results = run.results

set_configmethodclassmethod#

View source

def set_config(config: ProfilingConfig) -> None

Set a new profiling configuration and update the region class.

Parameters

config : ProfilingConfig The new profiling configuration to apply.

setupmethodclassmethod#

View source

def setup(options: ProfilingOptions | None = None, *, auto_finalize: bool = False, output=_UNSET, replace: bool = False, region_tags=None, config_path: str | os.PathLike[str] | None = None, **overrides: Unpack[SetupOptions])

Initialize and configure the profiling system.

Parameters

options : ProfilingOptions, optional A :class:~scope_profiler.profile_config.ProfilingOptions bag holding any of the settings below, for reuse across calls or construction away from the call site::

    options = ProfilingOptions(use_likwid=True, file_path="run.h5")
    ProfileManager.setup(options=options)

An explicit keyword argument passed alongside ``options`` wins
over the same field on ``options``, which in turn wins over
``config_path`` and the defaults below.

config_path : str or os.PathLike, optional TOML file containing a [profiling] table with these settings. Values passed directly to setup() take precedence. See :func:~scope_profiler.profile_config.load_profiling_config. auto_finalize : bool, optional Finalize once at normal interpreter exit, unless finalize() is called explicitly first (default: False). This is convenient for scripts that want setup without a surrounding session. Avoid it when MPI ranks may not exit together. output : str, os.PathLike or None, optional Direct spelling of the output destination. None keeps results in memory and writes no file. This is an alias for file_path and deactivate_file_output; do not combine those spellings. replace : bool, optional Replace an active configuration (default: False). Without this, accidental setup of an already active manager raises. **overrides Any of the settings below, passed as keyword arguments::

    ProfileManager.setup(file_path="run.h5", use_likwid=True)

They are the fields of
:class:`~scope_profiler.profile_config.ProfilingOptions`, which
is where they are declared once and typed; an unrecognised name
raises ``TypeError`` naming the closest match. Prefixed settings
can also be given as groups on ``options`` - see
:class:`~scope_profiler.profile_config.MemrayOptions`,
:class:`~scope_profiler.profile_config.GPUOptions` and
:class:`~scope_profiler.profile_config.HDF5Options`.

file_path : str, optional Path to the output profiling data file (default: “profiling_data.h5”). label : str or None, optional Short name for this run (default: None, i.e. the output file’s stem). Post-processing uses it wherever a run has to be named — chart legends, the summary heading, scope-profiler inspect, the JSON statistics - which is what makes several runs distinguishable when they are compared::

    ProfileManager.setup(file_path="run_a.h5", label="128 ranks")

It is stored in the output file as the ``label`` metadata field,
so it survives into every later post-processing step.

use_likwid : bool, optional Enable LIKWID hardware counter collection (default: False). use_line_profiler : bool, optional Enable line-by-line profiling via line_profiler (default: False). use_memray : bool, optional Record process-wide allocations with Memray (default: False). The capture is a separate Memray .bin file and requires the separately installed memray package. memory_profile_path : str, optional Memray capture path (default: <file-stem>.memray.bin). memray_native_traces, memray_trace_python_allocators, memray_follow_fork : bool, optional Memray capture options. Python allocator tracing can create much larger traces and has materially higher overhead. deactivate_profiling : bool, optional Turn profiling off entirely (default: False). Every region becomes a no-op, so the instrumentation can stay in the code at near-zero cost instead of being removed. use_nvtx : bool, optional Add NVTX ranges to profiled regions for NVIDIA Nsight tools (default: False). Requires scope-profiler[nvtx]. use_gpu_timing : bool, optional Record CUDA-event elapsed device time for each profiled region (default: False). CPU timestamps are still recorded, so the normal timeline remains enqueue-side timing. gpu_sync_on_exit : bool, optional Wait for the current GPU device before recording CPU end time. Requires use_gpu_timing=True; defaults to False. Applies to contexts and decorators and removes CPU/GPU overlap. Custom backends must provide synchronize() when enabled. gpu_timing_backend : str or object, optional CUDA-event backend for use_gpu_timing: "auto", "torch", "cupy", or a custom object implementing record_event() and elapsed_time_ns(start_event, end_event). deactivate_file_output : bool, optional Write no HDF5 file at all (default: False), not even the run metadata. Use it with finalize(return_results=True) to analyse a run entirely in memory::

    ProfileManager.setup(deactivate_file_output=True)
    ...
    results = ProfileManager.finalize(return_results=True)

bool, optional

Enable recursive profiling for all decorated functions by default (default: False). This can be overridden per decorator with @ProfileManager.profile(..., recursive=...).

aggregation_mode : bool, optional Record only count, inclusive total, minimum, maximum, and exclusive total per region. Timeline events are unavailable in this mode; it cannot be combined with line, GPU, NVTX, or LIKWID profiling. profile_mpi_calls : bool, optional Profile mpi4py operations made through predefined and derived communicators (default: False). mpi4py is imported only if the application imports it. With :meth:session, obtain MPI.COMM_WORLD inside the session so it receives the proxy. track_threads : bool, optional Profile every thread (default: False). Each thread gets its own buffers and scope stack, so regions entered concurrently no longer overwrite one another, and every recorded call carries the thread it ran on. The run also reports each thread’s name, OS ids, lifetime and CPU time - see :attr:~scope_profiler.results.ProfilingResults.threads::

    with ProfileManager.session(track_threads=True):
        ...

Cannot be combined with line, GPU, NVTX, LIKWID or aggregation
profiling.

track_async : bool, optional Follow asyncio tasks and greenlets as well (default: False, and implies track_threads). Each call additionally carries the task it ran in and the time that task spent suspended inside the call, so a with block held across an await reports its await time rather than charging it to the region. Per-task running and awaiting totals are available from :attr:~scope_profiler.results.ProfilingResults.tasks. metadata_detail : {“minimal”, “full”}, optional How much of the run’s environment is recorded (default: "minimal"): versions, platform, CPU model and thread/rank counts, with source files relative to the working directory. "full" adds the user name, host name, working directory, loaded modules, environment variables, SLURM_* variables and absolute source paths. Region source text is controlled separately by capture_region_source. capture_region_source : bool, optional Record where each region is defined - the with block or the decorated function - once per distinct source file, the first time any of its regions is created (default: False). See :attr:~scope_profiler.region.Region.source_text. Off by default because the cost, while cheap for a typical file, is not always: it is one ast.parse + tree walk of that file, so it tracks the file’s total size, not the size or number of the regions in it - under a millisecond for a typical few-hundred- line file, but tenths of a second per rank for one containing thousands of lines across many regions. Every rank pays that independently, so it can compound to whole seconds under contention on a job with more ranks than idle cores (measured: ~0.3s/rank at 8 ranks, ~2.9s/rank at 64, for a single ~10,000-line file, on a shared/oversubscribed node)::

    ProfileManager.setup(capture_region_source=True)

int, optional

Initial number of profiling events preallocated per region (default: 1024). Buffers grow on demand, so this is a starting size rather than a limit; raise it for very hot regions to avoid repeated reallocation.

output_mode : {“auto”, “direct”, “parallel”}, optional MPI file writer. auto prefers MPI-enabled h5py when compatible with the active instrumentation and otherwise lets ranks append directly to one serial-HDF5 file in token order. hdf5_compression : {“gzip”, “lzf”, “zstd”} or None, optional Compression filter for timestamp and GPU-duration datasets. hdf5_compression_level : int or None, optional GZIP level 0—9 or Zstandard level 1—22. hdf5_chunk_size : int or None, optional Maximum events per dataset chunk. Enables chunked partial reads even without compression. enable_tags : iterable of str, optional Opt-in tags (see :meth:register_opt_in_tag) whose regions this run records; "*" enables all of them. The comma-separated SCOPE_PROFILER_ENABLE_TAGS environment variable adds to this list, as do the tags in region_tags.

Notes

The run’s start time is the moment setup() is called; it is stored as the start_time_ns metadata field and is the origin of the relative timeline in post-processing. MPI rank detection is separate from profile_mpi_calls: collectives are used exactly when the process was started by an MPI launcher. See :mod:scope_profiler.mpi_launch for detection and its SCOPE_PROFILER_MPI override.

setup_from_envmethodclassmethod#

View source

def setup_from_env(**overrides)

Configure profiling from SCOPE_PROFILER_* environment variables.

ProfilingOptions

The settings setup() and session() accept, as an object that can be built away from the call site and reused across runs. It is also where each setting is declared once: the keyword arguments of setup() are its fields.

Settings that share a prefix can be given as groups instead, which spell the same thing without the prefix:

from scope_profiler import HDF5Options, ProfilingOptions

options = ProfilingOptions(
    file_path="run.h5",
    hdf5=HDF5Options(compression="gzip", compression_level=4),
)

ProfilingOptionsclassdataclass#

View source

class ProfilingOptions(file_path: str | None = None, label: str | None = None, use_likwid: bool | None = None, perf_events: list[str] | tuple[str, ...] | str | None = None, use_line_profiler: bool | None = None, use_memray: bool | None = None, memory_profile_path: str | None = None, memray_native_traces: bool | None = None, memray_trace_python_allocators: bool | None = None, memray_follow_fork: bool | None = None, deactivate_profiling: bool | None = None, use_nvtx: bool | None = None, use_gpu_timing: bool | None = None, gpu_timing_backend: Any = None, deactivate_file_output: bool | None = None, recursive_profile: bool | None = None, aggregation_mode: bool | None = None, profile_mpi_calls: bool | None = None, track_threads: bool | None = None, track_async: bool | None = None, capture_region_source: bool | None = None, metadata_detail: str | None = None, buffer_limit: int | None = None, output_mode: str | None = None, hdf5_compression: str | None = None, hdf5_compression_level: int | None = None, hdf5_chunk_size: int | None = None, enable_tags: list[str] | tuple[str, ...] | None = None, memray: MemrayOptions | None = None, gpu: GPUOptions | None = None, hdf5: HDF5Options | None = None, gpu_sync_on_exit: bool | None = None)

A bag of :meth:ProfileManager.setup settings, to reuse or pass around.

Every field mirrors a setup()/session() keyword argument and defaults to None, meaning “unset - let setup()’s own default, or its config_path file, decide”. Pass one to setup() or session() instead of repeating the same handful of keyword arguments at every call site::

options = ProfilingOptions(use_likwid=True, file_path="run.h5")
ProfileManager.setup(options=options)
# ... or, equivalently:
with ProfileManager.session(options=options):
    ...

An explicit keyword argument passed alongside options still wins over the same field on options, so a shared ProfilingOptions can be reused across runs with one-off overrides::

ProfileManager.setup(options=options, file_path="run_b.h5")

Settings that share a prefix can also be given as groups — :class:MemrayOptions, :class:GPUOptions and :class:HDF5Options — which spell the same settings without the prefix::

options = ProfilingOptions(
    file_path="run.h5",
    hdf5=HDF5Options(compression="gzip", compression_level=4),
)

Setting one thing both ways raises ValueError rather than picking a winner.

Attributes

file_path : str or None Path to the output profiling data file (default: "profiling_data.h5"). label : str or None Short name for this run (default: None, i.e. the output file’s stem). Post-processing uses it wherever a run has to be named — chart legends, the summary heading, scope-profiler inspect, the JSON statistics - and it is stored in the output file as the label metadata field, so it survives into every later post-processing step. use_likwid : bool or None Enable LIKWID hardware counter collection (default: False). perf_events : sequence of str or None Linux perf_event_open events to collect per region (default: None). use_line_profiler : bool or None Enable line-by-line profiling via line_profiler (default: False). use_memray : bool or None Record process-wide memory allocations with Memray (default: False). deactivate_profiling : bool or None Turn profiling off entirely (setup() default: False). Every region becomes a no-op, so the instrumentation can stay in the code at near-zero cost instead of being removed. use_nvtx : bool or None Add NVTX ranges to profiled regions for NVIDIA Nsight tools (default: False). Requires scope-profiler[nvtx]. use_gpu_timing : bool or None Record CUDA-event elapsed device time for each profiled region (default: False). CPU timestamps are still recorded, so the normal timeline remains enqueue-side timing. gpu_sync_on_exit : bool or None Wait for the GPU device at region exit. Requires use_gpu_timing and removes CPU/GPU overlap; disabled by default. gpu_timing_backend : str, object, or None CUDA-event backend for use_gpu_timing: "auto", "torch", "cupy", or a custom object implementing record_event() and elapsed_time_ns(start_event, end_event) (default: "auto"). deactivate_file_output : bool or None Write no HDF5 file at all (default: False), not even the run metadata. Pair it with finalize(return_results=True) to analyse a run entirely in memory. recursive_profile : bool or None Enable recursive profiling for all decorated functions by default (default: False). Overridable per decorator with @ProfileManager.profile(..., recursive=...). aggregation_mode : bool or None Record only count, inclusive total, minimum, maximum, and exclusive total per region (default: False). Timeline events are unavailable in this mode; it cannot be combined with line, GPU, NVTX, or LIKWID profiling. profile_mpi_calls : bool or None Profile mpi4py operations made through predefined or derived communicators (default: False). mpi4py remains lazily imported. track_threads : bool or None Record which thread each call ran on, and describe every thread the run touched (default: False). Required for correct results from concurrently used regions; see :class:~scope_profiler.region_profiler.ThreadedProfileRegion. track_async : bool or None Also record which asyncio task or greenlet each call ran in, and how much of the call was spent awaiting (default: False). Implies track_threads. capture_region_source : bool or None Record where each region is defined - the with block or the decorated function - once per distinct source file, the first time any of its regions is created (default: False). See :attr:~scope_profiler.region.Region.source_text. Off by default: the cost tracks that file’s total size (one ast.parse + tree walk), so it stays under a millisecond for a typical file but can reach tenths of a second per rank for a single file with thousands of lines, paid independently by every rank. metadata_detail : {“minimal”, “full”} or None How much of the run’s environment is recorded (default: "minimal"). "minimal" stores versions, platform, CPU model and thread/rank counts, and each region’s and line profile’s source file relative to the working directory (or by name alone, outside it). "full" adds the user name, host name, working directory, loaded modules, environment variables such as PATH and VIRTUAL_ENV, SLURM_* variables and absolute source paths; opt into it for runs that stay on the machine they were recorded on. buffer_limit : int or None Initial number of profiling events preallocated per region (default: 1024). Buffers grow on demand, so this is a starting size rather than a limit; raise it for very hot regions to avoid repeated reallocation. output_mode : str or None MPI file writer: "auto", "direct", or "parallel" (default: "auto"). auto prefers MPI-enabled h5py when compatible with the active instrumentation and otherwise lets ranks append directly to one serial-HDF5 file in token order. hdf5_compression : str or None Compression filter for timestamp and GPU-duration datasets: None, "auto", "gzip", "lzf", or "zstd" (default: None). "auto" compresses at publication time, and only when the run is large enough for the saving to repay the write CPU. hdf5_compression_level : int or None GZIP level 0—9 or Zstandard level 1—22 (default: None). hdf5_chunk_size : int or None Maximum events per dataset chunk (default: None). Enables chunked partial reads even without compression. enable_tags : sequence of str or None Opt-in region tags to record (default: None). Regions carrying a tag registered with :meth:ProfileManager.register_opt_in_tag - typically a library’s own instrumentation - stay disabled unless one of their opt-in tags is listed here, in region_tags, or in the comma-separated SCOPE_PROFILER_ENABLE_TAGS environment variable. "*" enables every opt-in tag. memray : MemrayOptions or None The use_memray/memory_profile_path/memray_* settings as one group. gpu : GPUOptions or None The use_gpu_timing/gpu_timing_backend/use_nvtx settings as one group. hdf5 : HDF5Options or None The hdf5_* settings as one group.

aggregation_modeattributeclass attributeinstance attribute#

View source

aggregation_mode: bool | None = None

buffer_limitattributeclass attributeinstance attribute#

View source

buffer_limit: int | None = None

capture_region_sourceattributeclass attributeinstance attribute#

View source

capture_region_source: bool | None = None

deactivate_file_outputattributeclass attributeinstance attribute#

View source

deactivate_file_output: bool | None = None

deactivate_profilingattributeclass attributeinstance attribute#

View source

deactivate_profiling: bool | None = None

enable_tagsattributeclass attributeinstance attribute#

View source

enable_tags: list[str] | tuple[str, ...] | None = None

file_pathattributeclass attributeinstance attribute#

View source

file_path: str | None = None

gpuattributeclass attributeinstance attribute#

View source

gpu: GPUOptions | None = None

gpu_sync_on_exitattributeclass attributeinstance attribute#

View source

gpu_sync_on_exit: bool | None = None

gpu_timing_backendattributeclass attributeinstance attribute#

View source

gpu_timing_backend: Any = None

hdf5attributeclass attributeinstance attribute#

View source

hdf5: HDF5Options | None = None

hdf5_chunk_sizeattributeclass attributeinstance attribute#

View source

hdf5_chunk_size: int | None = None

hdf5_compressionattributeclass attributeinstance attribute#

View source

hdf5_compression: str | None = None

hdf5_compression_levelattributeclass attributeinstance attribute#

View source

hdf5_compression_level: int | None = None

labelattributeclass attributeinstance attribute#

View source

label: str | None = None

memory_profile_pathattributeclass attributeinstance attribute#

View source

memory_profile_path: str | None = None

memrayattributeclass attributeinstance attribute#

View source

memray: MemrayOptions | None = None

memray_follow_forkattributeclass attributeinstance attribute#

View source

memray_follow_fork: bool | None = None

memray_native_tracesattributeclass attributeinstance attribute#

View source

memray_native_traces: bool | None = None

memray_trace_python_allocatorsattributeclass attributeinstance attribute#

View source

memray_trace_python_allocators: bool | None = None

metadata_detailattributeclass attributeinstance attribute#

View source

metadata_detail: str | None = None

output_modeattributeclass attributeinstance attribute#

View source

output_mode: str | None = None

perf_eventsattributeclass attributeinstance attribute#

View source

perf_events: list[str] | tuple[str, ...] | str | None = None

profile_mpi_callsattributeclass attributeinstance attribute#

View source

profile_mpi_calls: bool | None = None

recursive_profileattributeclass attributeinstance attribute#

View source

recursive_profile: bool | None = None

track_asyncattributeclass attributeinstance attribute#

View source

track_async: bool | None = None

track_threadsattributeclass attributeinstance attribute#

View source

track_threads: bool | None = None

use_gpu_timingattributeclass attributeinstance attribute#

View source

use_gpu_timing: bool | None = None

use_likwidattributeclass attributeinstance attribute#

View source

use_likwid: bool | None = None

use_line_profilerattributeclass attributeinstance attribute#

View source

use_line_profiler: bool | None = None

use_memrayattributeclass attributeinstance attribute#

View source

use_memray: bool | None = None

use_nvtxattributeclass attributeinstance attribute#

View source

use_nvtx: bool | None = None

to_kwargsmethod#

View source

def to_kwargs() -> dict

This options’ explicitly-set fields, as setup() keyword arguments.

Group fields are expanded to the flat names they stand for. Setting the same thing both ways raises ValueError rather than picking a winner.

MemrayOptionsclassdataclass#

View source

class MemrayOptions(enabled: bool | None = None, path: str | None = None, native_traces: bool | None = None, trace_python_allocators: bool | None = None, follow_fork: bool | None = None)

Bases: _OptionGroup

Memray capture settings, as a group.

ProfilingOptions(memray=MemrayOptions(enabled=True, native_traces=True)) configures the same run as ProfilingOptions(use_memray=True, memray_native_traces=True); use whichever reads better at the call site. In a TOML config file the group is the [profiling.memray] table.

Attributes

enabled : bool or None Record process-wide allocations with Memray (use_memray). path : str or None Capture path (memory_profile_path); defaults to <file-stem>.memray.bin. native_traces : bool or None Capture native stack frames (memray_native_traces). trace_python_allocators : bool or None Trace Python’s own allocators (memray_trace_python_allocators). Creates much larger traces and has materially higher overhead. follow_fork : bool or None Keep tracking across fork() (memray_follow_fork).

enabledattributeclass attributeinstance attribute#

View source

enabled: bool | None = None

follow_forkattributeclass attributeinstance attribute#

View source

follow_fork: bool | None = None

native_tracesattributeclass attributeinstance attribute#

View source

native_traces: bool | None = None

pathattributeclass attributeinstance attribute#

View source

path: str | None = None

trace_python_allocatorsattributeclass attributeinstance attribute#

View source

trace_python_allocators: bool | None = None

Inherited from scope_profiler.profile_config._OptionGroup

to_kwargsmethod#

Inherited from scope_profiler.profile_config._OptionGroupView source

def to_kwargs() -> dict

This group’s explicitly-set fields, under their flat names.

GPUOptionsclassdataclass#

View source

class GPUOptions(timing: bool | None = None, backend: Any = None, nvtx: bool | None = None, sync_on_exit: bool | None = None)

Bases: _OptionGroup

GPU timing and NVTX settings, as a group.

In a TOML config file this is the [profiling.gpu] table.

Attributes

timing : bool or None Record CUDA-event elapsed device time per region (use_gpu_timing). backend : str, object, or None CUDA-event backend (gpu_timing_backend): "auto", "torch", "cupy", or an object implementing record_event() and elapsed_time_ns(start_event, end_event). sync_on_exit : bool or None Synchronize the device at region exit (gpu_sync_on_exit). Requires timing and removes CPU/GPU overlap; disabled by default. nvtx : bool or None Emit NVTX ranges for NVIDIA Nsight tools (use_nvtx).

backendattributeclass attributeinstance attribute#

View source

backend: Any = None

nvtxattributeclass attributeinstance attribute#

View source

nvtx: bool | None = None

sync_on_exitattributeclass attributeinstance attribute#

View source

sync_on_exit: bool | None = None

timingattributeclass attributeinstance attribute#

View source

timing: bool | None = None

Inherited from scope_profiler.profile_config._OptionGroup

to_kwargsmethod#

Inherited from scope_profiler.profile_config._OptionGroupView source

def to_kwargs() -> dict

This group’s explicitly-set fields, under their flat names.

HDF5Optionsclassdataclass#

View source

class HDF5Options(compression: str | None = None, compression_level: int | None = None, chunk_size: int | None = None)

Bases: _OptionGroup

HDF5 storage settings, as a group.

In a TOML config file this is the [profiling.hdf5] table.

Attributes

compression : str or None Compression filter for timestamp and GPU-duration datasets (hdf5_compression): "gzip", "lzf", "zstd", or "auto" to compress only runs large enough to repay the write CPU. compression_level : int or None GZIP level 0—9 or Zstandard level 1—22 (hdf5_compression_level). chunk_size : int or None Maximum events per dataset chunk (hdf5_chunk_size). Enables chunked partial reads even without compression.

chunk_sizeattributeclass attributeinstance attribute#

View source

chunk_size: int | None = None

compressionattributeclass attributeinstance attribute#

View source

compression: str | None = None

compression_levelattributeclass attributeinstance attribute#

View source

compression_level: int | None = None

Inherited from scope_profiler.profile_config._OptionGroup

to_kwargsmethod#

Inherited from scope_profiler.profile_config._OptionGroupView source

def to_kwargs() -> dict

This group’s explicitly-set fields, under their flat names.

load_profiling_configfunction#

View source

def load_profiling_config(path: str | os.PathLike[str]) -> dict

Load the [profiling] table from a TOML settings file.

A top-level table is also accepted for small files. Paths in the file retain TOML’s normal meaning and are interpreted relative to the current working directory, just like paths passed to :meth:ProfileManager.setup.

Settings may be written flat, or grouped into the memray, gpu and hdf5 sub-tables that mirror :class:MemrayOptions, :class:GPUOptions and :class:HDF5Options::

[profiling]
file_path = "run.h5"

[profiling.hdf5]
compression = "gzip"
compression_level = 4

Setting the same thing both ways is an error, not a silent precedence rule.

ProfilingConfig

Holds one manager’s profiling configuration. Normally you interact with it through ProfileManager.setup(), but you can also construct one directly for advanced use cases.

ProfilingConfigclass#

View source

class ProfilingConfig(file_path: str = 'profiling_data.h5', label: str | None = None, use_likwid: bool = False, perf_events: list[str] | tuple[str, ...] | str | None = None, use_line_profiler: bool = False, use_memray: bool = False, memory_profile_path: str | None = None, memray_native_traces: bool = False, memray_trace_python_allocators: bool = False, memray_follow_fork: bool = False, deactivate_profiling: bool = False, use_nvtx: bool = False, use_gpu_timing: bool = False, gpu_timing_backend='auto', deactivate_file_output: bool = False, recursive_profile: bool = False, aggregation_mode: bool = False, profile_mpi_calls: bool = False, track_threads: bool = False, track_async: bool = False, capture_region_source: bool = False, metadata_detail: str = 'minimal', buffer_limit: int = 1024, output_mode: str = 'auto', hdf5_compression: str | None = None, hdf5_compression_level: int | None = None, hdf5_chunk_size: int | None = None, gpu_sync_on_exit: bool = False, enable_tags: list[str] | tuple[str, ...] | str | None = None)

Configuration for one profiling manager.

This class centralizes configuration for LIKWID performance counters, buffer limits, and file paths. Each manager owns a separate instance, so independently configured profiling sessions can coexist. Constructing it is purely local: it reads the communicator for rank and size but issues no MPI call of its own, so setup() does not have to be collective.

aggregation_modeproperty#

View source

aggregation_mode: bool

Whether regions retain aggregates instead of individual events.

buffer_limitproperty#

View source

buffer_limit: int

Initial per-region buffer capacity; buffers grow beyond it as needed.

capture_region_sourceproperty#

View source

capture_region_source: bool

Return whether a region’s defining source is captured at creation.

commproperty#

View source

comm: Intercomm | None

MPI communicator or None if MPI is unavailable.

deactivate_file_outputproperty#

View source

deactivate_file_output: bool

Return whether the run writes no HDF5 file at all.

deactivate_profilingproperty#

View source

deactivate_profiling: bool

Return whether profiling is globally turned off.

enable_tagsproperty#

View source

enable_tags: tuple[str, ...]

Opt-in region tags this run records, or an empty tuple.

file_pathproperty#

View source

file_path: str

Global output file path for combined profiling data.

gpu_sync_on_exitproperty#

View source

gpu_sync_on_exit: bool

Whether GPU regions wait for the device before recording CPU end time.

gpu_timing_backendproperty#

View source

gpu_timing_backend

CUDA-event timing backend selector or backend object.

hdf5_chunk_sizeproperty#

View source

hdf5_chunk_size: int | None

Maximum number of events per timestamp chunk.

hdf5_compressionproperty#

View source

hdf5_compression: str | None

Compression filter used for timestamp datasets.

hdf5_compression_levelproperty#

View source

hdf5_compression_level: int | None

Configured GZIP or Zstandard compression level.

labelproperty#

View source

label: str | None

Short name for this run, or None if none was given to setup().

memory_profile_pathproperty#

View source

memory_profile_path: Path | None

Memray capture path, or None when allocation profiling is off.

metadataproperty#

View source

metadata: dict

Environment metadata collected on this rank (hostname, OpenMP threads, …).

metadata_detailproperty#

View source

metadata_detail: str

"full" or "minimal": how much environment metadata is recorded.

output_modeproperty#

View source

output_mode: str

MPI HDF5 output strategy: auto, direct, or parallel.

pausedproperty#

View source

paused: bool

Whether runtime timing collection is temporarily suspended.

perf_eventsproperty#

View source

perf_events: tuple[str, ...]

Requested Linux perf event names, or an empty tuple when disabled.

profile_mpi_callsproperty#

View source

profile_mpi_calls: bool

Whether mpi4py communicator operations are profiled.

recursive_profileproperty#

View source

recursive_profile: bool

Return whether recursive decorator profiling is enabled by default.

start_time_nsproperty#

View source

start_time_ns: int

The run’s start time (ns, perf_counter_ns clock).

The moment the configuration was created. Persisted as metadata and used as the timeline origin when reading the results back.

track_asyncproperty#

View source

track_async: bool

Whether every call records its asyncio task or greenlet.

track_threadsproperty#

View source

track_threads: bool

Whether every call records the thread it ran on.

trackerproperty#

View source

tracker

This run’s :class:~scope_profiler.concurrency.ConcurrencyTracker.

None unless track_threads is set, which is what keeps the default configuration free of any thread, asyncio or greenlet hook.

use_gpu_timingproperty#

View source

use_gpu_timing: bool

Return whether CUDA-event GPU timing is enabled.

use_likwidproperty#

View source

use_likwid: bool

Return whether LIKWID profiling is enabled.

use_line_profilerproperty#

View source

use_line_profiler: bool

Return whether line_profiler profiling is enabled.

use_memrayproperty#

View source

use_memray: bool

Return whether Memray allocation profiling is enabled.

use_nvtxproperty#

View source

use_nvtx: bool

Return whether NVTX annotations are enabled.

collect_likwid_resultsmethod#

View source

def collect_likwid_results(region_names) -> list

Close the LIKWID markers and return the run’s counter results.

Tries three sources, richest first, so a host where LIKWID cannot do the fancy parts still ends up with real numbers in the HDF5 file:

1. the perfmon read-back, run in a subprocess because it can crash the interpreter outright on hosts that cannot really count;
2. LIKWID’s marker file, parsed directly --- real values, but no event names or derived metrics;
3. a marker-API snapshot taken before the markers were closed, for when there is no marker file at all.

See :mod:scope_profiler.likwid_data for why the first one is fenced off.

Parameters

region_names : iterable of str Region names to snapshot via the marker API.

Returns

list of scope_profiler.likwid_data.LikwidRegionResult Empty when LIKWID is disabled, the process was not started under likwid-perfctr -m / likwid-mpirun -marker, or the markers were already collected.

Notes

A process can only do this once. Closing the markers tears the marker API down, and querying it afterwards crashes the interpreter rather than raising, so a second finalize() returns nothing instead of reading again. The counters of the whole process therefore end up in the file written by the first finalize().

likwid_environmentmethod#

View source

def likwid_environment() -> dict

Return the LIKWID_* environment variables of this process.

pylikwid_markerclosemethod#

View source

def pylikwid_markerclose()

Close LIKWID markers to finalize measurement regions.

Idempotent: repeated calls (a second finalize(), say) do nothing.

pylikwid_markerinitmethod#

View source

def pylikwid_markerinit()

Initialize LIKWID markers if LIKWID is enabled.

resetmethodclassmethod#

View source

def reset()

Compatibility no-op retained for callers of older releases.

Configurations are ordinary per-manager objects now, so there is no process-wide instance to reset.

stop_memory_profilingmethod#

View source

def stop_memory_profiling() -> None

Finish Memray’s process-wide allocation capture, if enabled.

Region classes

BaseProfileRegion

BaseProfileRegionclass#

View source

class BaseProfileRegion(region_name: str, config: ProfilingConfig, tags=())

Base class providing shared profiling logic.

Handles start/end time buffering and call counting. The buffers grow on demand and are copied out once, at the end of the run, by ProfileManager.finalize() - regions never touch HDF5 themselves.

buffer_limitattributeinstance attribute#

View source

buffer_limit = config.buffer_limit

capacityattributeinstance attribute#

View source

capacity = self.buffer_limit

configattributeinstance attribute#

View source

config = config

end_timesattributeinstance attribute#

View source

end_times = np.zeros(self.capacity, dtype=np.int64)

ptrattributeinstance attribute#

View source

ptr = 0

region_nameattributeinstance attribute#

View source

region_name = region_name

source_fileattributeinstance attribute#

View source

source_file = None

source_linenoattributeinstance attribute#

View source

source_lineno = None

source_textattributeinstance attribute#

View source

source_text = None

start_timesattributeinstance attribute#

View source

start_times = np.empty(self.capacity, dtype=np.int64)

tagsattributeinstance attribute#

View source

tags = tuple(tags)

has_sourceproperty#

View source

has_source: bool

Whether this region’s call-site source was captured.

num_callsproperty#

View source

num_calls: int

Times this region was entered, for the lifetime of the process.

Derived rather than counted: the slots in use (ptr) plus whatever earlier finalize() calls already copied out. Keeping it out of __enter__ removes an attribute write from every recorded call.

add_functionmethod#

View source

def add_function(func) -> None

Register a function for profiling. No-op except in LineProfilerRegion.

appendmethod#

View source

def append(start: float, end: float) -> None

Append a start/end time pair to the buffer, growing it if needed.

closed_slotsmethod#

View source

def closed_slots() -> np.ndarray | None

Mask of buffered slots this finalize() should copy out.

A slot qualifies once its end time is written and it has not already gone out with an earlier finalize(). None - the usual case - means every slot qualifies and the caller can skip the mask entirely.

get_durations_numpymethod#

View source

def get_durations_numpy() -> np.ndarray

Return durations (end - start) for buffered entries as a NumPy array.

get_end_times_numpymethod#

View source

def get_end_times_numpy() -> np.ndarray

Return end times offset by the run’s start time.

get_start_times_numpymethod#

View source

def get_start_times_numpy() -> np.ndarray

Return start times offset by the run’s start time.

mark_writtenmethod#

View source

def mark_written() -> None

Record that everything buffered so far has been handed to finalize().

Called by finalize() once the data has been copied out, so that a second run in the same process reports only its own events instead of re-reporting the first run’s. The timestamp buffer rewinds (the arrays are reused; anything past ptr is unread scratch), while num_calls keeps counting for the lifetime of the process — it is the in-memory view of the region, which callers inspect after finalize().

A call still running has a slot reserved that finalize() did not copy out, so it must survive the rewind. It is moved to the front of the buffer rather than pinning everything behind it: leaving the whole buffer in place would make the next finalize() re-report every completed call sitting below it, with a second set of call ids.

The one case that still cannot rewind is a call entered through the decorator form, which keeps its slot index in the wrapper’s own frame where nothing can remap it. Recognisable because such a slot is open without appearing in _scope_ptr_stack. There the buffer stays put and the slots already copied out are recorded instead, so the next finalize() skips them rather than reporting those calls again.

open_slotsmethod#

View source

def open_slots() -> np.ndarray

Buffer slots whose call is still running, in ascending order.

A slot is open until its end time is written, which is what distinguishes a call in flight from one that has returned. Both region forms are covered: the context manager writes its start on entry, the decorator writes nothing until the call returns.

set_sourcemethod#

View source

def set_source(filename, lineno, text) -> None

Record where this region is defined, the first time it is called.

First writer wins: a region name reused at more than one call site keeps only the first location, matching how their timings are already pooled together under one name (see issue #161).

wrapmethod#

View source

def wrap(func)

Wrap a function for profiling.

Subclasses must override this method to implement the appropriate profiling behavior.

DisabledProfileRegion

DisabledProfileRegionclass#

View source

class DisabledProfileRegion(BaseProfileRegion)

Bases: BaseProfileRegion

Profiling region that performs no measurements.

Used when profiling is disabled but code paths must remain valid.

appendmethod#

View source

def append(start, end)

Ignored: no data recorded.

get_durations_numpymethod#

View source

def get_durations_numpy()

Return an empty array since nothing is recorded.

wrapmethod#

View source

def wrap(func)

Return the original function unchanged — no wrapper, no overhead.

Inherited from scope_profiler.region_profiler.BaseProfileRegion

buffer_limitattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

buffer_limit = config.buffer_limit

capacityattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

capacity = self.buffer_limit

configattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

config = config

end_timesattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

end_times = np.zeros(self.capacity, dtype=np.int64)

ptrattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

ptr = 0

region_nameattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

region_name = region_name

source_fileattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

source_file = None

source_linenoattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

source_lineno = None

source_textattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

source_text = None

start_timesattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

start_times = np.empty(self.capacity, dtype=np.int64)

tagsattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

tags = tuple(tags)

has_sourceproperty#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

has_source: bool

Whether this region’s call-site source was captured.

num_callsproperty#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

num_calls: int

Times this region was entered, for the lifetime of the process.

Derived rather than counted: the slots in use (ptr) plus whatever earlier finalize() calls already copied out. Keeping it out of __enter__ removes an attribute write from every recorded call.

add_functionmethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def add_function(func) -> None

Register a function for profiling. No-op except in LineProfilerRegion.

closed_slotsmethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def closed_slots() -> np.ndarray | None

Mask of buffered slots this finalize() should copy out.

A slot qualifies once its end time is written and it has not already gone out with an earlier finalize(). None - the usual case - means every slot qualifies and the caller can skip the mask entirely.

get_end_times_numpymethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def get_end_times_numpy() -> np.ndarray

Return end times offset by the run’s start time.

get_start_times_numpymethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def get_start_times_numpy() -> np.ndarray

Return start times offset by the run’s start time.

mark_writtenmethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def mark_written() -> None

Record that everything buffered so far has been handed to finalize().

Called by finalize() once the data has been copied out, so that a second run in the same process reports only its own events instead of re-reporting the first run’s. The timestamp buffer rewinds (the arrays are reused; anything past ptr is unread scratch), while num_calls keeps counting for the lifetime of the process — it is the in-memory view of the region, which callers inspect after finalize().

A call still running has a slot reserved that finalize() did not copy out, so it must survive the rewind. It is moved to the front of the buffer rather than pinning everything behind it: leaving the whole buffer in place would make the next finalize() re-report every completed call sitting below it, with a second set of call ids.

The one case that still cannot rewind is a call entered through the decorator form, which keeps its slot index in the wrapper’s own frame where nothing can remap it. Recognisable because such a slot is open without appearing in _scope_ptr_stack. There the buffer stays put and the slots already copied out are recorded instead, so the next finalize() skips them rather than reporting those calls again.

open_slotsmethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def open_slots() -> np.ndarray

Buffer slots whose call is still running, in ascending order.

A slot is open until its end time is written, which is what distinguishes a call in flight from one that has returned. Both region forms are covered: the context manager writes its start on entry, the decorator writes nothing until the call returns.

set_sourcemethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def set_source(filename, lineno, text) -> None

Record where this region is defined, the first time it is called.

First writer wins: a region name reused at more than one call site keeps only the first location, matching how their timings are already pooled together under one name (see issue #161).

TimeOnlyProfileRegion

TimeOnlyProfileRegionclass#

View source

class TimeOnlyProfileRegion(BaseProfileRegion)

Bases: BaseProfileRegion

Region that records timing, collected once at the end of the run.

wrapmethod#

View source

def wrap(func)

Wrap a function to measure its execution time.

Inherited from scope_profiler.region_profiler.BaseProfileRegion

buffer_limitattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

buffer_limit = config.buffer_limit

capacityattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

capacity = self.buffer_limit

configattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

config = config

end_timesattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

end_times = np.zeros(self.capacity, dtype=np.int64)

ptrattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

ptr = 0

region_nameattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

region_name = region_name

source_fileattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

source_file = None

source_linenoattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

source_lineno = None

source_textattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

source_text = None

start_timesattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

start_times = np.empty(self.capacity, dtype=np.int64)

tagsattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

tags = tuple(tags)

has_sourceproperty#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

has_source: bool

Whether this region’s call-site source was captured.

num_callsproperty#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

num_calls: int

Times this region was entered, for the lifetime of the process.

Derived rather than counted: the slots in use (ptr) plus whatever earlier finalize() calls already copied out. Keeping it out of __enter__ removes an attribute write from every recorded call.

add_functionmethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def add_function(func) -> None

Register a function for profiling. No-op except in LineProfilerRegion.

appendmethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def append(start: float, end: float) -> None

Append a start/end time pair to the buffer, growing it if needed.

closed_slotsmethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def closed_slots() -> np.ndarray | None

Mask of buffered slots this finalize() should copy out.

A slot qualifies once its end time is written and it has not already gone out with an earlier finalize(). None - the usual case - means every slot qualifies and the caller can skip the mask entirely.

get_durations_numpymethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def get_durations_numpy() -> np.ndarray

Return durations (end - start) for buffered entries as a NumPy array.

get_end_times_numpymethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def get_end_times_numpy() -> np.ndarray

Return end times offset by the run’s start time.

get_start_times_numpymethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def get_start_times_numpy() -> np.ndarray

Return start times offset by the run’s start time.

mark_writtenmethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def mark_written() -> None

Record that everything buffered so far has been handed to finalize().

Called by finalize() once the data has been copied out, so that a second run in the same process reports only its own events instead of re-reporting the first run’s. The timestamp buffer rewinds (the arrays are reused; anything past ptr is unread scratch), while num_calls keeps counting for the lifetime of the process — it is the in-memory view of the region, which callers inspect after finalize().

A call still running has a slot reserved that finalize() did not copy out, so it must survive the rewind. It is moved to the front of the buffer rather than pinning everything behind it: leaving the whole buffer in place would make the next finalize() re-report every completed call sitting below it, with a second set of call ids.

The one case that still cannot rewind is a call entered through the decorator form, which keeps its slot index in the wrapper’s own frame where nothing can remap it. Recognisable because such a slot is open without appearing in _scope_ptr_stack. There the buffer stays put and the slots already copied out are recorded instead, so the next finalize() skips them rather than reporting those calls again.

open_slotsmethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def open_slots() -> np.ndarray

Buffer slots whose call is still running, in ascending order.

A slot is open until its end time is written, which is what distinguishes a call in flight from one that has returned. Both region forms are covered: the context manager writes its start on entry, the decorator writes nothing until the call returns.

set_sourcemethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def set_source(filename, lineno, text) -> None

Record where this region is defined, the first time it is called.

First writer wins: a region name reused at more than one call site keeps only the first location, matching how their timings are already pooled together under one name (see issue #161).

FullProfileRegion

FullProfileRegionclass#

View source

class FullProfileRegion(region_name: str, config: ProfilingConfig, tags=())

Bases: BaseProfileRegion

Region that records both timing and LIKWID metrics, and writes to HDF5.

This is the most complete profiling mode: users obtain LIKWID markers and nanosecond-resolution timing.

likwid_marker_startattributeinstance attribute#

View source

likwid_marker_start = pylikwid.markerstartregion

likwid_marker_stopattributeinstance attribute#

View source

likwid_marker_stop = pylikwid.markerstopregion

wrapmethod#

View source

def wrap(func)

Wrap a function to measure time and collect LIKWID metrics.

Inherited from scope_profiler.region_profiler.BaseProfileRegion

buffer_limitattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

buffer_limit = config.buffer_limit

capacityattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

capacity = self.buffer_limit

configattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

config = config

end_timesattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

end_times = np.zeros(self.capacity, dtype=np.int64)

ptrattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

ptr = 0

region_nameattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

region_name = region_name

source_fileattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

source_file = None

source_linenoattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

source_lineno = None

source_textattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

source_text = None

start_timesattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

start_times = np.empty(self.capacity, dtype=np.int64)

tagsattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

tags = tuple(tags)

has_sourceproperty#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

has_source: bool

Whether this region’s call-site source was captured.

num_callsproperty#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

num_calls: int

Times this region was entered, for the lifetime of the process.

Derived rather than counted: the slots in use (ptr) plus whatever earlier finalize() calls already copied out. Keeping it out of __enter__ removes an attribute write from every recorded call.

add_functionmethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def add_function(func) -> None

Register a function for profiling. No-op except in LineProfilerRegion.

appendmethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def append(start: float, end: float) -> None

Append a start/end time pair to the buffer, growing it if needed.

closed_slotsmethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def closed_slots() -> np.ndarray | None

Mask of buffered slots this finalize() should copy out.

A slot qualifies once its end time is written and it has not already gone out with an earlier finalize(). None - the usual case - means every slot qualifies and the caller can skip the mask entirely.

get_durations_numpymethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def get_durations_numpy() -> np.ndarray

Return durations (end - start) for buffered entries as a NumPy array.

get_end_times_numpymethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def get_end_times_numpy() -> np.ndarray

Return end times offset by the run’s start time.

get_start_times_numpymethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def get_start_times_numpy() -> np.ndarray

Return start times offset by the run’s start time.

mark_writtenmethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def mark_written() -> None

Record that everything buffered so far has been handed to finalize().

Called by finalize() once the data has been copied out, so that a second run in the same process reports only its own events instead of re-reporting the first run’s. The timestamp buffer rewinds (the arrays are reused; anything past ptr is unread scratch), while num_calls keeps counting for the lifetime of the process — it is the in-memory view of the region, which callers inspect after finalize().

A call still running has a slot reserved that finalize() did not copy out, so it must survive the rewind. It is moved to the front of the buffer rather than pinning everything behind it: leaving the whole buffer in place would make the next finalize() re-report every completed call sitting below it, with a second set of call ids.

The one case that still cannot rewind is a call entered through the decorator form, which keeps its slot index in the wrapper’s own frame where nothing can remap it. Recognisable because such a slot is open without appearing in _scope_ptr_stack. There the buffer stays put and the slots already copied out are recorded instead, so the next finalize() skips them rather than reporting those calls again.

open_slotsmethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def open_slots() -> np.ndarray

Buffer slots whose call is still running, in ascending order.

A slot is open until its end time is written, which is what distinguishes a call in flight from one that has returned. Both region forms are covered: the context manager writes its start on entry, the decorator writes nothing until the call returns.

set_sourcemethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def set_source(filename, lineno, text) -> None

Record where this region is defined, the first time it is called.

First writer wins: a region name reused at more than one call site keeps only the first location, matching how their timings are already pooled together under one name (see issue #161).

LineProfilerRegion

LineProfilerRegionclass#

View source

class LineProfilerRegion(region_name: str, config: ProfilingConfig, tags=())

Bases: BaseProfileRegion

Region that records timing and line-by-line profiling via line_profiler.

Uses line_profiler to collect per-line execution statistics for decorated functions. Also records nanosecond timestamps.

Line-by-line profiling is most useful with the decorator (wrap) path, which automatically registers the function with the line profiler. When used as a context manager, the profiler is enabled/disabled around the block - any functions previously added via the decorator path will be profiled while the context is active.

add_functionmethod#

View source

def add_function(func) -> None

Register a function for line-by-line profiling.

enter_framemethod#

View source

def enter_frame(frame)

Enter this region while registering frame with line_profiler.

enter_timing_onlymethod#

View source

def enter_timing_only()

Enter this region without registering or enabling line_profiler.

get_statsmethod#

View source

def get_stats()

Return the line_profiler stats object.

manual_line_recordsmethod#

View source

def manual_line_records(unit: float = 1e-09) -> list

Return manually traced line timings in the persisted record shape.

print_statsmethod#

View source

def print_stats()

Print line-by-line profiling statistics.

record_line_timingmethod#

View source

def record_line_timing(frame, lineno: int, duration_ns: int) -> None

Record one line timing sample from recursive CLI tracing.

wrapmethod#

View source

def wrap(func)

Wrap a function to measure execution time and collect line-by-line stats.

Inherited from scope_profiler.region_profiler.BaseProfileRegion

buffer_limitattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

buffer_limit = config.buffer_limit

capacityattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

capacity = self.buffer_limit

configattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

config = config

end_timesattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

end_times = np.zeros(self.capacity, dtype=np.int64)

ptrattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

ptr = 0

region_nameattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

region_name = region_name

source_fileattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

source_file = None

source_linenoattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

source_lineno = None

source_textattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

source_text = None

start_timesattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

start_times = np.empty(self.capacity, dtype=np.int64)

tagsattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

tags = tuple(tags)

has_sourceproperty#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

has_source: bool

Whether this region’s call-site source was captured.

num_callsproperty#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

num_calls: int

Times this region was entered, for the lifetime of the process.

Derived rather than counted: the slots in use (ptr) plus whatever earlier finalize() calls already copied out. Keeping it out of __enter__ removes an attribute write from every recorded call.

appendmethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def append(start: float, end: float) -> None

Append a start/end time pair to the buffer, growing it if needed.

closed_slotsmethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def closed_slots() -> np.ndarray | None

Mask of buffered slots this finalize() should copy out.

A slot qualifies once its end time is written and it has not already gone out with an earlier finalize(). None - the usual case - means every slot qualifies and the caller can skip the mask entirely.

get_durations_numpymethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def get_durations_numpy() -> np.ndarray

Return durations (end - start) for buffered entries as a NumPy array.

get_end_times_numpymethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def get_end_times_numpy() -> np.ndarray

Return end times offset by the run’s start time.

get_start_times_numpymethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def get_start_times_numpy() -> np.ndarray

Return start times offset by the run’s start time.

mark_writtenmethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def mark_written() -> None

Record that everything buffered so far has been handed to finalize().

Called by finalize() once the data has been copied out, so that a second run in the same process reports only its own events instead of re-reporting the first run’s. The timestamp buffer rewinds (the arrays are reused; anything past ptr is unread scratch), while num_calls keeps counting for the lifetime of the process — it is the in-memory view of the region, which callers inspect after finalize().

A call still running has a slot reserved that finalize() did not copy out, so it must survive the rewind. It is moved to the front of the buffer rather than pinning everything behind it: leaving the whole buffer in place would make the next finalize() re-report every completed call sitting below it, with a second set of call ids.

The one case that still cannot rewind is a call entered through the decorator form, which keeps its slot index in the wrapper’s own frame where nothing can remap it. Recognisable because such a slot is open without appearing in _scope_ptr_stack. There the buffer stays put and the slots already copied out are recorded instead, so the next finalize() skips them rather than reporting those calls again.

open_slotsmethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def open_slots() -> np.ndarray

Buffer slots whose call is still running, in ascending order.

A slot is open until its end time is written, which is what distinguishes a call in flight from one that has returned. Both region forms are covered: the context manager writes its start on entry, the decorator writes nothing until the call returns.

set_sourcemethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def set_source(filename, lineno, text) -> None

Record where this region is defined, the first time it is called.

First writer wins: a region name reused at more than one call site keeps only the first location, matching how their timings are already pooled together under one name (see issue #161).

ThreadedProfileRegion

Selected by track_threads=True; see Threads, tasks and coroutines.

ThreadedProfileRegionclass#

View source

class ThreadedProfileRegion(region_name: str, config: ProfilingConfig, tags=())

Bases: BaseProfileRegion

Region that records which thread - and which task - each call ran on.

Selected by track_threads=True. Two things change against :class:TimeOnlyProfileRegion:

• every thread gets its own buffers and its own scope stack, so concurrent calls no longer overwrite each other’s reserved slots or close each other’s scopes;
• each recorded call carries the lane it ran on, so the nesting reconstruction can group calls into stacks that really are stacks (see :func:~scope_profiler.concurrency.lane_ids).

With track_async=True a call additionally carries the id of the asyncio task or greenlet it ran in, and the time that task spent suspended inside the call - the await time of that one call, as opposed to the task’s total.

The per-call cost over :class:TimeOnlyProfileRegion is one thread-local lookup plus, in async mode, two integer reads off the current task record. Nothing here allocates or locks per call.

num_callsproperty#

View source

num_calls: int

Times this region was entered, over every thread, for the process.

threadsproperty#

View source

threads: list

Thread records that have entered this region, in index order.

closed_slotsmethod#

View source

def closed_slots()

Not meaningful per lane; :meth:snapshot_arrays applies the mask.

None, as the base class means it: every slot qualifies, and there is no mask for a caller to apply. A threaded region keeps no buffers of its own, so there is nothing here to mask in the first place.

mark_writtenmethod#

View source

def mark_written() -> None

Rewind every lane, keeping calls that are still running.

open_slotsmethod#

View source

def open_slots() -> np.ndarray

Slots still running, over every lane. For diagnostics only.

snapshot_arraysmethod#

View source

def snapshot_arrays()

This run’s completed calls, concatenated over every thread.

Returns

tuple or None (start_times, end_times, None, None, None, thread_ids, task_ids, await_ns) in nanoseconds, laid out on the snapshot tuple :meth:ProfileManager._snapshot_regions documents, or None when this region recorded nothing since the last finalize(). task_ids and await_ns are None unless async tracking is on.

wrapmethod#

View source

def wrap(func)

Wrap a function so its calls are recorded on the caller’s thread.

Inherited from scope_profiler.region_profiler.BaseProfileRegion

buffer_limitattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

buffer_limit = config.buffer_limit

capacityattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

capacity = self.buffer_limit

configattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

config = config

end_timesattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

end_times = np.zeros(self.capacity, dtype=np.int64)

ptrattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

ptr = 0

region_nameattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

region_name = region_name

source_fileattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

source_file = None

source_linenoattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

source_lineno = None

source_textattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

source_text = None

start_timesattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

start_times = np.empty(self.capacity, dtype=np.int64)

tagsattributeinstance attribute#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

tags = tuple(tags)

has_sourceproperty#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

has_source: bool

Whether this region’s call-site source was captured.

add_functionmethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def add_function(func) -> None

Register a function for profiling. No-op except in LineProfilerRegion.

appendmethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def append(start: float, end: float) -> None

Append a start/end time pair to the buffer, growing it if needed.

get_durations_numpymethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def get_durations_numpy() -> np.ndarray

Return durations (end - start) for buffered entries as a NumPy array.

get_end_times_numpymethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def get_end_times_numpy() -> np.ndarray

Return end times offset by the run’s start time.

get_start_times_numpymethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def get_start_times_numpy() -> np.ndarray

Return start times offset by the run’s start time.

set_sourcemethod#

Inherited from scope_profiler.region_profiler.BaseProfileRegionView source

def set_source(filename, lineno, text) -> None

Record where this region is defined, the first time it is called.

First writer wins: a region name reused at more than one call site keeps only the first location, matching how their timings are already pooled together under one name (see issue #161).

Threads, tasks and coroutines

A run profiled with track_threads describes the lanes its calls ran on. The records below are what ProfilingResults.threads and ProfilingResults.tasks hand back.

ThreadInfoclass#

View source

class ThreadInfo(index: int, rank: int, name: str, ident: int, native_id: int, daemon: bool, start_ns: int, end_ns: int, cpu_ns: int)

One row of a run’s thread table, as post-processing sees it.

The read-back counterpart of :class:ThreadRecord: same fields, but durations in seconds and timestamps relative to the start of the run, matching every other time reported by :class:~scope_profiler.results.ProfilingResults.

cpu_timeattributeinstance attribute#

View source

cpu_time = float(cpu_ns) / 1000000000.0

daemonattributeinstance attribute#

View source

daemon = bool(daemon)

end_timeattributeinstance attribute#

View source

end_time = None if int(end_ns) == UNKNOWN else float(end_ns) / 1000000000.0

identattributeinstance attribute#

View source

ident = int(ident)

indexattributeinstance attribute#

View source

index = int(index)

nameattributeinstance attribute#

View source

name = str(name)

native_idattributeinstance attribute#

View source

native_id = int(native_id)

rankattributeinstance attribute#

View source

rank = int(rank)

start_timeattributeinstance attribute#

View source

start_time = float(start_ns) / 1000000000.0

aliveproperty#

View source

alive: bool

Whether the thread was still running when the run was collected.

wall_timeproperty#

View source

wall_time: float | None

Seconds between the thread starting and ending, or None if alive.

TaskInfoclass#

View source

class TaskInfo(index: int, rank: int, kind: str, name: str, coro_name: str, thread_index: int, created_ns: int, done_ns: int, steps: int, running_ns: int, suspended_ns: int)

One row of a run’s task table: an asyncio task or a greenlet.

running_time and awaiting_time split the lane’s wall time into the part it held its thread and the part it did not, which is the number await-heavy code is usually looking for.

awaiting_timeattributeinstance attribute#

View source

awaiting_time = float(suspended_ns) / 1000000000.0

coro_nameattributeinstance attribute#

View source

coro_name = str(coro_name)

created_timeattributeinstance attribute#

View source

created_time = float(created_ns) / 1000000000.0

done_timeattributeinstance attribute#

View source

done_time = None if int(done_ns) == UNKNOWN else float(done_ns) / 1000000000.0

indexattributeinstance attribute#

View source

index = int(index)

kindattributeinstance attribute#

View source

kind = str(kind)

nameattributeinstance attribute#

View source

name = str(name)

rankattributeinstance attribute#

View source

rank = int(rank)

running_timeattributeinstance attribute#

View source

running_time = float(running_ns) / 1000000000.0

stepsattributeinstance attribute#

View source

steps = int(steps)

thread_indexattributeinstance attribute#

View source

thread_index = int(thread_index)

doneproperty#

View source

done: bool

Whether the task finished before the run was collected.

wall_timeproperty#

View source

wall_time: float | None

Seconds from creation to completion, or None if it never finished.

ConcurrencyTrackerclass#

View source

class ConcurrencyTracker(track_async: bool = False)

Per-run registry of threads, asyncio tasks and greenlets.

One tracker belongs to one :class:~scope_profiler.profile_config.ProfilingConfig, so two simultaneously active managers never share lane numbering. Creating it is free; :meth:install is what puts the (few, cheap) hooks in place, and :meth:uninstall takes them out again.

track_asyncattributeinstance attribute#

View source

track_async = bool(track_async)

tasksproperty#

View source

tasks: list[TaskRecord]

Every instrumented task and greenlet, in creation order.

threadsproperty#

View source

threads: list[ThreadRecord]

Every thread that recorded a call, in registration order.

current_threadmethod#

View source

def current_thread() -> ThreadRecord

This thread’s record, registering it on first use.

On the region hot path. The common case is one failed-then-cached attribute lookup on a :class:threading.local, which is why the registry itself is never consulted here.

installmethod#

View source

def install() -> None

Put the thread-start, asyncio and greenlet hooks in place.

Idempotent, so a second setup() on the same configuration does not stack wrappers.

instrument_loopmethod#

View source

def instrument_loop(loop) -> None

Route loop’s task creation through the step timer.

Public because a loop implementation that never reaches BaseEventLoop.run_forever (uvloop, say) can be handed here directly. Chains onto whatever task factory the application already installed instead of replacing it.

Instrumented once per loop, and keyed on the loop rather than on whichever factory is currently in place: an application that installs its own factory after the loop has started displaces ours, and re-wrapping to get back on top would make the two call each other forever, since its captured “previous” is us. Such a loop keeps recording regions and thread ids; the tasks created after the swap simply have no lane of their own.

snapshotmethod#

View source

def snapshot(origin_ns: int = 0) -> dict

The lane tables, as columns, ready to be written or returned.

Timestamps are shifted by origin_ns (the run’s start), matching what the event columns store, so a thread’s lifetime and the calls on it share one timeline. :data:UNKNOWN entries are left alone.

Returns

dict {"threads": {...}, "tasks": {...}}, each a dict of equal-length numpy arrays. Empty arrays when nothing was tracked.

uninstallmethod#

View source

def uninstall() -> None

Remove every hook and restore what was there before.

lane_idsfunction#

View source

def lane_ids(thread_ids: np.ndarray, task_ids: np.ndarray | None) -> np.ndarray

One id per call identifying the stack it belongs to.

A call belongs to its task when it ran inside one and to its bare thread otherwise. The two id spaces are disjoint by construction: thread lanes are mapped to -2 - thread, task lanes stay non-negative, and -1 is left for a call whose thread is unknown - a region recorded by the Fortran API, say, folded into an otherwise thread-aware run. This is the key :func:~scope_profiler.call_stack.build_call_arrays groups by, since only calls sharing a lane can nest.

Post-processing

Everything in this section is importable from the package root:

from scope_profiler import (
    build_call_arrays,
    build_call_stack,
    load,
    read_h5,
    read_h5_summary,
    read_json,
    read_profile,
    plot_flame_chart,
    plot_gantt,
    write_region_statistics_json,
)

All durations and timestamps exposed here are in seconds; the HDF5 file stores nanoseconds.

ProfilingResults

The post-processing API itself, and the only type the analysis layer has. ProfilingResults.from_h5() builds one from a merged file, and ProfileManager.finalize(return_results=True) builds the same thing straight from memory.

ProfilingResultsclass#

View source

class ProfilingResults(regions: dict[str, MPIRegion], metadata: dict | None = None, num_ranks: int | None = None, likwid: dict[int, dict[str, LikwidRegionResult]] | None = None, perf_events: dict[int, dict] | None = None, line_profile: dict[int, list] | None = None, file_path: str | Path = '', is_root: bool = True, exclusive_totals: dict[str, dict[int, int]] | None = None, event_data_available: bool = True, threads: dict[int, list] | None = None, tasks: dict[int, list] | None = None)

The profiling data of one run, as an ordered mapping of region name to :class:~scope_profiler.mpi_region.MPIRegion::

results = ProfileManager.finalize(return_results=True)
for region in results:
    print(region.name, region.total_duration)

solve = results["solve"]        # same as results.get_region("solve")
solve[0].average_duration       # rank 0, in seconds

The same object comes back from a merged output file, via :meth:from_h5 (or its module-level twin :func:~scope_profiler.h5reader.read_h5)::

results = ProfilingResults.from_h5("profiling_data.h5")

All durations are reported in seconds.

display_labelproperty#

View source

display_label: str

What to call this run in charts, tables and exports.

The :attr:label when the run has one, and otherwise the stem of its output file --- which is what post-processing named runs by before labels existed, and remains the default.

Returns

str A non-empty name for the run.

file_pathproperty#

View source

file_path: Path

The run’s output file.

Returns

Path The file path as a pathlib.Path object. For results taken straight from memory this is the path setup() was configured with, which need not exist on disk.

finalize_timeproperty#

View source

finalize_time: float | None

When finalize() was called, in seconds on the recording clock.

This is the finalize_time_ns metadata field, read as the first thing ProfileManager.finalize() does - before it spends any time collecting or writing the run’s data, so it marks the moment finalize() was reached rather than the moment it returned.

Returns

float or None The registered finalize time, or None for runs without one (older files, or a run that never reached finalize()).

has_event_dataproperty#

View source

has_event_data: bool

Whether per-call timestamps are available to event-based APIs.

has_likwidproperty#

View source

has_likwid: bool

Whether the run recorded LIKWID hardware counter results.

has_perf_eventsproperty#

View source

has_perf_events: bool

Whether this run recorded built-in Linux perf-event counters.

is_rootproperty#

View source

is_root: bool

Whether this rank holds the run’s data.

Always True for a file that was read back, and for serial runs. Under MPI, finalize(return_results=True) gathers everything on rank 0, so only rank 0’s results are the root ones; the others come back empty and with this False.

Everything that produces output - :meth:print_summary, the plot_* functions, the exporters - does nothing for non-root results. That is what lets a parallel script call them unguarded and still write each figure exactly once, from rank 0. Read it when a script needs to make the same distinction for output of its own.

Returns

bool True unless this is a non-root rank’s share of an MPI run.

labelpropertywritable#

View source

label: str | None

The run’s label, as given to ProfileManager.setup(label=...).

Returns

str or None The label, or None for a run that was not given one. Use :attr:display_label to name the run in output regardless.

likwid_ranksproperty#

View source

likwid_ranks: list[int]

Ranks that recorded LIKWID results, in ascending order.

line_profileproperty#

View source

line_profile: dict[int, list]

Persisted line-profiler records keyed by rank.

Each record contains region, filename, function, first_lineno, line_numbers, hits, times and unit. The elapsed seconds for a line are times * unit.

maximum_end_timeproperty#

View source

maximum_end_time: float

Get the maximum end time across all regions and ranks.

Returns

float Maximum end time in seconds, or 0.0 if no region recorded timing.

metadataproperty#

View source

metadata: dict

Get environment metadata for the run (gathered from rank 0).

Returns

dict Metadata dict (hostname, OpenMP thread count, platform, versions, etc.), or an empty dict if the run recorded none.

minimum_start_timeproperty#

View source

minimum_start_time: float

Get the minimum start time across all regions and ranks.

This is the origin of the timeline: subtract it from any timestamp to get seconds since the first region entry. Regions with no recorded calls are ignored; the result is 0.0 if no region recorded any.

Returns

float Minimum start time in seconds.

num_nodesproperty#

View source

num_nodes: int | None

Number of distinct hosts the run’s ranks ran on, or None if unknown.

Read from the num_nodes metadata field. Files written before it was recorded still answer for a single-rank run, which can only have used one node; a multi-rank run without the field returns None.

num_ranksproperty#

View source

num_ranks: int

Get the number of ranks recorded in the profiling data.

Returns

int Number of ranks.

region_namesproperty#

View source

region_names: list[str]

Names of all regions, in order of appearance.

run_start_timeproperty#

View source

run_start_time: float | None

When the run started, in seconds on the recording clock.

This is the start_time_ns metadata field, written by ProfileManager.setup() — by default the moment setup() was called, or an earlier instant if one was passed to it.

Returns

float or None The registered start time, or None for runs without one (older files, or runs that never called setup()). Use :attr:startup_time for the elapsed time before the first region, which stays defined either way.

startup_timeproperty#

View source

startup_time: float

Seconds between the start of the run and the first profiled region.

Time the instrumentation never saw: imports, reading input, building a mesh. Zero when there is no registered start time (:attr:run_start_time is None), since the run is then only known from its first region onwards.

Returns

float Elapsed time before the first region entry, in seconds.

tasksproperty#

View source

tasks: dict[int, list]

Rank -> the asyncio tasks and greenlets the run followed.

Each entry is a :class:~scope_profiler.concurrency.TaskInfo, whose running_time and awaiting_time split the lane’s life into the part it held a thread and the part it was suspended. Empty unless the run profiled with track_async=True.

threadsproperty#

View source

threads: dict[int, list]

Rank -> the threads that recorded calls, in registration order.

Each entry is a :class:~scope_profiler.concurrency.ThreadInfo: name, OS ids, when the thread started and ended, and the CPU time it burned. Empty unless the run profiled with track_threads=True::

for rank, threads in results.threads.items():
    for thread in threads:
        print(rank, thread.name, thread.cpu_time)

The per-call thread_ids column of every region indexes into this rank’s list, so results["solve"][0].for_thread(2) is the part of solve that ran on results.threads[0][2].

time_originproperty#

View source

time_origin: float

Zero point of the relative timeline, in seconds.

The registered :attr:run_start_time when there is one, and otherwise the first region entry (:attr:minimum_start_time). This is what :meth:events and :meth:call_stack measure from.

Note the plot_* functions instead frame their x axis on the first region entry, so that a long gap between setup() and the first region does not fill a chart with empty space. Pass origin=results.minimum_start_time to :meth:events to reproduce the numbers on a chart’s axis.

Returns

float Origin timestamp on the recording clock.

time_spanproperty#

View source

time_span: float

Wall-clock seconds between the first region entry and the last exit.

Returns

float Duration of the profiled window in seconds, or 0.0 if no region recorded timing.

total_timeproperty#

View source

total_time: float | None

Wall-clock seconds from setup() to finalize().

Unlike :attr:time_span (first region entry to last exit), this covers the whole instrumented program: startup work before the first region, gaps between regions, and any teardown after the last one but before finalize() is called - the number to report as “how long did the run take” alongside the region breakdown.

Returns

float or None :attr:finalize_time minus :attr:run_start_time, or None if either is missing - an older file, or a run that set up profiling (or called finalize) some other way than ProfileManager.setup()/finalize().

call_graphmethod#

View source

def call_graph(rank: int = 0, include=None, exclude=None) -> list[dict]

Return call relationships without timestamps or durations.

New profiles persist call_id and parent_id for every Python event. Legacy profiles fall back to the timestamp-based call stack. The returned nodes contain only call_id, parent_id, name, call_index and depth.

Filtering renests: a call whose parent was excluded is reported under its nearest surviving ancestor, with the depth that implies, exactly as :meth:call_stack does. Leaving the excluded id in parent_id would hand back a graph with edges pointing at nodes that are not in it.

call_id is unique within this rank, not across the file - each rank numbers its own calls.

call_stackmethod#

View source

def call_stack(rank: int = 0, include: list[str] | str | None = None, exclude: list[str] | str | None = None, relative: bool = True, origin: float | None = None) -> list[dict]

Reconstruct the nested call stack for one rank.

Regions record no call graph, so nesting is recovered from timestamp containment - the same reconstruction the flame chart, the .prof export and the speedscope export use.

Parameters

rank : int, optional Rank whose calls to reconstruct (default: 0). include, exclude : list of str or str, optional Regex patterns selecting which regions to include, matched as in :meth:get_regions. relative : bool, optional If True (default), timestamps start at zero; see :meth:events. origin : float, optional Measure from this timestamp instead; see :meth:events.

Returns

List[dict] One entry per call, parents before children, with keys call_id, name, start, end, duration, depth and parent (the enclosing call’s id, or None). See :func:~scope_profiler.call_stack.build_call_stack.

default_titlemethod#

View source

def default_title() -> str

Heading naming this run, for a summary table.

The label leads when the run has one, since that is the name the user chose; the file path follows either way, because when several runs are being compared it is what tells them apart on disk.

Returns

str e.g. "results/run_a.h5 (128 ranks)".

eventsmethod#

View source

def events(include: list[str] | str | None = None, exclude: list[str] | str | None = None, ranks: list[int] | int | None = None, relative: bool = True, origin: float | None = None) -> list[dict]

Return one dict per recorded call, across all regions and ranks.

This is the long-form (“tidy”) view to build custom plots from: one row per call rather than one per region.

Parameters

include, exclude : list of str or str, optional Regex patterns selecting which regions to include, matched as in :meth:get_regions. ranks : list of int or int, optional Restrict to these ranks (default: all). relative : bool, optional If True (default), timestamps are measured from :attr:time_origin — the start time registered by ProfileManager.setup(), or the first region entry for runs without one — so the timeline starts at zero. If False, the raw monotonic-clock timestamps are returned; those are only comparable within a single run. origin : float, optional Measure from this timestamp instead, in seconds on the recording clock. Overrides relative; pass origin=results.minimum_start_time to zero the timeline on the first region entry regardless of what the run registered.

Returns

List[dict] Entries with keys name, rank, call_index, start, end and duration, in seconds, ordered by region (as in :meth:get_regions) then rank then call order. Entries also carry gpu_duration when CUDA-event timing was enabled for that region.

Examples

for event in results.events(include=“solve”): # doctest: +SKIP … print(event[“rank”], event[“start”], event[“duration”])

from_h5methodclassmethod#

View source

def from_h5(file_path: str | Path, verbose: bool = False) -> ProfilingResults

Load a merged profiling file written by :meth:ProfileManager.finalize <scope_profiler.profile_manager.ProfileManager.finalize>::

results = ProfilingResults.from_h5("profiling_data.h5")
results.print_summary()

:func:scope_profiler.read_h5 is the same thing under a shorter name.

Parameters

file_path : str | Path Path to the merged HDF5 file containing profiling data. verbose : bool, optional Print each rank group as it is read (default: False).

Returns

ProfilingResults The run’s profiling data, of whichever class this was called on.

Raises

FileNotFoundError If the specified HDF5 file does not exist.

get_likwid_regionmethod#

View source

def get_likwid_region(tag: str, rank: int = 0) -> LikwidRegionResult

Get one region’s LIKWID results for a single rank.

Parameters

tag : str LIKWID marker region tag (the profiled region’s name). rank : int, optional Rank to read from (default: 0).

Returns

LikwidRegionResult The region’s counters, event names and derived metrics.

Raises

KeyError If the rank recorded no LIKWID data or has no such region.

get_likwid_regionsmethod#

View source

def get_likwid_regions(rank: int | None = None) -> dict

Get the LIKWID marker results of the run.

Parameters

rank : int, optional Return only this rank’s regions. By default every rank is included, keyed by rank.

Returns

dict With rank given, a mapping of region tag to :class:~scope_profiler.likwid_data.LikwidRegionResult; otherwise a mapping of rank to such a dict. Empty when the run did not use LIKWID.

Examples

::

results = read_h5("profiling_data.h5")
for rank, regions in results.get_likwid_regions().items():
    for tag, result in regions.items():
        for name, values in zip(result.metric_names, result.metrics):
            print(rank, tag, name, values)

get_perf_eventsmethod#

View source

def get_perf_events(rank: int | None = None) -> dict

Return aggregated Linux perf_event_open counts by region.

get_regionmethod#

View source

def get_region(region_name: str) -> MPIRegion

Retrieve profiling data for a specific region.

Parameters

region_name : str Name of the region to retrieve.

Returns

Region Region object containing profiling data for all ranks.

Raises

KeyError If the specified region name does not exist.

get_regionsmethod#

View source

def get_regions(include: list[str] | str | None = None, exclude: list[str] | str | None = None) -> list[MPIRegion]

Get a list of all regions in order of appearance.

Returns

List[Region] List of Region objects.

lane_labelmethod#

View source

def lane_label(lane: int, rank: int = 0) -> str

A human-readable name for one lane of rank.

Lanes are the stacks a rank’s calls were reconstructed into (see :func:~scope_profiler.concurrency.lane_ids). This turns one back into something worth putting on a chart or a speedscope profile selector: a task’s own name where the lane is a task, the thread’s name where it is a bare thread.

likwid_to_dataframemethod#

View source

def likwid_to_dataframe()

Return every LIKWID event and metric as a tidy pandas DataFrame.

One row per (rank, region, hardware thread), with a column per event and per derived metric, plus the region’s LIKWID runtime and call count. Regions measured with different event groups simply leave the other group’s columns empty.

Returns

pandas.DataFrame Empty if there is no LIKWID data.

Raises

ImportError If pandas is not installed.

print_likwid_summarymethod#

View source

def print_likwid_summary(stream=None) -> None

Print every LIKWID region’s events and derived metrics.

Parameters

stream : file-like, optional Destination (default: stdout).

print_summarymethod#

View source

def print_summary(include: list[str] | str | None = None, exclude: list[str] | str | None = None, ranks: list[int] | None = None, sort: str = 'start', title: str | None = None, stream=None, suppress_notes: bool = False, columns: list[str] | str | None = None, percentage_mode: str = 'coverage') -> None

Print a region summary table, aggregated over ranks.

Renders the same table as scope-profiler inspect and the summary printed by ProfileManager.finalize().

Parameters

include, exclude : list of str or str, optional Regex patterns selecting which regions to print, matched as in :meth:get_regions. ranks : list of int, optional Restrict the statistics to these ranks (default: all). sort : str, optional Column to order by: start (default), total, calls, avg, min, max, std or name. title : str, optional Heading above the table (default: the file path and rank count). stream : file-like, optional Where to write (default: stdout). columns : list of str or str, optional Region summary columns to print. Defaults to region, calls, percent, parent_percent, total and avg. percent is relative to scope_profiler.session; parent_percent is relative to the enclosing call path. Use region for the region-name column. percentage_mode : {“coverage”, “exclusive”}, optional Quantity used for % session. Defaults to wall-clock coverage; use exclusive to attribute time after nested regions.

Notes

Does nothing on a non-root rank (see :attr:is_root), so a parallel script can call it unguarded and print the table once.

summarymethod#

View source

def summary(include: list[str] | str | None = None, exclude: list[str] | str | None = None) -> list[dict]

Summarize every region, aggregated over ranks.

Parameters

include, exclude : list of str or str, optional Regex patterns selecting which regions to summarize, matched as in :meth:get_regions.

Returns

List[dict] One dict per region (see :meth:~scope_profiler.mpi_region.MPIRegion.get_summary), ordered by first start time. inclusive_duration includes nested regions; exclusive_duration excludes them. Durations are in seconds.

thread_summarymethod#

View source

def thread_summary(rank: int = 0) -> list[dict]

Per-thread wall time, CPU time and call count for one rank.

Wall and CPU time come from the thread table; the call count and the time attributed to regions come from the events, so a thread that burned CPU outside every profiled region shows the gap directly.

Returns

list of dict One entry per thread with keys index, name, alive, wall_time, cpu_time, num_calls and region_time (the summed inclusive duration of that thread’s top-level calls, so nested regions are not counted twice), in thread-table order. Empty when the run did not track threads.

to_dataframemethod#

View source

def to_dataframe(include: list[str] | str | None = None, exclude: list[str] | str | None = None, per_rank: bool = False)

Return the region summaries as a pandas DataFrame.

Parameters

include, exclude : list of str or str, optional Regex patterns selecting which regions to include, matched as in :meth:get_regions. per_rank : bool, optional If True, emit one row per (region, rank) with a rank column instead of one aggregated row per region (default: False).

Returns

pandas.DataFrame Region statistics, with durations in seconds.

Raises

ImportError If pandas is not installed.

to_events_dataframemethod#

View source

def to_events_dataframe(include: list[str] | str | None = None, exclude: list[str] | str | None = None, ranks: list[int] | int | None = None, relative: bool = True, origin: float | None = None)

Return every recorded call as a pandas DataFrame (one row per call).

Parameters

include, exclude, ranks, relative, origin As in :meth:events.

Returns

pandas.DataFrame Columns name, rank, call_index, start, end and duration, with times in seconds. A gpu_duration column is included when any selected event has CUDA-event timing.

Raises

ImportError If pandas is not installed.

load

One entry point for reading a run back, whatever format it is in. The format comes from the file’s contents rather than its name, so a caller that just has a path does not have to know which reader it wanted.

loadfunction#

View source

def load(path, verbose: bool = False) -> ProfilingResults

Load a profile for post-processing, whatever format it is in.

The one entry point for reading a run back::

import scope_profiler

results = scope_profiler.load("profiling_data.h5")
results.print_summary()

HDF5 and JSON profiles are both accepted, and the format is taken from the file’s contents rather than its name (see :func:sniff_profile_format), so the caller does not have to know which of :func:~scope_profiler.h5reader.read_h5 and :func:~scope_profiler.json_export.read_json it wanted. Those remain available for reading a file whose format is already known.

Parameters

path : str | Path The profile to read. verbose : bool, optional Print progress while reading (default: False).

Returns

ProfilingResults The run’s profiling data, with all durations in seconds.

Raises

FileNotFoundError If path does not exist. ValueError If path is an HTML report. A report is a rendering of a run, not a copy of it, so there is nothing to read back.

sniff_profile_formatfunction#

View source

def sniff_profile_format(path) -> str

Which format the contents of path are in.

:func:profile_format goes by the file name, which is all that can be known before a file exists. For a file that does exist the contents are the better authority: a run written as profile.dat, or a JSON profile someone renamed to .h5, still reads correctly.

Falls back to :func:profile_format when the leading bytes match nothing known, so an unrecognisable file is still treated as HDF5 and produces h5py’s own error rather than one invented here.

Raises

FileNotFoundError If path does not exist.

read_h5

The module-level spelling of ProfilingResults.from_h5(); the two are interchangeable.

read_h5function#

View source

def read_h5(file_path: str | Path, verbose: bool = False) -> ProfilingResults

Load a merged profiling file for post-processing.

The discoverable spelling of :meth:ProfilingResults.from_h5 <scope_profiler.results.ProfilingResults.from_h5>; the two are interchangeable::

from scope_profiler import read_h5

results = read_h5("profiling_data.h5")
results.print_summary()

solve = results["solve"]        # same as results.get_region("solve")
solve[0].average_duration       # rank 0, in seconds

Parameters

file_path : str | Path Path to the merged HDF5 file containing profiling data. verbose : bool, optional Print each rank group as it is read (default: False).

Returns

ProfilingResults The run’s profiling data. All durations are reported in seconds.

Raises

FileNotFoundError If the specified HDF5 file does not exist.

read_h5_summary

Load fixed-size scalar statistics without reading per-call event columns.

read_h5_summaryfunction#

View source

def read_h5_summary(file_path: str | Path, verbose: bool = False, *, fallback: bool = True, include_likwid: bool = True, include_line_profile: bool = True, regions: str | list[str] | tuple[str, ...] | None = None, ranks: int | list[int] | tuple[int, ...] | None = None) -> ProfilingResults

Load fixed-size statistics, optionally falling back for older files.

The returned object supports metadata and scalar region statistics. Event, timeline, call-stack and percentile APIs intentionally have no per-call data; use :func:read_h5 when those are required.

read_json and write_json

The same run as a JSON document: write_json() writes one, read_json() reads it back into the very same ProfilingResults, and read_profile() / write_profile() pick between JSON and HDF5 by the file name — the rule scope-profiler run -o follows. See CLI reference for when to prefer which.

read_jsonfunction#

View source

def read_json(file_path: str | Path, verbose: bool = False) -> ProfilingResults

Read a JSON profile back into :class:ProfilingResults.

The JSON twin of :func:~scope_profiler.h5reader.read_h5::

results = read_json("profiling_data.json")
results.print_summary()

write_jsonfunction#

View source

def write_json(results: ProfilingResults, filepath: str | Path, *, include=None, exclude=None, ranks: list[int] | int | None = None, indent: int | None = None) -> Path

Write one run to filepath as JSON, and return the path written.

export_jsonfunction#

View source

def export_json(profiling_data: ProfilingResults | Sequence[ProfilingResults], filepath: str | Path, ranks: list[int] | int | None = None, include: list[str] | str | None = None, exclude: list[str] | str | None = None, verbose: bool = True, indent: int | None = None) -> list[Path]

Write one JSON document per run, exactly as export_speedscope does.

Parameters

profiling_data : ProfilingResults | Sequence[ProfilingResults] The run(s) to export: file runs, in-memory results, or a mix. filepath : str | Path Base output path, e.g. figures/profile.json. Ending it in .gz compresses the output. The run’s label is appended to the stem when more than one run is exported. ranks : list[int] | int, optional Ranks to export (default: every rank). include, exclude : list[str] | str, optional Region name filters, as for the plotting functions. indent : int, optional Passed to json.dumps; the default writes the document on one line.

Returns

list[Path] The files written, in the order they were written.

read_profilefunction#

View source

def read_profile(file_path, verbose: bool = False) -> ProfilingResults

Read a profile from HDF5 or JSON, whichever the name says it is.

Dispatches on the file name; :func:load dispatches on the contents and is the better default for a file that already exists.

write_profilefunction#

View source

def write_profile(results, file_path, **kwargs) -> Path

Write results to file_path in the format its name asks for.

Parameters

results : ProfilingResults The run to write. file_path : str | Path Destination. .json/.json.gz writes a JSON profile, .html an HTML report, and anything else an HDF5 file. **kwargs Passed to the format’s own writer.

Returns

Path The file written.

Native (C and Fortran) traces

Reading what the C and Fortran region APIs write - they share one format; see Profiling C and C++ code and Profiling Fortran code.

load_tracesfunction#

View source

def load_traces(inputs, label: str | None = None)

Read native output into the standard post-processing API.

Both formats are accepted, and mix: .spt traces from a Fortran or plain C build, and the per-rank .h5 profiles an SP_USE_HDF5 C build writes. Every rank of a run lands in one result either way.

Parameters

inputs : path or sequence of paths Files and/or directories containing them (see :func:find_traces). label : str, optional Name for the run in summaries, charts and exports.

Returns

ProfilingResults The same object a Python run produces, so every summary, plot and exporter works on it unchanged.

Raises

TraceFormatError If two files claim the same rank.

convert_tracesfunction#

View source

def convert_traces(inputs, output_path, label: str | None = None)

Convert Fortran traces into a standard scope-profiler HDF5 file.

The result is indistinguishable from one a Python run wrote, so scope-profiler plot / inspect and :func:~scope_profiler.read_h5 work on it directly.

Parameters

inputs : path or sequence of paths Trace files and/or directories containing them. output_path : str or Path HDF5 file to write. label : str, optional Name for the run; defaults to the output file’s stem.

Returns

Path The file that was written.

read_tracefunction#

View source

def read_trace(path) -> tuple

Read one rank’s trace file.

Parameters

path : str or Path A .spt file written by sp_finalize().

Returns

tuple (rank, regions), where regions maps a region name to (start_times, end_times) int64 arrays in nanoseconds - exactly the shape :class:~scope_profiler.profile_manager.RankPayload carries. Each value also carries .source_file / .source_lineno attributes (both None on a version-1 trace, or a region registered without sp_region_at()), without changing how it unpacks.

Raises

TraceFormatError If the file is not a trace, is truncated, or was written by a newer format version.

find_tracesfunction#

View source

def find_traces(inputs) -> list

Collect native output files from paths, directories, or a mix of both.

Parameters

inputs : path or sequence of paths Files to read, and/or directories to search (non-recursively) for *.spt traces and *_rank<NNNNN>.h5 per-rank profiles.

Returns

list of Path The files, sorted, with duplicates removed.

Raises

FileNotFoundError If an input does not exist, or a directory holds neither kind of file.

write_resultsfunction#

View source

def write_results(results, output_path)

Write any :class:ProfilingResults out as a standard HDF5 file.

Goes through :class:~scope_profiler.h5writer.ProfilingWriter, so the result has exactly the layout a Python run produces - which is what lets an imported (or merged) run be read back by :func:~scope_profiler.read_h5 and fed to every plot and exporter.

Parameters

results : ProfilingResults The run to write. output_path : str or Path HDF5 file to create.

Returns

Path The file that was written.

fortran_source_pathfunction#

View source

def fortran_source_path() -> Path

Path to scope_profiler.f90, the module to compile into your program.

It ships with the package, so this works from an installed wheel::

gfortran -c $(python -c             "import scope_profiler.native_trace as t; print(t.fortran_source_path())")

Returns

Path The Fortran module source.

c_source_pathfunction#

View source

def c_source_path() -> Path

Path to scope_profiler.c, the implementation entry point to compile.

Its public and private implementation headers sit next to it; :func:c_include_dir is what to put on the compiler’s include path::

cc -c $(python -c             "import scope_profiler.native_trace as t; print(t.c_source_path())")            -I$(python -c             "import scope_profiler.native_trace as t; print(t.c_include_dir())")

Returns

Path The C source file.

c_include_dirfunction#

View source

def c_include_dir() -> Path

Directory holding scope_profiler.h, for the compiler’s -I.

Returns

Path The include directory.

Combining runs

merge_resultsfunction#

View source

def merge_results(*result_sets, label: str | None = None, file_path=None)

Combine several result sets into one.

The case this exists for is a mixed-language run: a Python driver records its own regions while the Fortran (or other) code it calls records theirs, and what the user wants at the end is one profile covering both. Ranks line up by number, so rank 3’s Python regions and rank 3’s Fortran regions end up side by side.

Parameters

*result_sets : ProfilingResults The sets to combine. Non-root sets (the empty ones every rank but 0 gets back under MPI) are ignored, so this can be called unguarded in a parallel script. label : str, optional Name for the combined run. Defaults to the first set’s label. file_path : str or Path, optional Output path to attribute the combined set to. Defaults to the first set’s.

Returns

ProfilingResults One result set holding every region of every input.

Raises

ValueError If no result sets were given, or if a region name appears in more than one of them. Merging same-named regions would silently double-count a Python wrapper and the native region inside it, so the collision has to be resolved by the caller - name the regions apart, for instance with a "fortran:" prefix.

Region

Regionclass#

View source

class Region(start_times: np.ndarray, end_times: np.ndarray, gpu_durations: np.ndarray | None = None, call_ids: np.ndarray | None = None, parent_ids: np.ndarray | None = None, thread_ids: np.ndarray | None = None, task_ids: np.ndarray | None = None, await_times: np.ndarray | None = None, event_metadata: list[dict] | tuple[dict, ...] | None = None, source_file: str | None = None, source_lineno: int | None = None, source_text: str | None = None, tags=(), aggregate: dict | None = None, event_data_available: bool = True)

Timing data for one region on one rank.

All duration and timestamp properties are reported in seconds; the underlying HDF5 data is stored in nanoseconds.

average_durationproperty#

View source

average_duration: float

Average duration per call in seconds.

await_timesproperty#

View source

await_times: np.ndarray | None

Seconds each call spent with its task suspended, or None.

The wall time inside the with block during which the task was not running: awaiting, or switched away from. Subtracting it from the call’s duration leaves the time the lane actually held its thread.

await_times_nsproperty#

View source

await_times_ns: np.ndarray | None

Nanoseconds each call spent with its task suspended, or None.

call_idsproperty#

View source

call_ids

Explicit call ids, or None for legacy profiles.

durationsproperty#

View source

durations: np.ndarray

Duration of all calls in seconds.

durations_nsproperty#

View source

durations_ns: np.ndarray

Duration of all calls in nanoseconds, as integers.

end_timesproperty#

View source

end_times: np.ndarray

End times of all calls in seconds.

end_times_nsproperty#

View source

end_times_ns: np.ndarray

End times of all calls in nanoseconds, exactly as stored.

event_metadataproperty#

View source

event_metadata: tuple[dict, ...]

Per-call scoped metadata, aligned with the event arrays.

exclusive_durationproperty#

View source

exclusive_duration: float

Alias for :attr:total_exclusive_duration.

exclusive_durationsproperty#

View source

exclusive_durations: np.ndarray

Exclusive duration of every call in seconds.

exclusive_durations_nsproperty#

View source

exclusive_durations_ns: np.ndarray

Exclusive duration of every call in nanoseconds.

first_durationproperty#

View source

first_duration: float

Duration of the first recorded call, in seconds.

first_start_timeproperty#

View source

first_start_time: float

First start time in seconds.

gpu_average_durationproperty#

View source

gpu_average_duration: float | None

Average CUDA-event elapsed device time in seconds, or None if absent.

gpu_durationsproperty#

View source

gpu_durations: np.ndarray | None

CUDA-event elapsed device times in seconds, or None if absent.

gpu_durations_nsproperty#

View source

gpu_durations_ns: np.ndarray | None

CUDA-event elapsed device times in nanoseconds, or None if absent.

gpu_total_durationproperty#

View source

gpu_total_duration: float | None

Total CUDA-event elapsed device time in seconds, or None if absent.

has_event_dataproperty#

View source

has_event_data: bool

Whether per-call timestamps were loaded for this region.

has_gpu_timingproperty#

View source

has_gpu_timing: bool

Whether this region has CUDA-event elapsed timings.

has_sourceproperty#

View source

has_source: bool

Whether this region’s call-site source was captured.

has_thread_dataproperty#

View source

has_thread_data: bool

Whether this region recorded the thread each call ran on.

has_timingproperty#

View source

has_timing: bool

Whether this region recorded any calls at all.

inclusive_durationproperty#

View source

inclusive_duration: float

Total inclusive time, including nested regions, in seconds.

inclusive_durationsproperty#

View source

inclusive_durations: np.ndarray

Inclusive duration of every call in seconds.

inclusive_durations_nsproperty#

View source

inclusive_durations_ns: np.ndarray

Inclusive duration of every call in nanoseconds.

last_durationproperty#

View source

last_duration: float

Duration of the last recorded call, in seconds.

last_end_timeproperty#

View source

last_end_time: float

Last end time in seconds.

max_durationproperty#

View source

max_duration: float

Maximum duration among all calls in seconds.

min_durationproperty#

View source

min_duration: float

Minimum duration among all calls in seconds.

num_callsproperty#

View source

num_calls: int

Number of recorded calls.

p50_durationproperty#

View source

p50_duration: float | None

Median duration in seconds.

p95_durationproperty#

View source

p95_duration: float | None

95th-percentile duration in seconds.

p99_durationproperty#

View source

p99_duration: float | None

99th-percentile duration in seconds.

parent_idsproperty#

View source

parent_ids

Explicit parent ids, or None for legacy profiles.

source_fileproperty#

View source

source_file: str | None

Path of the file this region is defined in, or None if not captured.

source_linenoproperty#

View source

source_lineno: int | None

Line the region’s call site starts at, or None if not captured.

source_textproperty#

View source

source_text: str | None

Source text of the region’s with block or decorated function.

None if it was not captured - either the file it came from is no longer readable, or the file predates this being recorded.

start_timesproperty#

View source

start_times: np.ndarray

Start times of all calls in seconds.

start_times_nsproperty#

View source

start_times_ns: np.ndarray

Start times of all calls in nanoseconds, exactly as stored.

std_durationproperty#

View source

std_duration: float

Standard deviation of durations in seconds.

stored_summaryproperty#

View source

stored_summary: dict | None

Fixed-size statistics used by aggregate and summary-only results.

tagsproperty#

View source

tags: tuple[str, ...]

User-defined tags attached to this region.

task_idsproperty#

View source

task_ids: np.ndarray | None

Task-table index of every call, -1 outside a task.

None unless the run used track_async.

thread_idsproperty#

View source

thread_ids: np.ndarray | None

Thread-table index of every call, or None if threads were not tracked.

threadsproperty#

View source

threads: list[int]

Sorted thread-table indices this region recorded calls on.

total_await_durationproperty#

View source

total_await_duration: float | None

Total awaited seconds over every call, or None if not tracked.

total_durationproperty#

View source

total_duration: float

Total time spent in this region in seconds (sum of all durations).

total_exclusive_durationproperty#

View source

total_exclusive_duration: float

Total time excluding nested regions, in seconds.

eventsmethod#

View source

def events(origin: float = 0.0) -> list[dict[str, Any]]

Return one dict per recorded call.

Parameters

origin : float, optional Seconds subtracted from every timestamp, so passing results.minimum_start_time yields a timeline starting at zero (default: 0.0, i.e. raw timestamps).

Returns

list of dict One entry per call with keys call_index, start, end and duration, in seconds and in recorded order.

for_threadmethod#

View source

def for_thread(thread_index: int) -> Region

A view of this region holding only the calls of one thread.

Every column is sliced with the same mask, so the returned region is an ordinary :class:Region - summaries, durations and percentiles all describe that thread alone.

Raises

ValueError If this region has no thread column (the run did not use track_threads).

get_summarymethod#

View source

def get_summary() -> dict[str, Any]

Return a summary of the region’s statistics as a dictionary.

Returns

Dict[str, Any] Dictionary containing statistics: num_calls, total_duration, average_duration, min_duration, max_duration, first_duration, last_duration, and std_duration. Durations are in seconds.

percentile_durationmethod#

View source

def percentile_duration(percentile: float) -> float | None

Return a duration percentile in seconds.

percentile follows :func:numpy.percentile and must be between 0 and 100. Empty regions return 0.0 for consistency with the other duration statistics.

MPIRegion

MPIRegionclass#

View source

class MPIRegion(name: str, regions: dict[int, Region])

One named region across all ranks that recorded it.

Indexing gives the per-rank :class:~scope_profiler.region.Region (region[0]), while the properties on this class aggregate over every rank. All durations are in seconds.

colorattributeinstance attribute#

View source

color = None

average_durationproperty#

View source

average_duration: float

Get the mean duration over every call on every rank.

Note this pools all calls, so ranks with more calls weigh more heavily; use average_durations() for the per-rank breakdown.

Returns

float Average duration in seconds, or 0.0 if no timing was recorded.

durationsproperty#

View source

durations: np.ndarray

Get every recorded call duration on every rank, in seconds.

Returns

np.ndarray Pooled durations. Empty if no rank recorded timing.

exclusive_durationproperty#

View source

exclusive_duration: float

Alias for :attr:total_exclusive_duration.

exclusive_durationsproperty#

View source

exclusive_durations: np.ndarray

Exclusive durations pooled across ranks, in seconds.

first_durationproperty#

View source

first_duration: float

Get the duration of the call that started earliest across all ranks.

Returns

float Duration in seconds of the chronologically first call, or 0.0 if no rank recorded timing.

first_start_timeproperty#

View source

first_start_time: float

Get the earliest start time across all ranks.

Returns

float The earliest start time among all ranks, in seconds, or 0.0 if no rank recorded timing.

gpu_average_durationproperty#

View source

gpu_average_duration: float | None

Average CUDA-event elapsed device time across ranks, or None if absent.

gpu_durationsproperty#

View source

gpu_durations: np.ndarray | None

CUDA-event elapsed device times pooled across ranks, or None if absent.

gpu_total_durationproperty#

View source

gpu_total_duration: float | None

Total CUDA-event elapsed device time across ranks, or None if absent.

has_event_dataproperty#

View source

has_event_data: bool

Whether every represented rank has per-call timestamps loaded.

has_gpu_timingproperty#

View source

has_gpu_timing: bool

Whether any rank recorded CUDA-event timings for this region.

has_sourceproperty#

View source

has_source: bool

Whether any rank captured this region’s call-site source.

has_timingproperty#

View source

has_timing: bool

Whether any rank recorded timestamps for this region.

inclusive_durationproperty#

View source

inclusive_duration: float

Total inclusive time across ranks, in seconds.

inclusive_durationsproperty#

View source

inclusive_durations: np.ndarray

Inclusive durations pooled across ranks, in seconds.

last_durationproperty#

View source

last_duration: float

Get the duration of the call that ended latest across all ranks.

Returns

float Duration in seconds of the chronologically last call, or 0.0 if no rank recorded timing.

last_end_timeproperty#

View source

last_end_time: float

Get the latest end time across all ranks.

Returns

float The latest end time among all ranks, in seconds, or 0.0 if no rank recorded timing.

max_durationproperty#

View source

max_duration: float

Get the maximum duration across all ranks.

Returns

float The maximum duration among all ranks, in seconds.

min_durationproperty#

View source

min_duration: float

Get the minimum duration across all ranks.

Returns

float The minimum duration among all ranks, in seconds.

nameproperty#

View source

name: str

Name of the region.

num_callsproperty#

View source

num_calls: int

Total number of calls summed over all ranks.

Returns

int Summed call count.

p50_durationproperty#

View source

p50_duration: float | None

Median duration across all ranks, in seconds.

p95_durationproperty#

View source

p95_duration: float | None

95th-percentile duration across all ranks, in seconds.

p99_durationproperty#

View source

p99_duration: float | None

99th-percentile duration across all ranks, in seconds.

rank_imbalanceproperty#

View source

rank_imbalance: float

Maximum per-rank total divided by the mean per-rank total.

A value of 1.0 means perfectly balanced ranks. Values are 0.0 when no calls were recorded or only one rank has timing data.

rank_imbalance_pctproperty#

View source

rank_imbalance_pct: float

Excess of the slowest rank over the mean, as a percentage.

ranksproperty#

View source

ranks: list[int]

Sorted list of ranks that recorded this region.

regionsproperty#

View source

regions: dict[int, Region]

Dictionary of rank IDs to their corresponding Region objects.

source_fileproperty#

View source

source_file: str | None

Path of the file this region is defined in, or None if not captured.

source_linenoproperty#

View source

source_lineno: int | None

Line the region’s call site starts at, or None if not captured.

source_textproperty#

View source

source_text: str | None

Source text of the region’s with block or decorated function.

std_durationproperty#

View source

std_duration: float

Get the standard deviation over every call on every rank.

Returns

float Standard deviation in seconds, or 0.0 if no timing was recorded.

tagsproperty#

View source

tags: tuple[str, ...]

User-defined tags attached to this region.

total_durationproperty#

View source

total_duration: float

Get the total duration summed over all ranks and calls.

Returns

float Total duration in seconds.

total_exclusive_durationproperty#

View source

total_exclusive_duration: float

Total time excluding nested regions, across ranks, in seconds.

average_durationsmethod#

View source

def average_durations() -> dict[int, float]

Get the average duration for each rank.

Returns

Dict[int, float] Dictionary mapping rank IDs to their average durations.

eventsmethod#

View source

def events(ranks: list[int] | int | None = None, origin: float = 0.0) -> list[dict[str, Any]]

Return one dict per recorded call, on every rank.

This is the long-form view custom plots and dataframes want: each entry is a single call rather than a per-region aggregate.

Parameters

ranks : list of int or int, optional Restrict to these ranks (default: all ranks that recorded the region). Ranks without data for this region are skipped. origin : float, optional Seconds subtracted from every timestamp, so passing results.minimum_start_time yields a timeline starting at zero (default: 0.0, i.e. raw timestamps).

Returns

list of dict Entries with keys name, rank, call_index, start, end and duration, in seconds, ordered by rank and then by call order.

get_summarymethod#

View source

def get_summary() -> dict[str, Any]

Return statistics aggregated over every rank.

Returns

Dict[str, Any] Dictionary with name, num_ranks, num_calls (summed over ranks) and duration statistics in seconds, pooled over all calls on all ranks.

max_durationsmethod#

View source

def max_durations() -> dict[int, float]

Get the maximum duration for each rank.

Returns

Dict[int, float] Dictionary mapping rank IDs to their maximum durations.

min_durationsmethod#

View source

def min_durations() -> dict[int, float]

Get the minimum duration for each rank.

Returns

Dict[int, float] Dictionary mapping rank IDs to their minimum durations.

num_calls_per_rankmethod#

View source

def num_calls_per_rank() -> dict[int, int]

Get the call count for each rank.

Returns

Dict[int, int] Dictionary mapping rank IDs to their call counts.

percentile_durationmethod#

View source

def percentile_duration(percentile: float) -> float | None

Return a pooled duration percentile across all ranks.

total_durationsmethod#

View source

def total_durations() -> dict[int, float]

Get the total duration for each rank.

Returns

Dict[int, float] Dictionary mapping rank IDs to their total durations.

MPI call wrappers

Opt-in profiling wrappers for mpi4py communicators and requests.

Importing this module never imports mpi4py. Pass an existing communicator to profile_mpi_comm, or explicitly call profile_mpi4py to patch predefined communicators for the duration of a run. Importing scope-profiler therefore cannot initialize MPI in a serial process.

Classes

Name	Description
MPIRegionMetadata	Structured metadata decoded from a Python or native MPI region name.
ProfiledMPIComm	A transparent communicator proxy for common mpi4py operations.
ProfiledMPIRequest	A request proxy that attributes blocking completion time to MPI wait.

Functions

Name	Description
format_mpi_region	Return the canonical region name shared by Python and C++ wrappers.
message_nbytes	Estimate payload bytes for an mpi4py object or buffer specification.
parse_mpi_region	Decode a canonical MPI region name, or return None for another region.
profile_mpi4py	Temporarily profile mpi4py's predefined communicators.
profile_mpi_comm	Wrap a communicator without importing mpi4py.

MPIRegionMetadataclassdataclass#

View source

class MPIRegionMetadata(operation: str, kind: str, bytes: int, peer: int, root: int, tag: int, communicator: int, request: str | None = None)

Structured metadata decoded from a Python or native MPI region name.

bytesattributeinstance attribute#

View source

bytes: int

communicatorattributeinstance attribute#

View source

communicator: int

kindattributeinstance attribute#

View source

kind: str

operationattributeinstance attribute#

View source

operation: str

peerattributeinstance attribute#

View source

peer: int

requestattributeclass attributeinstance attribute#

View source

request: str | None = None

rootattributeinstance attribute#

View source

root: int

tagattributeinstance attribute#

View source

tag: int

ProfiledMPICommclass#

View source

class ProfiledMPIComm(communicator: Any, *, pickle_object_sizes: bool = False)

A transparent communicator proxy for common mpi4py operations.

communicatorproperty#

View source

communicator: Any

The unwrapped mpi4py communicator.

Allreducemethod#

View source

def Allreduce(sendbuf, recvbuf, op=None)

Barriermethod#

View source

def Barrier()

Bcastmethod#

View source

def Bcast(buf, root: int = 0)

Irecvmethod#

View source

def Irecv(buf, source: int = -1, tag: int = -1) -> ProfiledMPIRequest

Isendmethod#

View source

def Isend(buf, dest: int, tag: int = 0) -> ProfiledMPIRequest

Recvmethod#

View source

def Recv(buf, source: int = -1, tag: int = -1, status=None)

Reducemethod#

View source

def Reduce(sendbuf, recvbuf, op=None, root: int = 0)

Sendmethod#

View source

def Send(buf, dest: int, tag: int = 0)

allreducemethod#

View source

def allreduce(sendobj, op=None)

barriermethod#

View source

def barrier()

bcastmethod#

View source

def bcast(obj, root: int = 0)

irecvmethod#

View source

def irecv(buf=None, source: int = -1, tag: int = -1) -> ProfiledMPIRequest

isendmethod#

View source

def isend(obj, dest: int, tag: int = 0) -> ProfiledMPIRequest

recvmethod#

View source

def recv(buf=None, source: int = -1, tag: int = -1, status=None)

reducemethod#

View source

def reduce(sendobj, op=None, root: int = 0)

sendmethod#

View source

def send(obj, dest: int, tag: int = 0)

ProfiledMPIRequestclass#

View source

class ProfiledMPIRequest(request: Any, *, operation: str, bytes: int, peer: int, tag: int, communicator: int)

A request proxy that attributes blocking completion time to MPI wait.

Waitmethod#

View source

def Wait(status=None)

waitmethod#

View source

def wait(status=None)

format_mpi_regionfunction#

View source

def format_mpi_region(operation: str, *, bytes: int = -1, peer: int = -1, root: int = -1, tag: int = -1, communicator: int = -1, request: str | None = None) -> str

Return the canonical region name shared by Python and C++ wrappers.

message_nbytesfunction#

View source

def message_nbytes(value: Any, *, pickle_objects: bool = True) -> int

Estimate payload bytes for an mpi4py object or buffer specification.

Buffer-mode calls are exact when their datatype exposes Get_size. Object-mode values use their buffer size when possible. With pickle_objects=True (the default for this explicit helper), other objects use the size of their protocol-5 pickle. -1 means unknown.

parse_mpi_regionfunction#

View source

def parse_mpi_region(name: str) -> MPIRegionMetadata | None

Decode a canonical MPI region name, or return None for another region.

profile_mpi4pyfunction#

View source

def profile_mpi4py(mpi_module: Any = None, *, pickle_object_sizes: bool = False, lazy: bool = False) -> Iterator[None]

Temporarily profile mpi4py’s predefined communicators.

When lazy is true, mpi4py is wrapped only if the target imports it. This lets scope-profiler run --mpi-calls enable MPI-call profiling without importing or initializing MPI for an ordinary serial program. Both globals and Python’s import function are restored even when the program raises.

profile_mpi_commfunction#

View source

def profile_mpi_comm(communicator: Any, *, pickle_object_sizes: bool = False) -> ProfiledMPIComm

Wrap a communicator without importing mpi4py.

Set pickle_object_sizes=True to measure serialized object-mode payloads at the cost of one extra serialization per call. It is off by default to avoid overhead and repeated __reduce__ side effects.

Call stack reconstruction

Regions record no call graph, so nesting is recovered from timestamp containment. ProfilingResults.call_stack() is the usual entry point; the functions below operate on its result and let you walk the reconstructed tree when building your own nested visualisation.

build_call_stackfunction#

View source

def build_call_stack(regions: Iterable, rank: int, origin: float = 0.0) -> list[dict]

Reconstruct per-call nesting for one rank, one dict per call.

A convenience wrapper over :func:build_call_arrays for callers that want to iterate calls rather than columns. It allocates a dict per call (~700 bytes), so prefer the arrays for anything that has to scale with a long run.

Parameters

regions : iterable of MPIRegion Regions to include, e.g. results.get_regions(). rank : int Rank whose calls to reconstruct. Regions with no data for this rank are skipped. origin : float, optional Seconds subtracted from every timestamp, so passing results.minimum_start_time yields a timeline starting at zero (default: 0.0, i.e. raw timestamps).

Returns

list of dict One dict per call, ordered by start time (parents before children), with keys call_id, name, call_path, lane, start, end, duration (the inclusive duration), inclusive_duration, exclusive_duration (seconds), depth (0 for a top-level call) and parent (the call_id of the enclosing call in this same list, or None). call_path joins a call and each of its ancestors with " > ", keeping same-named regions at different call sites distinct. A color key carries whatever the plotting code assigned to the region. call_id is stable for the returned list and is unique within this rank/stack reconstruction.

Raises

NestingError If the intervals are not properly nested.

Notes

Calls are indexed by position in the returned list rather than by name, because a region called more than once - or recursively - contributes several calls under one name.

call_stack_rootsfunction#

View source

def call_stack_roots(calls: list[dict]) -> list[int]

Indices of the top-level calls in a :func:build_call_stack result.

call_stack_childrenfunction#

View source

def call_stack_children(calls: list[dict]) -> list[list[int]]

Child indices per call, for walking a :func:build_call_stack result.

Returns a list parallel to calls: entry i holds the indices of the calls directly nested inside call i, in start-time order.

Intervals must be properly nested, within a lane

Two calls recorded on one lane — one thread, or one asyncio task or greenlet running on it — either nest completely or do not overlap at all. A call that starts inside another on the same lane and ends after it raises NestingError rather than being reconstructed under a guessed parent — there is no call stack that describes it, and every consumer (flame chart, .prof, speedscope, exclusive time) would otherwise have to invent one.

Calls on different lanes overlap freely and are reconstructed as separate stacks. A run profiled with track_threads records a lane per call, which is what makes that possible; without it a whole rank is one lane, so profiling concurrent threads raises here instead. See Threads, tasks and coroutines.

A stack of with blocks or decorated functions on one lane always satisfies this. The other realistic way to violate it is a mismatched sp_begin/sp_end pair in C or Fortran; see the C and Fortran guides.

finalize() is deliberately more forgiving than the readers: a rank whose intervals do not nest keeps its timings and loses only the call graph, with a warning, rather than throwing away a run that has already finished computing.

NestingErrorclass#

View source

class NestingError(ValueError)

Bases: ValueError

Raised when recorded intervals are not properly nested.

A lane’s calls must form a forest: any two intervals on one lane are either disjoint or one contains the other. Partial overlap has no call stack to reconstruct, and every consumer of this module (flame chart, .prof, speedscope, exclusive time) would have to invent one.

In practice this means a mismatched sp_begin/sp_end pair in C or Fortran, or regions driven from several threads in a run that did not profile with track_threads=True - without that, every thread’s calls land in one lane and interleave.

Reconstruction at scale

build_call_stack allocates a dict per call (~700 bytes), which is fine for inspecting a run and not fine for a long one. build_call_arrays returns the same reconstruction as numpy columns and is what the exporters and finalize() use — a run with ten million events per rank reconstructs in a couple of seconds instead of a couple of minutes.

build_call_arraysfunction#

View source

def build_call_arrays(regions: Iterable, rank: int) -> CallArrays

Reconstruct one rank’s nesting as numpy arrays.

The workhorse behind :func:build_call_stack, and what anything processing a whole run should call: it never builds a dict per call, so it stays usable on the tens of millions of events a long simulation produces.

Parameters

regions : iterable of MPIRegion Regions to include, e.g. results.get_regions(). Regions with no data for rank are skipped. rank : int Rank whose calls to reconstruct.

Returns

CallArrays

Raises

NestingError If any interval ends before it starts, or two intervals overlap without one containing the other.

CallArraysclass#

View source

class CallArrays(NamedTuple)

Bases: NamedTuple

One rank’s reconstructed nesting, in start-sorted (call_id) order.

Every array is indexed by call_id, so parent indexes straight back into the same arrays. region_index and call_index point at where a call came from: region names[region_index[i]], call number call_index[i] within that region’s buffers.

call_indexattributeinstance attribute#

View source

call_index: np.ndarray

depthattributeinstance attribute#

View source

depth: np.ndarray

end_nsattributeinstance attribute#

View source

end_ns: np.ndarray

exclusive_nsattributeinstance attribute#

View source

exclusive_ns: np.ndarray

laneattributeclass attributeinstance attribute#

View source

lane: np.ndarray = _NO_LANE

Which stack each call belongs to; see :func:scope_profiler.concurrency.lane_ids.

Empty for a run that did not track threads, meaning the whole rank is one stack - rather than a full-length array of -1, which would cost three more passes over the events of every single-threaded run to say nothing. Consumers must therefore check lane.size before indexing it. Defaulted so the field can be omitted by older callers constructing this tuple themselves.

namesattributeinstance attribute#

View source

names: list[str]

parentattributeinstance attribute#

View source

parent: np.ndarray

region_indexattributeinstance attribute#

View source

region_index: np.ndarray

source_filesattributeinstance attribute#

View source

source_files: list[str | None]

source_linesattributeinstance attribute#

View source

source_lines: list[int | None]

start_nsattributeinstance attribute#

View source

start_ns: np.ndarray

Plotting

plot_ganttfunction#

Re-exported from scope_profiler.plotting_scripts.ganttView source

def plot_gantt(profiling_data: ProfilingResults | Sequence[ProfilingResults], ranks: list[int] | int | None = None, include: list[str] | str | None = None, exclude: list[str] | str | None = None, filepath: str | None = None, show: bool = False, verbose: bool = True, cmap: str = DEFAULT_CMAP, data_filepath: str | Path | None = None, data_format: str = 'csv', backend: str = 'matplotlib', return_fig: bool = False, min_duration: float = 0.0, start_time: float | None = None, end_time: float | None = None, aggregate_calls: int = 1, collapse_depth: int | None = None) -> object | None

Plot a Gantt chart of all (or selected) regions with per-rank lanes using maxplotlib.

Parameters

backend : str Backend to use for rendering: “matplotlib” (default) or “plotly”. return_fig : bool Return the rendered figure instead of the default None. Matplotlib returns (fig, axes); Plotly returns its figure object.

plot_flame_chartfunction#

Re-exported from scope_profiler.plotting_scripts.flameView source

def plot_flame_chart(profiling_data: ProfilingResults | Sequence[ProfilingResults], ranks: list[int] | int | None = None, include: list[str] | str | None = None, exclude: list[str] | str | None = None, filepath: str | None = None, show: bool = False, verbose: bool = True, cmap: str = FLAME_CMAP, data_filepath: str | Path | None = None, data_format: str = 'csv', backend: str = 'matplotlib', return_fig: bool = False) -> object | None

Plot a flame chart reconstructing the call stack from region timings.

Parameters

backend : str Backend to use for rendering: “matplotlib” (default) or “plotly”. return_fig : bool Return the rendered figure instead of the default None. Matplotlib returns (fig, axes); Plotly returns its figure object.

plot_flame_graphfunction#

Re-exported from scope_profiler.plotting_scripts.flameView source

def plot_flame_graph(profiling_data: ProfilingResults | Sequence[ProfilingResults], ranks: list[int] | int | None = None, include: list[str] | str | None = None, exclude: list[str] | str | None = None, filepath: str | None = None, show: bool = False, verbose: bool = True, cmap: str = FLAME_CMAP, data_filepath: str | Path | None = None, data_format: str = 'csv', backend: str = 'matplotlib', return_fig: bool = False) -> object | None

Plot an aggregated flame graph whose x-axis represents total time.

plot_durationsfunction#

Re-exported from scope_profiler.plotting_scripts.durationsView source

def plot_durations(profiling_data: ProfilingResults | Sequence[ProfilingResults], ranks: list[int] | int | None = None, include: list[str] | str | None = None, exclude: list[str] | str | None = None, labels: Sequence[str] | None = None, metric: str = 'total', metrics: Sequence[str] | None = None, sort_by: str | None = None, top_n: int | None = None, combine_regions: dict[str, list[str] | str] | None = None, stack_children: bool = False, filepath: str | None = None, show: bool = False, verbose: bool = True, cmap: str = DEFAULT_CMAP, log_scale: bool = False, data_filepath: str | Path | None = None, data_format: str = 'csv', backend: str = 'matplotlib', return_fig: bool = False) -> list[str] | list[object] | object

Plot duration bar charts for one or more profiling files using maxplotlib.

Parameters

combine_regions : dict[str, list[str] | str], optional Merge several regions into a single bar, e.g. {"setup": ["setup: .*"]} combines every setup: ... region into one bar named “setup”, pooling their calls the same way sort_by and the other duration statistics pool a single region’s calls. Each value is one or more regex patterns (matched like include); a region matching several groups is claimed by whichever group is listed first. metric : str Duration metric to render (avg, min, max, total, first or last). first/last are the durations of the chronologically first and last call, which show how much of a region’s cost is one-off warmup rather than steady state. metrics : Sequence[str], optional Render several metrics instead of one. Each gets its own figure — filepath is suffixed with the metric name, as it always was for multiple metrics - but a single data_filepath holds them all, with the metric named per row, so one export can back a chart whose metric the viewer switches. Overrides metric when given. stack_children : bool Split each bar into the region’s own (exclusive) time plus one segment per region called directly from it, stacked on top of each other, so a bar shows where its time went instead of only how much there was. The nesting comes from timestamp containment, the same reconstruction the flame chart uses, over every region in the run — a child that is filtered out of the bars still gets its own segment. Only total and avg decompose this way; min/max are rejected. Colors then identify segments rather than runs, so with several runs the bars are still grouped per region in run order but the legend names the segments. backend : str Backend to use for rendering: “matplotlib” (default) or “plotly”. return_fig : bool Return the rendered figure instead of the saved filepath list.

Returns

list[str] List of filepaths that were written (empty if filepath is None). With return_fig, the rendered figure, or the list of figures when several metrics were requested.

plot_duration_timeseriesfunction#

Re-exported from scope_profiler.plotting_scripts.duration_timeseriesView source

def plot_duration_timeseries(profiling_data: ProfilingResults | Sequence[ProfilingResults], ranks: list[int] | int | None = None, include: list[str] | str | None = None, exclude: list[str] | str | None = None, filepath: str | None = None, show: bool = False, verbose: bool = True, cmap: str = DEFAULT_CMAP, log_scale: bool = False, data_filepath: str | Path | None = None, data_format: str = 'csv', backend: str = 'matplotlib', return_fig: bool = False) -> None

Plot each region’s call duration over wall-clock time, with a min-max band.

One line per region tracks the mean duration over the ranks that recorded each call, shaded between the minimum and maximum duration seen across those ranks, so rank imbalance shows up as a widening band.

Parameters

backend : str Backend to use for rendering: “matplotlib” (default) or “plotly”. return_fig : bool Return the rendered figure instead of the default None. Matplotlib returns (fig, axes); Plotly returns its figure object.

plot_speedupfunction#

Re-exported from scope_profiler.plotting_scripts.scalingView source

def plot_speedup(profiling_data: ProfilingResults | Sequence[ProfilingResults], x_field: str = 'num_ranks', ranks: list[int] | int | None = None, include: list[str] | str | None = None, exclude: list[str] | str | None = None, filepath: str | None = None, show: bool = False, verbose: bool = True, cmap: str = DEFAULT_CMAP, data_filepath: str | Path | None = None, data_format: str = 'csv', backend: str = 'matplotlib', return_fig: bool = False) -> object | None

Plot scope speedup versus a chosen parallelism/metadata field using maxplotlib.

Parameters

backend : str Backend to use for rendering: “matplotlib” (default) or “plotly”.

Statistics export

collect_region_statisticsfunction#

Re-exported from scope_profiler.plotting_scripts.statisticsView source

def collect_region_statistics(profiling_data: ProfilingResults | Sequence[ProfilingResults], ranks: list[int] | int | None = None, include: list[str] | str | None = None, exclude: list[str] | str | None = None, labels: Sequence[str] | None = None) -> dict

Collect aggregate region-duration statistics for one or more profiling files.

write_region_statistics_jsonfunction#

Re-exported from scope_profiler.plotting_scripts.statisticsView source

def write_region_statistics_json(profiling_data: ProfilingResults | Sequence[ProfilingResults], filepath: str | Path, ranks: list[int] | int | None = None, include: list[str] | str | None = None, exclude: list[str] | str | None = None, labels: Sequence[str] | None = None) -> dict

Write aggregate region-duration statistics to a JSON file.

Exporting to other tools

export_speedscopefunction#

View source

def export_speedscope(profiling_data: ProfilingResults | Sequence[ProfilingResults], filepath: str | Path, ranks: list[int] | int | None = None, include: list[str] | str | None = None, exclude: list[str] | str | None = None, verbose: bool = True) -> list[Path]

Write speedscope JSON files for the selected ranks.

One file is written per input HDF5 file, holding one profile per rank --- or, for a run profiled with track_threads, one per thread and task. Unlike .prof, the format carries several profiles per file, and speedscope switches between them from its profile selector.

Parameters

profiling_data : ProfilingResults | Sequence[ProfilingResults] The run(s) to export: file runs, in-memory results from ProfileManager.finalize(return_results=True), or a mix. filepath : str | Path Base output path, e.g. figures/profile.speedscope.json. The input file’s stem is appended when more than one file is exported. ranks : list[int] | int, optional Ranks to export (default: rank 0 only). include, exclude : list[str] | str, optional Region name filters, as for the plotting functions.

Returns

list[Path] The files written, in the order they were written.

export_chrome_tracefunction#

View source

def export_chrome_trace(profiling_data: ProfilingResults | Sequence[ProfilingResults], filepath: str | Path, ranks: list[int] | int | None = None, include: list[str] | str | None = None, exclude: list[str] | str | None = None, verbose: bool = True) -> list[Path]

Write Chrome Trace JSON files, one per input profiling run.

export_proffunction#

View source

def export_prof(profiling_data: ProfilingResults | Sequence[ProfilingResults], filepath: str | Path, ranks: list[int] | int | None = None, include: list[str] | str | None = None, exclude: list[str] | str | None = None, call_paths: bool = True, verbose: bool = True) -> list[Path]

Write per-rank .prof files readable by pstats/snakeviz.

Parameters

profiling_data : ProfilingResults | Sequence[ProfilingResults] The run(s) to export: file runs, in-memory results from ProfileManager.finalize(return_results=True), or a mix. filepath : str | Path Base output path, e.g. figures/profile.prof. A _rank<N> suffix is appended per rank (and the input file’s stem too, when more than one file is exported), since .prof has no notion of ranks or runs. ranks : list[int] | int, optional Ranks to export (default: rank 0 only). include, exclude : list[str] | str, optional Region name filters, as for the plotting functions. call_paths : bool, optional Preserve each reconstructed parent/child path as a separate node in the exported pstats tree. This makes SnakeViz distinguish same-named regions called below different parents. Set to False for a compact, name-aggregated export. verbose : bool, optional Print each file written, with the snakeviz command that opens it.

Returns

list[Path] The files written, in the order they were written.

to_pstatsfunction#

View source

def to_pstats(profiling_data: ProfilingResults | Sequence[ProfilingResults], ranks: list[int] | int | None = None, include: list[str] | str | None = None, exclude: list[str] | str | None = None, call_paths: bool = True) -> dict[tuple[str, int], pstats.Stats]

Build a :class:pstats.Stats per (run label, rank), without disk I/O.

The in-memory twin of :func:export_prof: same call reconstruction and filtering, but returns real pstats.Stats objects keyed by (run_label, rank) instead of writing .prof files.

Parameters

profiling_data : ProfilingResults | Sequence[ProfilingResults] The run(s) to export: file runs, in-memory results from ProfileManager.finalize(return_results=True), or a mix. ranks : list[int] | int, optional Ranks to export (default: rank 0 only). include, exclude : list[str] | str, optional Region name filters, as for the plotting functions. call_paths : bool, optional Preserve each reconstructed parent/child path as a separate node in the resulting tree; see :func:build_pstats_dict.

Returns

dict[tuple[str, int], pstats.Stats] Maps (run_label, rank) to its pstats.Stats, empty on ranks that own no files.

build_pstatsfunction#

View source

def build_pstats(calls: CallArrays, root_name: str | None = None, call_paths: bool = True) -> pstats.Stats

Build reconstructed calls straight into a :class:pstats.Stats.

The in-memory twin of :func:build_pstats_dict, for callers that want to sort_stats/print_stats immediately instead of writing a .prof file first. A run with no calls yields an empty Stats, which pstats would otherwise refuse to construct from an empty dict.

load_proffunction#

View source

def load_prof(filepath: str | Path) -> pstats.Stats

Read a .prof file written by :func:export_prof back as a Stats.

The .prof twin of :func:~scope_profiler.json_export.read_json.

ScopeProfileclass#

View source

class ScopeProfile(stats: dict[tuple[str, int, str], tuple])

Bases: cProfile.Profile

A :class:cProfile.Profile carrying reconstructed region stats.

pstats.Stats only accepts a filename or an object with a create_stats() method that leaves the finished pstats dict in .stats (see pstats.Stats.load_stats) - cProfile.Profile is the real class satisfying that contract. Subclassing it, instead of merely duck-typing it, lets :func:build_pstats hand pstats.Stats a type it actually declares support for, and skips Profile.__init__ (which would start a C-level profiler this class never uses).

Inherited Profile methods that only need .stats work as usual, so ScopeProfile(stats).dump_stats(path) writes the same .prof file :func:write_prof_file does. Like a real Profile, an instance is consumed by pstats.Stats: it takes .stats and leaves the profile empty, so build a new one per Stats.

statsattributeinstance attribute#

View source

stats = stats

create_statsmethod#

View source

def create_stats() -> None

No-op: self.stats is already the finished pstats dict.

export_jsonfunction#

View source

def export_json(profiling_data: ProfilingResults | Sequence[ProfilingResults], filepath: str | Path, ranks: list[int] | int | None = None, include: list[str] | str | None = None, exclude: list[str] | str | None = None, verbose: bool = True, indent: int | None = None) -> list[Path]

Write one JSON document per run, exactly as export_speedscope does.

Parameters

profiling_data : ProfilingResults | Sequence[ProfilingResults] The run(s) to export: file runs, in-memory results, or a mix. filepath : str | Path Base output path, e.g. figures/profile.json. Ending it in .gz compresses the output. The run’s label is appended to the stem when more than one run is exported. ranks : list[int] | int, optional Ranks to export (default: every rank). include, exclude : list[str] | str, optional Region name filters, as for the plotting functions. indent : int, optional Passed to json.dumps; the default writes the document on one line.

Returns

list[Path] The files written, in the order they were written.

LIKWID

See LIKWID hardware counters for the workflow and the HDF5 layout.

LikwidRegionResult

LikwidRegionResultclassdataclass#

View source

class LikwidRegionResult(tag: str, group_id: int = -1, group_name: str = '', cpus: list[int] = list(), times: np.ndarray = lambda : np.zeros(0)(), call_counts: np.ndarray = lambda : np.zeros(0, dtype=np.int64)(), event_names: list[str] = list(), counter_names: list[str] = list(), events: np.ndarray = lambda : np.zeros((0, 0))(), metric_names: list[str] = list(), metrics: np.ndarray = lambda : np.zeros((0, 0))(), source: str = 'full_api')

Counter results for a single LIKWID marker region.

The per-thread arrays are all indexed by the same thread axis, so events[e, t] and metrics[m, t] refer to the thread whose runtime is times[t] and whose CPU is cpus[t].

Attributes

tag : str Region name as passed to markerstartregion. group_id : int Index of the LIKWID event group this region was measured with. group_name : str Name of that group (e.g. "CLOCK"), empty if unknown. cpus : list of int Hardware threads that took part in the region. times : numpy.ndarray Per-thread accumulated runtime in seconds, shape (nthreads,). call_counts : numpy.ndarray Per-thread number of times the region was entered, shape (nthreads,). event_names : list of str Names of the raw hardware events, length nevents. Not unique: a group may program one event on several counters (see :attr:event_labels). counter_names : list of str Hardware counter register each event was programmed on (FIXC0, PMC1, MBOX3C0, …), length nevents. Empty for files written before counter names were recorded. events : numpy.ndarray Raw counter values, shape (nevents, nthreads). metric_names : list of str Names of LIKWID’s derived metrics, length nmetrics. metrics : numpy.ndarray Derived metric values, shape (nmetrics, nthreads). source : str Which collection path produced this result: "full_api" (perfmon read-back, the only one with real event names and metrics), "marker_file" (LIKWID’s marker file parsed directly --- real values, placeholder event names, no metrics) or "marker_api" (a markergetregion snapshot of the calling thread only).

call_countsattributeclass attributeinstance attribute#

View source

call_counts: np.ndarray = field(default_factory=lambda : np.zeros(0, dtype=np.int64))

counter_namesattributeclass attributeinstance attribute#

View source

counter_names: list[str] = field(default_factory=list)

cpusattributeclass attributeinstance attribute#

View source

cpus: list[int] = field(default_factory=list)

event_namesattributeclass attributeinstance attribute#

View source

event_names: list[str] = field(default_factory=list)

eventsattributeclass attributeinstance attribute#

View source

events: np.ndarray = field(default_factory=lambda : np.zeros((0, 0)))

group_idattributeclass attributeinstance attribute#

View source

group_id: int = -1

group_nameattributeclass attributeinstance attribute#

View source

group_name: str = ''

metric_namesattributeclass attributeinstance attribute#

View source

metric_names: list[str] = field(default_factory=list)

metricsattributeclass attributeinstance attribute#

View source

metrics: np.ndarray = field(default_factory=lambda : np.zeros((0, 0)))

sourceattributeclass attributeinstance attribute#

View source

source: str = 'full_api'

tagattributeinstance attribute#

View source

tag: str

timesattributeclass attributeinstance attribute#

View source

times: np.ndarray = field(default_factory=lambda : np.zeros(0))

event_labelsproperty#

View source

event_labels: list[str]

Unique per-event labels, safe to use as dict or column keys.

event_names alone is not unique. A group such as MEM_DP programs the same event on one counter per memory channel, so CAS_COUNT_RD legitimately appears eight times on a socket with eight channels (the ones reading zero are unpopulated). Keying anything by the bare name silently keeps only the last channel.

Names that occur once are returned unchanged; repeated ones get the hardware counter appended --- CAS_COUNT_RD:MBOX0C0, CAS_COUNT_RD:MBOX1C0, … --- or a positional suffix if the file predates counter names being recorded.

as_dictmethod#

View source

def as_dict() -> dict

Return the region’s results as a plain dictionary.

Convenience for callers that want to build a DataFrame or dump the numbers without touching the array layout.

Summary tables

likwid_tablesfunction#

View source

def likwid_tables(results, include=None, exclude=None, ranks=None) -> list

Build one LIKWID counter table per (rank, event group).

Regions become columns and counters become rows: a run typically has a handful of regions but a few dozen counters, so this orientation stays readable where the transpose would not. Splitting per event group keeps every column of a table comparable, since a group fixes which events and metrics exist.

Columns are ordered by descending LIKWID runtime (ties alphabetically), so the costliest regions lead, as in the region table.

Parameters

results : ProfilingResults Source of the LIKWID results. include, exclude : list of str or str, optional Regex patterns selecting which regions to report, matched as for the region table. ranks : list of int, optional Restrict to these ranks (default: all).

Returns

list of dict One entry per table, each with rank, group, columns (the region labels) and sections ((title, rows) pairs, where a row is (name, values)). Empty when the file holds no LIKWID data.

print_likwid_tablefunction#

View source

def print_likwid_table(table, title=None, stream=None) -> None

Print one LIKWID counter table from :func:likwid_tables.

print_likwid_tablesfunction#

View source

def print_likwid_tables(results, include=None, exclude=None, ranks=None, stream=None)

Print every LIKWID counter table a result set exposes.

A no-op for files recorded without LIKWID, so callers can invoke it unconditionally.

Collection and storage

collect_marker_results_isolatedfunction#

View source

def collect_marker_results_isolated(timeout: float = 120.0)

Run the perfmon read-back in a child process and return its results.

:func:collect_marker_results has to re-initialize LIKWID’s perfmon module, and on hosts where the counters are not really usable that can abort the process outright instead of raising --- which at finalize() time would destroy a completed run’s output. Running it behind a process boundary turns that worst case into a missing enrichment.

Parameters

timeout : float, optional Seconds to wait for the child before giving up (default: 120).

Returns

list of LikwidRegionResult or None None when the child could not deliver results (crashed, timed out, or LIKWID refused to re-open the counters), which tells the caller to fall back to :func:parse_marker_file.

parse_marker_filefunction#

View source

def parse_marker_file(path=None) -> list[LikwidRegionResult]

Read LIKWID’s marker file directly, without calling into LIKWID.

The file markerclose() writes is plain text::

<nthreads> <nregions> <ngroups>
<region_id>:<tag>-<group_id>          (one line per region)
<region_id> <group_id> <cpu> <call_count> <time> <nevents> <values...>

Parsing it here is the crash-proof path: it touches no counters and calls no LIKWID function, so it cannot take the interpreter down the way re-initializing perfmon can. The cost is that event names, counter registers and derived metrics are not in the file --- those exist only inside LIKWID’s group definitions --- so events come back positionally named and the metric list is empty.

Parameters

path : str, optional Marker file to read (default: $LIKWID_FILEPATH).

Returns

list of LikwidRegionResult One entry per region, ordered by region id. Empty if the file is missing or malformed.

collect_marker_resultsfunction#

View source

def collect_marker_results(pylikwid) -> list[LikwidRegionResult]

Read every region of the run back from LIKWID’s marker file.

Must be called after markerclose(), which is what writes the file. Because markerclose() also tears the perfmon module down, the event sets named in LIKWID_EVENTS are re-registered first; that is what makes event and metric names available for the values in the file.

Parameters

pylikwid : module The imported pylikwid module.

Returns

list of LikwidRegionResult One entry per region recorded by LIKWID, in file order. Empty when the process is not running under the marker API, or when the performance counters cannot be re-opened.

collect_region_snapshotsfunction#

View source

def collect_region_snapshots(pylikwid, region_names: Iterable[str]) -> list[dict]

Read the current counter values for each named region.

Uses the marker API’s markergetregion, so it must be called before markerclose() and reports only the calling thread. Regions that LIKWID does not know about (never entered, or measured under a different name) are skipped.

Parameters

pylikwid : module The imported pylikwid module. region_names : iterable of str Region tags to query.

Returns

list of dict One entry per region that returned data, with keys tag, nevents, events, time and count.

snapshots_to_resultsfunction#

View source

def snapshots_to_results(snapshots: Iterable[dict]) -> list[LikwidRegionResult]

Convert marker-API snapshots into the common result structure.

Used as the fallback when the full API cannot re-open the counters, so the snapshots taken before markerclose() can still be written to HDF5.

write_likwid_resultsfunction#

View source

def write_likwid_results(h5file, results: Iterable[LikwidRegionResult], environment: dict | None = None) -> None

Write collected LIKWID results into an open HDF5 file.

Creates (replacing any previous copy) a likwid group holding one subgroup per region under likwid/regions/<tag>.

Parameters

h5file : h5py.File or h5py.Group Destination, typically the per-rank profiling file. results : iterable of LikwidRegionResult Regions to store. environment : dict, optional LIKWID environment variables to record as attributes on the group, so the event set a file was measured with stays with the data.

markers_availablefunction#

View source

def markers_available() -> bool

Whether this process runs under the LIKWID marker API.

markerinit() silently degrades to a no-op when LIKWID’s environment is absent, so this is what distinguishes “there will be counter data” from “the script was started as a plain python script.py”.

likwid_environmentfunction#

View source

def likwid_environment() -> dict

Return the LIKWID environment of the current process.

Returns

dict The LIKWID_* variables set by the launcher. Empty when the process was not started under likwid-perfctr -m.