4. Profiling modes and configuration

Download notebook View on GitHubRun when the site was built.

Install for this tutorial: pip install scope-profiler covers the executable examples. The optional sections require scope-profiler[line-profiler], scope-profiler[mpi], or scope-profiler[likwid] as stated below.

ProfileManager.setup() decides what is recorded. Under the hood it selects a region class specialised for that combination of options, so the recording path contains no configuration checks at all. The most important choices, in order of precedence (the first one that applies wins):

Every active region records nanosecond timestamps; the flags decide what it records on top of them.

deactivate_file_output is deliberately absent from the table: recording works the same either way, and the flag only decides whether finalize() writes the buffers out.

import os
import subprocess
import sys
import time

from scope_profiler import ProfileManager, load, read_h5


def workload():
    time.sleep(0.002)


def cli(*args):
    """Run ``scope-profiler <args>`` and show what it prints."""
    command = [sys.executable, "-m", "scope_profiler", *args]
    output = subprocess.run(command, capture_output=True, text=True, check=True).stdout
    if output.strip():
        print(output)


def show_region_class():
    region = ProfileManager.region("probe")
    print("  region class:", type(region).__name__)

Turning profiling off

deactivate_profiling=True swaps in a region class whose decorator returns the original function — no wrapper, no timestamps, nothing to strip out of your code before a production run.

ProfileManager.setup(deactivate_profiling=True)
show_region_class()


@ProfileManager.profile
def maybe_profiled():
    workload()


maybe_profiled()
# Nothing was recorded, and finalize() writes no file.
print("  calls recorded:", ProfileManager.get_region("probe").num_calls)

  region class: DisabledProfileRegion
  calls recorded: 0

Keeping data in memory

deactivate_file_output=True records timings but writes no HDF5 file at all, so the buffers stay in-process. Use it for short runs, or when the filesystem is the thing you are trying not to disturb.

ProfileManager.setup(deactivate_file_output=True, file_path="unused.h5")
show_region_class()

for _ in range(3):
    with ProfileManager.region("in_memory"):
        workload()

region = ProfileManager.get_region("in_memory")
print("  calls   :", region.num_calls)
print("  durations [ns]:", region.get_durations_numpy())

  region class: TimeOnlyProfileRegion
  calls   : 3
  durations [ns]: [2069351 2061689 2039175]

results = ProfileManager.finalize(verbose=False, return_results=True)

print("file written  :", os.path.exists("unused.h5"))
print("num_calls     :", results["in_memory"].num_calls)
print("total [s]     :", results["in_memory"].total_duration)

file written  : False
num_calls     : 3
total [s]     : 0.006170215

Note that the in-memory buffers hold raw nanosecond timestamps — the seconds-based API starts at read_h5().

Buffer size

Timings accumulate in a numpy buffer that doubles in size whenever it fills, so a region can record as many calls as memory allows. buffer_limit sets the initial capacity (1024 by default, i.e. 16 KB per region); raising it for a very hot region avoids a handful of reallocations, and lowering it saves memory when profiling produces many sparsely-called regions.

Everything is written to HDF5 once, at finalize(). Because the final length is known by then, the datasets are stored exactly sized — a region with 5 calls costs a few hundred bytes rather than a full chunk.

ProfileManager.setup(buffer_limit=4, file_path="buffered.h5")  # tiny on purpose

for _ in range(10):
    with ProfileManager.region("grows_often"):
        pass

region = ProfileManager.get_region("grows_often")
print("  calls    :", region.num_calls)
print("  capacity :", region.capacity, "(grew from 4)")

ProfileManager.finalize(verbose=False)
print("recorded  :", read_h5("buffered.h5")["grows_often"].num_calls)

  calls    : 10
  capacity : 16 (grew from 4)
recorded  : 10

Recursive profiling

recursive_profile=True traces the functions called by a profiled function and gives each its own region, named <module>.<qualname> — the same idea as cProfile, but scoped to the region you care about. It can also be enabled per decorator with @ProfileManager.profile(recursive=True).

It uses sys.setprofile, so it is much more expensive than explicit regions: reach for it when exploring, not in production.

ProfileManager.setup(file_path="recursive.h5")


def leaf():
    time.sleep(0.001)


def middle():
    for _ in range(2):
        leaf()


@ProfileManager.profile("entry", recursive=True)
def entry():
    middle()


entry()
results = ProfileManager.finalize(verbose=False, return_results=True)
results.print_summary(suppress_notes=True)

  ╭──────────────────────────────────────────╮
  │ region                     total [s]     │
  ├──────────────────────────────────────────┤
  │ entry                      0.002191      │
  │ └─ (own)                      0.000037   │
  │ └─ __main__.middle            0.002153   │
  │ │ └─ (own)                      0.000016 │
  │ │ └─ __main__.leaf (2x)         0.002137 │
  ╰──────────────────────────────────────────╯

Aggregation mode for very hot regions

A region entered a million times would store a million timestamps. aggregation_mode=True keeps only each region’s count, total, minimum, maximum and exclusive time instead — bounded memory, and cheap enough to leave in a hot loop. The trade-off is that there is no per-call timeline, so events(), Gantt charts and flame charts have nothing to show; summaries, diff and reports work as usual. It cannot be combined with line, GPU, NVTX or LIKWID profiling.

ProfileManager.setup(aggregation_mode=True, deactivate_file_output=True)
show_region_class()

for _ in range(100_000):
    with ProfileManager.region("tick"):
        pass

results = ProfileManager.finalize(verbose=False, return_results=True)
tick = results["tick"]
print("  calls     :", tick.num_calls)
print("  total [s] :", round(tick.total_duration, 3))
print("  events    :", results.events())

  region class: AggregateProfileRegion
  calls     : 100000
  total [s] : 0.014
  events    : []

Line-by-line profiling

use_line_profiler=True routes decorated functions through line_profiler as well as recording region timings (it needs the line-profiler extra):

ProfileManager.setup(use_line_profiler=True, file_path="line.h5")

finalize(verbose_line_profiler=True) prints the per-line tables at the end of the run, and the timings are stored in the output file, so scope-profiler line-profile can print them later. With the context-manager form, pass the functions explicitly — region("name", functions=[fn]) — since there is no decorated function to register.

Line profiling re-wraps every function decorated so far, so this notebook runs the example as a script instead, with the CLI’s --line-profile flag:

script = """
import time

import scope_profiler as sp


@sp.profile("hot_function")
def hot_function(n):
    total = 0.0
    for i in range(n):
        total += i**0.5
    time.sleep(0.001)
    return total


hot_function(20_000)
"""
with open("line_demo.py", "w") as handle:
    handle.write(script)

cli("run", "--line-profile", "-q", "-o", "line.h5", "line_demo.py")
cli("line-profile", "line.h5")

Per-line timings

Line profile: line.h5

Rank 0 | hot_function | hot_function (line_demo.py:7)
╭────────┬────────┬────────────┬───────────────┬──────────┬─────────────────────╮
│ line   │ hits   │ time [s]   │ per hit [s]   │ % time   │ source              │
├────────┼────────┼────────────┼───────────────┼──────────┼─────────────────────┤
│ 9      │ 1      │ 7.82e-07   │ 7.82e-07      │ 0.01     │ total = 0.0         │
│ 10     │ 20001  │ 0.00269025 │ 1.34506e-07   │ 36.79    │ for i in range(n):  │
│ 11     │ 20000  │ 0.00355299 │ 1.7765e-07    │ 48.59    │     total += i**0.5 │
│ 12     │ 1      │ 0.0010667  │ 0.0010667     │ 14.59    │ time.sleep(0.001)   │
│ 13     │ 1      │ 1.733e-06  │ 1.733e-06     │ 0.02     │ return total        │
╰────────┴────────┴────────────┴───────────────┴──────────┴─────────────────────╯

Profiling a script you cannot edit

scope-profiler run activates the explicit regions of a script — no setup() call needed, the runner configures and finalizes the profiler around it. For a script with no instrumentation, add --recursive to trace its function calls:

scope-profiler run my_script.py --arg value                  # explicit regions only
scope-profiler run --recursive my_script.py -o profiling_data.h5
scope-profiler run --recursive --all my_script.py            # also stdlib and installed packages

By default recursive tracing skips stdlib and site-packages frames so the output stays focused on your code, and --include / --exclude take globs for the function names. Here is the command for real, on a two-function script:

script = """
import time


def leaf():
    time.sleep(0.001)


def main():
    for _ in range(3):
        leaf()


main()
"""
with open("my_script.py", "w") as handle:
    handle.write(script)

cli("run", "--recursive", "-q", "-o", "script.h5", "my_script.py")

load("script.h5").print_summary(suppress_notes=True)

  ╭──────────────────────────────────────────────╮
  │ region                       total [s]       │
  ├──────────────────────────────────────────────┤
  │ my_script.py                 0.005529        │
  │ └─ (own)                        0.002315     │
  │ └─ __main__.<module>            0.003214     │
  │ │ └─ (own)                        0.000023   │
  │ │ └─ __main__.main                0.003191   │
  │ │ │ └─ (own)                        0.000028 │
  │ │ │ └─ __main__.leaf (3x)           0.003163 │
  ╰──────────────────────────────────────────────╯

The Python equivalent is ProfileManager.profile_script(...), which returns the results; ProfileManager.run_script(...) is the lower-level call between your own setup() and finalize().

More options

A few more setup() options, each covered on its own guide page: track_threads=True / track_async=True give every thread or asyncio task its own lane (Threads, tasks and coroutines), profile_mpi_calls=True times the mpi4py communication calls, and use_memray=True records allocations. Profiling modes has the full list and the matching scope-profiler run flags.

MPI

Under MPI each rank collects its own events and finalize() gathers them onto rank 0, which writes the single merged output (finalizing is therefore collective: every rank must call it). Nothing changes in your code — install the mpi extra and run:

mpirun -n 4 python my_simulation.py

The merged file holds every rank’s events, and every MPIRegion property aggregates over them (see 2. Post-processing).

LIKWID hardware counters

With use_likwid=True each region is wrapped in a LIKWID marker region, so hardware counters are attributed per region. This needs LIKWID and pylikwid, and the job must be launched through LIKWID:

likwid-mpirun -n 2 -g FLOPS_SP -mpi openmpi -marker python my_simulation.py

Raw events and derived metrics are collected from LIKWID and stored beside the timing data in scope-profiler’s HDF5 output.

Choosing a mode

Production runs — the defaults. Overhead is about a third of a microsecond per region entry.
Very hot regions — wrap a coarser scope, or use aggregation_mode=True. A region entered millions of times will dominate its own measurement.
Exploration — recursive_profile=True or scope-profiler run --recursive to find the hotspots, then place explicit regions and turn it back off.
Shipping the instrumentation — leave the decorators in and set deactivate_profiling=True; they cost nothing.

examples/benchmark_overhead.py in the repository measures the per-call cost of each mode on your own machine.

5. Custom analysis and 6. Building your own plots pick the recorded data apart call by call, for the questions the built-in summaries and charts do not answer.