CLI reference#
All subcommands live under the single scope-profiler executable (also
runnable as python -m scope_profiler).
scope-profiler tui#
Open an interactive browser for a profiling HDF5 file. The Plots
section reuses the plotting functions from scope-profiler plot:
scope-profiler tui profiling_data.h5
Simple duration plots are rendered with Plotext inside the TUI when
selected and update automatically as settings change. Press t to
refresh them manually if needed. Press g to display a plot in a
Matplotlib window, or p to open it as an interactive Plotly figure in
a browser. Press s to save the Matplotlib version as PNG. Saved plots
go to <file-stem>_plots/ by default; use --plot-output DIR to choose
another directory.
For the Flame plot, press v to export the selected data to .prof
format and open it in Snakeviz (pip install snakeviz).
When a plot is selected, its settings panel shows the regions matched by the include/exclude filters as you type. It also supports rank ranges, colormap, log scale, duration metrics, duration sorting and top-N selection, histogram bin count, and imbalance metric. The include and exclude fields autocomplete region names from the loaded profile and accept comma-separated values. Press Apply settings before displaying or saving the plot.
Available TUI plot types include Gantt, flame chart, flame graph,
durations, timeseries, histogram, imbalance, and any LIKWID metrics
recorded in the file. Only the durations plot is rendered with Plotext
inside the TUI; the other plots can be shown with Matplotlib, opened
with Plotly, or saved as files. Install the optional TUI/plotting
dependencies with pip install "scope-profiler[pproc]".
scope-profiler run#
Profile a script’s function calls without modifying it, similar to
python -m cProfile. By default only the script’s own code is
instrumented (the standard library and installed packages are skipped)
to keep overhead low; pass --all to trace everything.
usage: scope-profiler run [-h] [-o OUTFILE] [-q] [--all] [--line-profile]
[--buffer-limit BUFFER_LIMIT]
script ...
Flag |
Description |
|---|---|
|
Path to the merged HDF5 output file (default: |
|
Suppress the per-region summary printed after the run |
|
Also instrument standard-library/installed-package calls (default: only the script’s own code) |
|
Also collect line-by-line timings via |
|
Initial buffer capacity per region; grows as needed (default: 1024) |
scope-profiler run my_script.py [script args...]
scope-profiler inspect#
Print what is inside a profiling file — the run metadata in full, and an overall statistics line per region — without producing any plots. Useful for checking which environment a run came from, and where its time went, at a glance.
usage: scope-profiler inspect [-h] [--include INCLUDE [INCLUDE ...]]
[--exclude EXCLUDE [EXCLUDE ...]]
[--ranks RANKS [RANKS ...]]
[--sort {total,calls,avg,min,max,first,last,std,p50,p95,p99,imbalance,name}] [--full]
[--columns {region,ranks,calls,total,avg,min,max,first,last,std,p50,p95,p99,imbalance} [...]]
[--source NAME [NAME ...]]
[--metadata-only | --regions-only]
files [files ...]
Flag |
Description |
|---|---|
|
Only report regions matching these regex patterns |
|
Skip regions matching these regex patterns |
|
Restrict region statistics to these ranks, e.g. |
|
Order regions by any region-table statistic, or by |
|
Restrict the region table to selected columns; default is |
|
Print long metadata values ( |
|
Print the captured call-site source (the |
|
Also write the metadata of every inspected file to this JSON file |
|
Suppress the printed summary (useful with |
|
Print only the metadata section |
|
Print only the region statistics |
scope-profiler inspect profiling_data.h5 --regions-only
scope-profiler inspect 'run_*.h5' --regions-only --sort calls
scope-profiler inspect profiling_data.h5 --regions-only --columns region ranks calls total avg
scope-profiler inspect profiling_data.h5 --export-metadata metadata.json --quiet
scope-profiler inspect profiling_data.h5 --source solve assemble
Example output:
==============================================================================
profiling_data.h5
1 rank(s), 3 region(s), 0.22 MiB, 0.00275862 s wall clock, 0.00409068 s total (setup to finalize)
==============================================================================
╭────────────────────────┬──────┬─────────────┬─────────────┬───────────╮
│ region │ n │ % session │ total [s] │ avg [s] │
├────────────────────────┼──────┼─────────────┼─────────────┼───────────┤
│ scope_profiler.session │ 1 │ 100.00% │ 2.8e-03 │ 2.8e-03 │
│ └─ matrix_multiply │ 100 │ 5.23% │ 1.4e-04 │ 1.4e-06 │
│ └─ time_step │ 1k │ 52.29% │ 1.4e-03 │ 1.4e-06 │
│ TOTAL │ 1.1k │ 100.00% │ 4.3e-03 │ │
╰────────────────────────┴──────┴─────────────┴─────────────┴───────────╯
╭─ Info ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Summary: Regions (3) │
│ │
│ Explore: │
│ Inspect: scope-profiler inspect ../../../../..profiling_data.h5 │
│ TUI: scope-profiler tui ../../../../..profiling_data.h5 │
│ │
│ Visualize and export: │
│ Plot: scope-profiler plot default ../../../../..profiling_data.h5 -o plots --show │
│ Report: scope-profiler report ../../../../..profiling_data.h5 -o report.html │
│ Export: scope-profiler export plot-data ../../../../..profiling_data.h5 -o data │
│ Lines: scope-profiler line-profile ../../../../..profiling_data.h5 │
│ │
│ Compare runs: │
│ Diff: scope-profiler diff BASE.h5 CANDIDATE.h5 │
│ Check: scope-profiler check BASE.h5 CANDIDATE.h5 │
│ │
│ Durations are in seconds. │
│ Regions may nest, so the summed total can exceed the wall-clock time. │
│ % session uses wall-clock coverage; overlapping recursive calls count once. │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Region durations are in seconds, aggregated over the selected ranks. See HDF5 output & post-processing from Python for what each metadata field means.
Showing a region’s source#
A region can remember the with block or decorated function it was
defined with – captured once when the region is first created, but only
if the run was profiled with capture_region_source=True (off by
default; see Configuration for its cost and
HDF5 output & post-processing from Python for the Python API). --source prints
it, after the region table:
scope-profiler inspect profiling_data.h5 --source solve assemble
Source (2)
solve (kernels.py:42)
with ProfileManager.profile_region("solve"):
return solver.step(state)
assemble (kernels.py:18)
@ProfileManager.profile("assemble")
def assemble(size):
return build_matrix(size)
A name with nothing captured (capture_region_source was left at its
default, the file predates this feature, or the region was only ever
created by the recursive tracer) prints “source not captured” instead of
failing the whole command; an unknown name prints the list of regions
the file actually has.
Exporting metadata to JSON#
--export-metadata writes the metadata of every inspected file to one
JSON document. Values are never clipped there, regardless of --full:
scope-profiler inspect profiling_data.h5 --export-metadata metadata.json --quiet
{
"files": [
{
"file_path": "/scratch/run/profiling_data.h5",
"num_ranks": 2,
"metadata": {
"chip_information": "AMD EPYC 9654 96-Core Processor",
"modules": ["profile/base", "gcc/12.3.0", "python/3.11.7"],
"SLURM_JOB_ID": "9988776",
"...": "..."
}
}
]
}
Several files (or a glob) produce one entry each, which makes the export convenient for comparing the environments of a set of runs. The same is available from Python:
from scope_profiler.inspection import collect_file_metadata, write_metadata_json
payload = write_metadata_json("profiling_data.h5", "metadata.json")
payload = collect_file_metadata(["run_1.h5", "run_2.h5"]) # no file written
scope-profiler report#
Write a self-contained HTML report containing the selected per-region statistics and the complete metadata for one or more profiling files. It has no plotting dependency, so it is suitable for copying from a compute node and opening later in a browser.
scope-profiler report profiling_data.h5 -o report.html
scope-profiler report 'run_*.h5' -o comparison.html --sort calls
scope-profiler report profiling_data.h5 -o report.html --include solve --ranks 0-3
--include, --exclude, --ranks, --sort, and --columns use the
same meaning as scope-profiler inspect and control the
region-statistics table. Metadata is always retained in the report. With
scope-profiler[pproc] installed, reports also embed interactive Plotly
timelines and duration charts; pass --no-charts for the compact
table-only version.
Each run’s section opens with a short auto-generated overview – the region that dominates the recorded time, a load-imbalance callout when one region’s slowest rank is well above the per-rank average, and a note about regions called often enough that timer overhead itself becomes measurable. Clicking a row in the region table expands it in place to show the captured call-site source, tags, and a per-rank calls/total/avg/min/max breakdown, all without any JavaScript dependency beyond a small inline toggle script.
scope-profiler diff#
Compare region statistics between two merged HDF5 profiling files, region by region, so a regression (or an improvement) between two runs – two commits, two configs, two job sizes – shows up as a single table.
usage: scope-profiler diff [-h] [--include INCLUDE [INCLUDE ...]]
[--exclude EXCLUDE [EXCLUDE ...]]
[--ranks RANKS [RANKS ...]]
[--metric {total,avg,min,max,p50,p95,p99,imbalance,calls}]
[--sort {delta,pct,name}] [--threshold PCT]
file_a file_b
Flag |
Description |
|---|---|
|
Only compare regions matching these regex patterns |
|
Skip regions matching these regex patterns |
|
Restrict the statistics to these ranks, e.g. |
|
Statistic to compare: |
|
Order regions by descending ` |
|
Only show regions whose absolute percent change is at least this many percent |
scope-profiler diff baseline.h5 candidate.h5
scope-profiler diff baseline.h5 candidate.h5 --metric avg --threshold 5
scope-profiler check#
Use check in CI to fail when a candidate exceeds a performance budget:
scope-profiler check baseline.h5 candidate.h5 --max-regression 5
The command prints the comparison table and returns exit code 1 when
any region is more than 5% slower. Use --fail-on-new to treat newly
appearing regions as failures, and --metric p95 or
--metric imbalance to enforce a tail-latency or MPI-balance budget.
Example output:
Baseline: ../../../../..baseline.h5 (1 rank)
Candidate: ../../../../..candidate.h5 (1 rank)
Regression budget: +5% (total)
Regression check: FAIL
╭────────────────────────┬─────────────────┬─────────────────┬─────────────┬─────────────╮
│ region │ total [s] (a) │ total [s] (b) │ delta │ delta [%] │
├────────────────────────┼─────────────────┼─────────────────┼─────────────┼─────────────┤
│ solve │ 0.00306308 │ 0.00606336 │ +0.00300028 │ +97.95% │
│ scope_profiler.session │ 0.00417164 │ 0.00823824 │ +0.0040666 │ +97.48% │
│ setup │ 0.00107655 │ 0.00107037 │ -6.179e-06 │ -0.574% │
│ teardown │ - │ 0.00106315 │ +0.00106315 │ - │
╰────────────────────────┴─────────────────┴─────────────────┴─────────────┴─────────────╯
Only in b: teardown
Budget violations: solve, scope_profiler.session
A region present in only one file still gets a delta (treating the
missing side as 0 calls), but a percent change is only reported when the
file it is missing from is b – dropping out from a nonzero baseline in
a is a well-defined -100%, while a region appearing fresh in b has
no baseline to divide by, and is listed under “Only in b” instead.
--threshold never drops those regions, since there is nothing to
compare against.
scope-profiler plot#
Post-process one or more HDF5 profiling files and render a named plot or
plot preset. For text/JSON summaries (including LIKWID hardware
counters), see scope-profiler inspect above instead. For
machine-readable exports without rendering charts, see
scope-profiler export below.
usage: scope-profiler plot [-h]
{list,default,all,quick,gantt,flame_chart,flame_graph,
durations,timeseries,speedup,histogram,imbalance,likwid}
...
scope-profiler plot list prints the available plot kinds and presets.
Positional arguments#
Argument |
Description |
|---|---|
|
One of |
|
Path(s) or glob patterns for |
default renders gantt and the total durations plot. quick
renders durations and speedup. all renders every plot except
likwid; pass --metric to include likwid.
Selecting data#
Flag |
Description |
|---|---|
|
Override a file’s display label in the outputs (repeat once per file, in order) |
|
Region names to include (regex patterns) |
|
Region names to exclude (regex patterns) |
|
Ranks to include; supports ranges (e.g. |
Choosing and rendering plots#
Flag |
Description |
|---|---|
|
Display the plot interactively (default: off) |
|
Directory to save generated outputs; for a single plot kind this may be a target |
|
Renderer: |
|
Matplotlib colormap used to color regions/files in all plots (default: |
Plot-specific options#
Flag |
Plot(s) |
Description |
|---|---|---|
|
durations |
Duration statistics to draw as bar columns: any of |
|
durations, likwid |
Order the bar chart’s regions by this statistic, descending ( |
|
durations, likwid |
Keep only the top N regions after |
|
durations |
Merge several regions into one bar: |
|
durations |
Split each bar into the region’s own (exclusive) time plus one stacked segment per region called directly from it, so a bar shows where its time went. Nesting is reconstructed from timestamps, as in the flame chart; children filtered out of the bars still get a segment. Only |
|
durations, timeseries, histogram, imbalance, likwid |
Logarithmic y-axis |
|
histogram |
Number of duration bins (default: 30) |
|
imbalance |
Per-call duration statistic plotted per rank: any of |
|
likwid |
Name of the LIKWID derived metric or raw event to plot, e.g. |
|
speedup |
X-axis: |
When -o/--output is supplied, the CLI saves one <name>_plot.png per
plot selected by the plot kind or preset (durations_plot.png becomes
one durations_plot_<metric>.png per metric when --metrics requests
several, e.g. durations_plot_avg.png, durations_plot_total.png),
plus region_statistics.json. speedup is skipped unless multiple
files are passed. With --backend plotly the plots are written as
.html instead of .png.
The plotext backend is available for simple plots (durations,
timeseries, speedup, weak_scaling, scaling_efficiency,
histogram, and imbalance). It writes plain terminal renderings as
.txt files. Complex plots such as Gantt charts, flame charts, flame
graphs, heatmaps, and LIKWID charts remain on the Matplotlib or Plotly
backends.
For multiple files, the JSON includes per-file region statistics and the set of common regions across all inputs.
scope-profiler export#
Export profiling data without rendering plot images.
usage: scope-profiler export [-h] {prof,speedscope,plot-data} ...
Plot data#
plot-data writes the raw data behind selected charts as CSV or JSON,
so plots can be reconstructed later without the original HDF5 files:
scope-profiler export plot-data profiling_data.h5 -o data --format json
scope-profiler export plot-data run_1.h5 run_2.h5 -o data \
--plots durations speedup --format json
With JSON, each data file includes a colors map matching the plot
colors. plot-data supports the same data-selection flags as plot,
plus --plots, --format, --metrics, --bins, --imbalance-metric,
--likwid-metric, and --x.
Exporting to .prof for snakeviz#
export prof writes the profile in the format cProfile uses, so
regions can be browsed with any pstats-based viewer:
scope-profiler export prof profiling_data.h5 -o figures
snakeviz figures/profile_rank0.prof
One file is written per exported rank (profile_rank0.prof, …; only the
ranks selected with -r/--ranks are exported, default rank 0), prefixed
with the input file’s stem when several HDF5 files are passed. Regions
become “functions”, cumtime is a region’s total wall time and
tottime is that minus the time spent in its nested regions, with a
synthetic <file rank N> frame as the root of the tree.
Since regions carry no call graph, the caller/callee relations are reconstructed from timestamp containment, exactly as the flame chart does. So:
Regions that only partially overlap (async work, threads) are attributed to whichever region enclosed their start, and the enclosing region’s
tottimeis clamped at zero rather than going negative.A region called from several places is merged into one entry, as pstats is keyed by function rather than by call path; recursion is reported as
2/1-style call counts, likecProfile.LIKWID counters have no place in the
.profformat and are left out.
Exporting to speedscope#
export speedscope writes a speedscope
JSON file. Where .prof keeps aggregates per region, speedscope keeps
every individual call, so the timeline shows the run as it actually
happened — closest in spirit to the Gantt and flame charts, but
interactive:
scope-profiler export speedscope profiling_data.h5 -o figures
Then open figures/profile.speedscope.json at
https://www.speedscope.app (the file never leaves the browser), or run
npx speedscope figures/profile.speedscope.json. Its three views are
all useful here: “Time Order” is the run’s timeline, “Left Heavy”
aggregates identical call paths (the flame graph), and “Sandwich” ranks
regions by self and total time.
One file is written per input HDF5 file, holding one profile per
exported rank (only the ranks selected with -r/--ranks, default rank
0) — the format carries several profiles per file, and speedscope
switches between them from the dropdown in its top bar. Every profile in
a file shares one time origin, so ranks stay aligned with each other.
The stem of the input file is added to the name when several HDF5 files
are passed.
The call graph is reconstructed from timestamp containment, as for
.prof, with one extra consequence: speedscope replays the events as a
stack machine, so a region that starts inside another but ends after it
is clipped to its parent instead of overhanging it. The same caveats
otherwise apply — regions called from several places and recursion
behave as described above.
Examples#
Save plots for a single file:
scope-profiler plot profiling_data.h5 -o figures/
Omitting the plot kind selects the default preset (gantt and total
durations). The explicit equivalent is
scope-profiler plot default profiling_data.h5 -o figures/.
Compare multiple files:
scope-profiler plot default run_1.h5 run_2.h5 run_4.h5 -o figures/
Select files via wildcard patterns:
scope-profiler plot default files/*.h5 -o figures/
scope-profiler plot default "files/file_*.h5" -o figures/
Display interactively with region filtering:
scope-profiler plot profiling_data.h5 --show \
--include "solver.*" "rhs.*" \
--exclude "io"
Select specific MPI ranks:
scope-profiler plot default profiling_data.h5 --show --ranks 0-3 8
The --ranks flag accepts comma-separated values and dash ranges that
can be combined: 0,2,4-7 expands to ranks 0, 2, 4, 5, 6, 7.
Only export average and total duration plots:
scope-profiler plot durations profiling_data.h5 -o figures/ --metrics avg total
Only generate the duration bar chart and speedup plot:
scope-profiler plot quick run_1.h5 run_2.h5 run_4.h5 -o figures/