8. Reports and comparison

Download notebook View on GitHubRun when the site was built.

Install for this tutorial: pip install "scope-profiler[pproc]" (interactive charts in the HTML report). The site command also needs scope-profiler[site] (PyYAML), which pproc already includes.

A profile is most useful next to another one: the same code before and after a change, two job sizes, yesterday’s run and today’s. This notebook takes the command-line tools that work on finished profile files:

scope-profiler inspect — summarize one or more runs
scope-profiler diff — compare two runs region by region
scope-profiler report — write a self-contained HTML report
scope-profiler check — fail a CI job when a run regressed
scope-profiler site — publish a catalogue of runs as a website

To have something to compare, the first step records three runs of a toy solver: a baseline, a candidate whose solve has become twice as slow, and a “fixed” version that is back to baseline speed.

Three runs

time.sleep stands in for real work, as in the other tutorials, so the timings are stable on any machine. Each run gets a label, which is the name diff, inspect and the report use for it.

import time

from scope_profiler import ProfileManager


def simulate(path, label, solve_cost=0.004, steps=20):
    """Record one run of the toy solver."""
    with ProfileManager.session(file_path=path, label=label, verbose=False):
        with ProfileManager.region("setup"):
            time.sleep(0.02)
        for _ in range(steps):
            with ProfileManager.region("timestep"):
                with ProfileManager.region("assemble"):
                    time.sleep(0.002)
                with ProfileManager.region("solve"):
                    time.sleep(solve_cost)


simulate("baseline.h5", "v1 (baseline)")
simulate("candidate.h5", "v2 (slower solve)", solve_cost=0.008)
simulate("fixed.h5", "v3 (fixed)")

Everything below is a command-line tool, so the next cell defines a small helper that runs scope-profiler in a subprocess and shows the command, its output and its exit code. In a terminal you would type the same command directly. (The helper uses python -m scope_profiler, which is the same program as the scope-profiler executable.)

import os
import shlex
import subprocess
import sys


def sp(command):
    """Run `scope-profiler <command>` and print the command, output and exit code."""
    env = dict(os.environ, COLUMNS="100")  # the width of the tables
    args = [sys.executable, "-m", "scope_profiler", *shlex.split(command)]
    result = subprocess.run(args, check=False, capture_output=True, text=True, env=env)
    print("$ scope-profiler", command)
    print(result.stdout + result.stderr)
    print("exit code:", result.returncode)

Summarize with `inspect`

inspect prints a run’s metadata and its region statistics — the same table as results.print_summary(). --regions-only and --metadata-only pick one half; --sort, --columns, --include, --exclude and --ranks shape the table. It also takes several files or a glob such as 'run_*.h5', printing one summary per file.

sp("inspect baseline.h5 --regions-only --sort total")

inspect output

$ scope-profiler inspect baseline.h5 --regions-only --sort total
==============================================================================
v1 (baseline) - baseline.h5
1 rank(s), 5 region(s), 0.02 MiB, 0.142691 s wall clock, 0.145195 s total (setup to finalize)
==============================================================================

  ╭────────────────────────────────────────────────────────╮
  │ region                    % session      total [s]     │
  ├────────────────────────────────────────────────────────┤
  │ scope_profiler.session    100.00%        0.142691      │
  │ └─ (own)                     0.09%          0.000128   │
  │ └─ timestep (20x)            85.85%         0.122499   │
  │ │ └─ (own)                     0.05%          0.000077 │
  │ │ └─ solve (20x)               56.90%         0.081197 │
  │ │ └─ assemble (20x)            28.89%         0.041224 │
  │ └─ setup                     14.06%         0.020064   │
  ╰────────────────────────────────────────────────────────╯

  ╭─ Info ──────────────────────────────────────────────────────────────────────╮
  │ Summary: Regions (5)                                                        │
  │                                                                             │
  │ Explore:                                                                    │
  │   Inspect: scope-profiler inspect baseline.h5                               │
  │   TUI:     scope-profiler tui baseline.h5                                   │
  │                                                                             │
  │ Visualize and export:                                                       │
  │   Plot:    scope-profiler plot default baseline.h5 -o plots --show          │
  │   Report:  scope-profiler report baseline.h5 -o report.html                 │
  │   Export:  scope-profiler export plot-data baseline.h5 -o data              │
  │   Lines:   scope-profiler line-profile baseline.h5                          │
  │                                                                             │
  │ Compare runs:                                                               │
  │   Diff:    scope-profiler diff BASE.h5 CANDIDATE.h5                         │
  │   Check:   scope-profiler check BASE.h5 CANDIDATE.h5                        │
  │                                                                             │
  │ Durations are in seconds.                                                   │
  │ Regions may nest, so the summed total can exceed the wall-clock time.       │
  │ % session uses wall-clock coverage; overlapping recursive calls count once. │
  ╰─────────────────────────────────────────────────────────────────────────────╯


exit code: 0

Compare with `diff`

diff a.h5 b.h5 lines up the regions of two runs, with a as the baseline, and prints the change in each. --metric picks the statistic (total by default; also avg, min, max, p50, p95, p99, imbalance or calls), --sort the row order (delta, pct or name) and --threshold PCT hides regions that changed by less than that many percent. It always exits with code 0 — it reports, it does not judge.

sp("diff baseline.h5 candidate.h5")

$ scope-profiler diff baseline.h5 candidate.h5
==============================================================================
a: v1 (baseline) - baseline.h5 (1 rank)
b: v2 (slower solve) - candidate.h5 (1 rank)
==============================================================================

Regions (5)
  ╭─────────────────────────────────────────────────────────────────────────────────────────────╮
  │ region                    total [s] (a)      total [s] (b)      delta           delta [%]   │
  ├─────────────────────────────────────────────────────────────────────────────────────────────┤
  │ scope_profiler.session    0.142691           0.223011           +0.0803206      +56.29%     │
  │ timestep                  0.122499           0.202814           +0.0803152      +65.56%     │
  │ solve                     0.0811973          0.161373           +0.0801756      +98.74%     │
  │ assemble                  0.041224           0.0413398          +0.000115788    +0.2809%    │
  │ setup                     0.0200638          0.0200667          +2.824e-06      +0.01408%   │
  ╰─────────────────────────────────────────────────────────────────────────────────────────────╯


exit code: 0

solve is +100% and everything that contains it (timestep, the session) grows with it, while assemble and setup are unchanged. Per-call averages, and only the regions that moved by more than 5%:

sp("diff baseline.h5 candidate.h5 --metric avg --threshold 5")

$ scope-profiler diff baseline.h5 candidate.h5 --metric avg --threshold 5
==============================================================================
a: v1 (baseline) - baseline.h5 (1 rank)
b: v2 (slower solve) - candidate.h5 (1 rank)
==============================================================================

Regions (3)
  ╭────────────────────────────────────────────────────────────────────────────────────────╮
  │ region                    avg [s] (a)      avg [s] (b)      delta          delta [%]   │
  ├────────────────────────────────────────────────────────────────────────────────────────┤
  │ scope_profiler.session    0.142691         0.223011         +0.0803206     +56.29%     │
  │ timestep                  0.00612493       0.0101407        +0.00401576    +65.56%     │
  │ solve                     0.00405986       0.00806865       +0.00400878    +98.74%     │
  ╰────────────────────────────────────────────────────────────────────────────────────────╯


exit code: 0

The same comparison is available in Python as scope_profiler.diff.diff_rows(), which returns one dict per region:

from scope_profiler import load
from scope_profiler.diff import diff_rows

rows = diff_rows(load("baseline.h5"), load("candidate.h5"), metric="total")
for row in rows[:3]:
    print(
        f"{row['name']:<24} {row['a']:.3f} s -> {row['b']:.3f} s ({row['pct']:+.0f}%)"
    )

scope_profiler.session   0.143 s -> 0.223 s (+56%)
timestep                 0.122 s -> 0.203 s (+66%)
solve                    0.081 s -> 0.161 s (+99%)

An HTML report

report writes one self-contained HTML file: headline numbers, findings, ranked hotspots, the region table with its call tree, per-rank load balance and, with scope-profiler[pproc], interactive charts. Metadata is always included. -o names the file (default report.html). Given one run you get a full report; given several, the first is the baseline of a comparison report that shows what changed in each run, and a full report for every run is written next to it as <output>-<index>-<label>.html.

The charts embed the Plotly runtime, which makes a report about 5 MB whatever the profile. --charts-cdn loads Plotly from a CDN instead — about 30 times smaller, at the price of needing a network connection to draw the charts (the tables never do). --no-charts leaves the charts out altogether.

sp("report baseline.h5 -o report.html")
sp("report baseline.h5 -o report_cdn.html --charts-cdn")
sp("report baseline.h5 candidate.h5 fixed.h5 -o comparison.html --charts-cdn")

$ scope-profiler report baseline.h5 -o report.html
Report written to: report.html

exit code: 0
$ scope-profiler report baseline.h5 -o report_cdn.html --charts-cdn
Report written to: report_cdn.html

exit code: 0
$ scope-profiler report baseline.h5 candidate.h5 fixed.h5 -o comparison.html --charts-cdn
Report written to: comparison.html

exit code: 0

from pathlib import Path

for path in sorted(Path(".").glob("*.html")):
    print(f"{path.name:<40} {path.stat().st_size / 1e3:9.0f} kB")

comparison-0-v1-baseline.html                  150 kB
comparison-1-v2-slower-solve.html              150 kB
comparison-2-v3-fixed.html                     150 kB
comparison.html                                141 kB
report.html                                   4966 kB
report_cdn.html                                150 kB

Opening a report

A report is an ordinary HTML file, so any browser opens it. Add --show to open it as soon as it is written, or open it yourself:

scope-profiler report baseline.h5 candidate.h5 -o comparison.html --show
open comparison.html          # macOS; `xdg-open` on Linux

or from Python, with webbrowser.open(Path("comparison.html").resolve().as_uri()). On a cluster, copy the file to your machine — it needs nothing else.

scope-profiler run my_script.py -o report.html skips the HDF5 file and writes the report directly, since the extension picks the format. From Python the function behind all of this is scope_profiler.html_report.create_html_report(). To list several reports on one page, scope-profiler catalogue writes a small index:

sp("catalogue comparison.html report_cdn.html -o index.html --title 'Solver reports'")

$ scope-profiler catalogue comparison.html report_cdn.html -o index.html --title 'Solver reports'
Catalogue written to: index.html

exit code: 0

Gate a regression with `check`

check is diff with a verdict, meant for CI. It prints the comparison table and exits with code 1 when any region got more than --max-regression percent worse in the chosen metric (the default budget is 0%, so any slowdown fails). --include / --exclude restrict which regions count, --metric chooses the statistic (for example p95 for tail latency, or imbalance for MPI balance) and --fail-on-new also fails on regions that did not exist in the baseline.

Timings of a real program wobble from run to run, so give the budget some headroom. 50% is plenty here, where the regression is +100%:

sp("check baseline.h5 candidate.h5 --max-regression 50")

$ scope-profiler check baseline.h5 candidate.h5 --max-regression 50
Baseline: v1 (baseline) - baseline.h5 (1 rank)
Candidate: v2 (slower solve) - candidate.h5 (1 rank)
Regression budget: +50% (total)
Regression check: FAIL
  ╭─────────────────────────────────────────────────────────────────────────────────────────────╮
  │ region                    total [s] (a)      total [s] (b)      delta           delta [%]   │
  ├─────────────────────────────────────────────────────────────────────────────────────────────┤
  │ solve                     0.0811973          0.161373           +0.0801756      +98.74%     │
  │ timestep                  0.122499           0.202814           +0.0803152      +65.56%     │
  │ scope_profiler.session    0.142691           0.223011           +0.0803206      +56.29%     │
  │ assemble                  0.041224           0.0413398          +0.000115788    +0.2809%    │
  │ setup                     0.0200638          0.0200667          +2.824e-06      +0.01408%   │
  ╰─────────────────────────────────────────────────────────────────────────────────────────────╯

Budget violations: solve, timestep, scope_profiler.session

exit code: 1

The slowed-down candidate fails and names the regions over budget. The fixed version is back within budget and passes:

sp("check baseline.h5 fixed.h5 --max-regression 50")

$ scope-profiler check baseline.h5 fixed.h5 --max-regression 50
Baseline: v1 (baseline) - baseline.h5 (1 rank)
Candidate: v3 (fixed) - fixed.h5 (1 rank)
Regression budget: +50% (total)
Regression check: PASS
  ╭────────────────────────────────────────────────────────────────────────────────────────────╮
  │ region                    total [s] (a)      total [s] (b)      delta          delta [%]   │
  ├────────────────────────────────────────────────────────────────────────────────────────────┤
  │ assemble                  0.041224           0.0411968          -2.7246e-05    -0.06609%   │
  │ scope_profiler.session    0.142691           0.142609           -8.1443e-05    -0.05708%   │
  │ timestep                  0.122499           0.122455           -4.3478e-05    -0.03549%   │
  │ setup                     0.0200638          0.0200659          +2.102e-06     +0.01048%   │
  │ solve                     0.0811973          0.0811974          +1.09e-07      +0.0001342% │
  ╰────────────────────────────────────────────────────────────────────────────────────────────╯


exit code: 0

In a CI job the exit code is all that is needed — the step simply fails:

scope-profiler run -o candidate.h5 benchmark.py
scope-profiler check baseline.h5 candidate.h5 --max-regression 10 --exclude 'setup.*'

The Python spelling is scope_profiler.diff.check_files(), which prints the same report and returns the exit code (0 or 1):

from scope_profiler.diff import check_files

with open(os.devnull, "w") as devnull:
    status = check_files(
        "baseline.h5", "candidate.h5", max_regression=50, stream=devnull
    )
print("check status:", status)

check status: 1

A catalogue website with `site`

For a history of runs — nightly jobs, scaling studies, one case per code version — standalone reports get unwieldy. scope-profiler site reads a YAML manifest that lists cases (sets of runs shown together) and writes the pages and chart data of a catalogue website into an Astro/Starlight project, whose build turns them into a static site you can publish on GitHub Pages. Reading the manifest needs PyYAML (the site extra) and the charts need pproc.

The scope-profiler-catalog repository is a template for such a site. The manifest is small:

manifest = """
title: Solver profiling history
description: Three versions of the toy solver.
base: /solver-profiles
cases:
  - id: solver-versions
    title: Solver versions
    scaling: none
    runs:
      - {path: baseline.h5, label: v1}
      - {path: candidate.h5, label: v2}
      - {path: fixed.h5, label: v3}
"""
with open("manifest.yaml", "w") as handle:
    handle.write(manifest)

sp("site manifest.yaml --check")
sp("site manifest.yaml -o astro --no-charts")

$ scope-profiler site manifest.yaml --check
manifest.yaml: 1 case(s), 3 run(s), ok

exit code: 0
$ scope-profiler site manifest.yaml -o astro --no-charts
Catalogue written to: astro (1 case(s), 3 run(s), 10 files)

exit code: 0

for path in sorted(Path("astro/src/content/docs/catalogue").rglob("*.mdx")):
    print(path.relative_to("astro"))

src/content/docs/catalogue/index.mdx
src/content/docs/catalogue/solver-versions/v1.mdx
src/content/docs/catalogue/solver-versions/v2.mdx
src/content/docs/catalogue/solver-versions/v3.mdx
src/content/docs/catalogue/solver-versions.mdx

--check only validates the manifest and the files it lists, which is what a CI job wants before a build. Without --no-charts the command also writes the chart data under astro/public/catalogue. See the CLI reference for the manifest’s other keys (executions for nightly jobs, attached files, images) and for the sibling scope-profiler benchmark command, which runs repeatable benchmarks and compares their results.

Where to go next

2. Post-processing — the Python side of everything inspect prints.
3. Visualizing results and 6. Building your own plots — charts from code.
CLI reference — every flag of every command.

8. Reports and comparison

Three runs

Summarize with inspect

Compare with diff