Jupyter/IPython magics

Jupyter/IPython magics#

scope_profiler.ipython_magics adds eleven magics to a notebook or IPython session, so a quick “how long is this cell/line taking, and where” doesn’t need the ProfileManager.session()/profile_region() boilerplate spelled out by hand each time. It is a thin adapter, not a second implementation: every magic calls straight into the same API the HDF5 output & post-processing from Python page describes – ProfileManager.session/profile_region, ProfilingResults.print_summary/to_dataframe, read_h5, the diff_rows/print_diff_table functions behind scope-profiler diff, and prof_export/speedscope_export.

%%scope / %scope_timeit / %%scope_line / %%scope_recursive / %%scope_agg
    |  ProfileManager.session(return_results=True)
    v
ProfilingResults  ---kept in memory, keyed by name---   <-- %scope_load (read_h5)
    |
    +-- print_summary()      (the recording magics, %scope_last)
    +-- to_dataframe() / to_events_dataframe()   (%scope_df)
    +-- plot_durations()     (-p / --plot)
    +-- diff_rows() + print_diff_table()   (%scope_compare)
    +-- export_prof() / export_speedscope()   (%scope_export)

A rule of thumb for which recording magic to reach for:

You want

Magic

Time this cell as one block

%%scope

Time one statement, repeatedly

%scope_timeit

Find the slow line in a function

%%scope_line

Find the slow function, with nothing instrumented

%%scope_recursive

Time a region entered millions of times

%%scope_agg

Installing and loading#

pip install "scope-profiler[notebook]"

This is a separate extra from the base install (it pulls in IPython); pip install scope-profiler alone is unaffected. -p/--plot additionally needs the pproc extra, exactly as scope-profiler plot does.

Load the extension once per kernel:

%load_ext scope_profiler.ipython_magics

%%scope – profile a cell#

%%scope solve
result = solve(problem)

Runs the cell inside a profiling region named solve (the name is optional; it defaults to "cell") and prints the region summary table immediately after. Nothing is written to disk – the result lives in the kernel’s memory for the rest of the session, stored under that name for %scope_last and %scope_compare to refer back to.

Options:

  • -q/--quiet – suppress the printed table (still records the run).

  • -p/--plot – also show a duration bar chart (plot_durations).

  • --include PATTERN – regex restricting which regions the table/plot show, same matching ProfilingResults.get_regions uses.

To profile sub-parts of the cell separately, nest ProfileManager.profile_region calls inside it as usual – %%scope only adds the outer region and the in-memory session, everything nested inside works exactly as it does in a script.

%scope_timeit – time a statement#

%scope_timeit -n 20 solve(problem)

Runs the given statement n times (default 7) inside a profiling region named "timeit" and prints calls/total/avg/min/max. It answers the same question as the built-in %timeit, but through scope-profiler’s own region timer and table instead of the timeit module – useful when you want the result to be comparable with %%scope/%scope_compare output, or plan to follow up with -p to see run-to-run variance as a bar chart.

-q suppresses the printed table, same as for %%scope.

%%scope_line – line-by-line profiling for a cell#

%%scope_line
from scope_profiler import ProfileManager

@ProfileManager.profile("compute")
def compute(n):
    total = 0
    for i in range(n):
        total += i * i
    return total

compute(1_000_000)

Runs the cell with use_line_profiler=True and prints per-line hit counts/time for whatever the cell itself hands to line_profiler, the same as running the cell as a script (see guide/line_profiler and examples/ex_line_profiling.py). Needs the line-profiler extra:

pip install "scope-profiler[line-profiler]"

Prefer decorating the function you care about with @ProfileManager.profile, as above – it registers cleanly regardless of where it’s called from. A bare with ProfileManager.profile_region(...): at the cell’s top level also works, but line_profiler’s own handling of a module-level code object containing a nested def can cut the printed table off at that def – the same limitation a plain script hits, not something specific to the magic.

-q suppresses the region summary table, -Q suppresses the line-by-line tables; both can be combined.

%%scope_recursive – profile a cell with nothing instrumented#

%%scope_recursive
result = solve(problem)

Records every Python call the cell makes as its own region, named <module>.<qualname>, with no decorators and no profile_region blocks – recursive_profile=True, the same thing scope-profiler run does to a script. The summary comes back sorted by total time, so the answer to “where did this cell’s time actually go?” is the top of the table:

│ scope_profiler.session                       │ 1     │ 100.00%  │ 5.0e-03 │
│ └─ cell                                      │ 1     │  94.22%  │ 4.7e-03 │
│ │ └─ __main__.<module>                       │ 1     │  93.90%  │ 4.7e-03 │
│ │ │ └─ __main__.outer                        │ 1     │  93.63%  │ 4.7e-03 │
│ │ │ │ └─ __main__.inner                      │ 2     │  93.47%  │ 4.7e-03 │
│ │ │ │ │ └─ __main__.inner.<locals>.<genexpr> │ 4k    │  21.46%  │ 1.1e-03 │

This is the magic to reach for first, before you know what to instrument: use it to find the hot function, then wrap that one in %%scope/%%scope_line for a cheap, focused measurement.

Two things to know, both inherited from how recursive profiling works in a script rather than from the magic:

  • Tracing is not limited to your own functions – library internals called from the cell are recorded too, and each traced call carries per-call overhead. Keep the cell small and use --include to filter the table.

  • The cell is run with exec rather than IPython’s own execution, so a trailing expression is not echoed as Out[n]. Assignments land in the notebook namespace exactly as usual.

A cell that raises is reported the way any other failing cell is, and the profile of everything up to the failure is still recorded and printed – which is often exactly what you want to see when a run dies halfway. That holds for %%scope and %%scope_agg too.

%%scope_agg – aggregation mode for very hot regions#

%%scope_agg
for step in range(1_000_000):
    with ProfileManager.profile_region("step"):
        advance(step)

Runs the cell with aggregation_mode=True: each region keeps its count, inclusive total, min/max and exclusive total, and no per-call timeline at all. A region entered a million times therefore costs a few numbers instead of a million timestamps, which is what makes such a cell measurable at all.

The trade-off is that the per-call event data is gone: the Gantt chart, %scope_df --events and the timeline exports have nothing to show. The summary table, %scope_compare and %scope_df (region rows) work as usual. Aggregation mode cannot be combined with line, GPU, NVTX or LIKWID timing.

%scope_load – bring in a run from outside the notebook#

%scope_load results/run_128ranks.h5
%scope_load -n baseline results/before.h5

Reads an HDF5 file with read_h5 and stores it under the same names the other magics use – so a run from an MPI job, a scope-profiler run, or a colleague becomes directly comparable with what you just measured in a cell:

%%scope candidate
solve(problem)

%scope_compare run_128ranks candidate

The name defaults to the file’s stem; -n/--name overrides it and -q skips the summary table.

%scope_df – a run as a pandas DataFrame#

df = %scope_df
df.nlargest(5, "total_duration")

Returns the recorded run as a DataFrame (to_dataframe), so it both renders as a table on its own and can be captured for analysis, grouping or plotting in the notebook – everything 2. Post-processing profiling data does with a file, on the run you just measured. Needs pandas (the pproc extra).

  • -n/--name – which recorded run (default: the most recent).

  • --per-rank – one row per (region, rank) instead of one aggregated row.

  • --events – one row per recorded call (to_events_dataframe) instead of one per region. An aggregation-mode run records no per-call events, so this comes back empty with a note saying why.

  • --include/--exclude – regex region filters.

%scope_last – reprint a previous run#

%scope_last            # the most recently recorded run
%scope_last solve      # a specific one, by name

Reprints the summary table for a run recorded earlier by %%scope or %scope_timeit, without re-running anything. Takes the same -p/--plot and --include options as %%scope. Useful after scrolling past a result, or to re-render it with a different --include filter.

%scope_compare – compare two runs#

%%scope baseline
solve_naive(problem)

%%scope candidate
solve_optimized(problem)

%scope_compare baseline candidate

Aligns the two runs’ regions by name and prints one row per region with the change in a chosen metric – the same table scope-profiler diff prints for two HDF5 files, here for two in-notebook runs. With no arguments, it compares the two most recently recorded runs, so %scope_compare right after the two cells above works without naming them.

  • --metric – one of total, avg, min, max, p50, p95, p99, imbalance, calls (default total).

  • --sort – delta, pct, or name (default delta).

%scope_export – export a run to a file#

%scope_export candidate.prof
%scope_export -n baseline baseline.speedscope.json

Writes a previously recorded run to a .prof file (openable in snakeviz or any pstats viewer) or a speedscope JSON file, via scope_profiler.prof_export.export_prof/ scope_profiler.speedscope_export.export_speedscope – the same functions scope-profiler export uses on an HDF5 file, here applied directly to an in-memory ProfilingResults. Format is inferred from filepath’s extension (--format prof/--format speedscope to force one); -n/--name selects which recorded run to export (default: the most recent one); --include/ --exclude restrict which regions are written.

%scope_reset – drop recorded runs#

%scope_reset            # clear every recorded run
%scope_reset baseline   # drop just one, by name

Recorded runs otherwise accumulate in memory for the life of the kernel. Useful once %scope_last/%scope_compare have several names to choose from and you want a clean slate, or to free a large run’s timeline once you are done with it.

Tutorial#

7. Notebook magics walks through all eleven magics against a small toy workload, including comparing a “before” and “after” version of a function.

Contents