scope_profiler.prof_export

Export merged HDF5 profiling data to the cProfile .prof (pstats) format.

A .prof file is nothing more than a :mod:marshal dump of the dict that cProfile.Profile.dump_stats writes: it maps a (filename, lineno, funcname) key to (cc, nc, tt, ct, callers), where callers maps a caller key to its own (cc, nc, tt, ct) sub-tuple. Writing that dict is enough for :mod:pstats, snakeviz and friends to read the data.

Regions carry no call graph of their own, so the caller/callee relations are reconstructed from timestamp containment (:func:~scope_profiler.call_stack.build_call_arrays). The reconstruction is consumed as columns rather than as one dict per call: a long run can export millions of events while the pstats dictionary holds one entry per distinct call path (or, on request, per region name).

Attributes

Name	Description
NS_PER_SECOND	No description.
PSEUDO_FILENAME	No description.
PSEUDO_LINENO	No description.

Classes

Name	Description
ScopeProfile	A :class:cProfile.Profile carrying reconstructed region stats.

Functions

Name	Description
build_pstats	Build reconstructed calls straight into a :class:pstats.Stats.
build_pstats_dict	Turn reconstructed calls into a pstats-format statistics dict.
export_flamegraph_svg	Render per-rank flame graphs to standalone SVG files with flameprof.
export_prof	Write per-rank .prof files readable by pstats/snakeviz.
load_prof	Read a .prof file written by :func:export_prof back as a Stats.
to_pstats	Build a :class:pstats.Stats per (run label, rank), without disk I/O.
write_prof_file	Marshal a pstats dict to filepath, as cProfile would.

NS_PER_SECONDattributemodule attribute#

View source

NS_PER_SECOND = 1000000000.0

PSEUDO_FILENAMEattributemodule attribute#

View source

PSEUDO_FILENAME = '~'

PSEUDO_LINENOattributemodule attribute#

View source

PSEUDO_LINENO = 0

ScopeProfileclass#

View source

class ScopeProfile(stats: dict[tuple[str, int, str], tuple])

Bases: cProfile.Profile

A :class:cProfile.Profile carrying reconstructed region stats.

pstats.Stats only accepts a filename or an object with a create_stats() method that leaves the finished pstats dict in .stats (see pstats.Stats.load_stats) - cProfile.Profile is the real class satisfying that contract. Subclassing it, instead of merely duck-typing it, lets :func:build_pstats hand pstats.Stats a type it actually declares support for, and skips Profile.__init__ (which would start a C-level profiler this class never uses).

Inherited Profile methods that only need .stats work as usual, so ScopeProfile(stats).dump_stats(path) writes the same .prof file :func:write_prof_file does. Like a real Profile, an instance is consumed by pstats.Stats: it takes .stats and leaves the profile empty, so build a new one per Stats.

statsattributeinstance attribute#

View source

stats = stats

create_statsmethod#

View source

def create_stats() -> None

No-op: self.stats is already the finished pstats dict.

build_pstatsfunction#

View source

def build_pstats(calls: CallArrays, root_name: str | None = None, call_paths: bool = True) -> pstats.Stats

Build reconstructed calls straight into a :class:pstats.Stats.

The in-memory twin of :func:build_pstats_dict, for callers that want to sort_stats/print_stats immediately instead of writing a .prof file first. A run with no calls yields an empty Stats, which pstats would otherwise refuse to construct from an empty dict.

build_pstats_dictfunction#

View source

def build_pstats_dict(calls: CallArrays, root_name: str | None = None, call_paths: bool = True) -> dict[tuple[str, int, str], tuple]

Turn reconstructed calls into a pstats-format statistics dict.

Parameters

calls : CallArrays Calls as returned by :func:~scope_profiler.call_stack.build_call_arrays. root_name : str, optional When given, a synthetic frame of this name is added as the caller of every top-level call, so viewers that draw a single tree (snakeviz) show the whole run instead of only its largest region. call_paths : bool, optional Keep a separate pstats entry for each parent/child path, naming it "parent > child". This makes tree viewers such as SnakeViz show distinct uses of a region instead of merging every call named "child". Set this to False for a compact, name-aggregated export.

Returns

dict Maps (filename, lineno, funcname) to (cc, nc, tt, ct, callers) with times in seconds.

export_flamegraph_svgfunction#

View source

def export_flamegraph_svg(profiling_data: ProfilingResults | Sequence[ProfilingResults], filepath: str | Path, ranks: list[int] | int | None = None, include: list[str] | str | None = None, exclude: list[str] | str | None = None, call_paths: bool = True, width: int = 1200, threshold: float = 0.001, scalable: bool = True, verbose: bool = True) -> list[Path]

Render per-rank flame graphs to standalone SVG files with flameprof.

The aggregated, call-stack view of the same reconstruction :func:export_prof writes: every call of a region below one parent is one frame, sized by cumulative time. :func:~scope_profiler.plot_flame draws the same thing through the plotting stack; this goes through pstats instead, so the result is a self-contained SVG with no JavaScript, which is what a static site or an emailed report wants.

Requires flameprof (pip install "scope-profiler[pproc]").

Parameters

filepath : str | Path Base output path, e.g. figures/flamegraph.svg; a _rank<N> suffix is appended per rank, as for :func:export_prof. width : int Width of the rendered document in pixels. With scalable this only sets the aspect ratio and the point at which frame labels are elided. threshold : float Smallest fraction of total time a frame must take to be drawn (flameprof’s own default, 0.001, is a tenth of a percent). scalable : bool Replace the fixed pixel size with a viewBox so the graph scales to whatever element it is placed in, and drop the XML prolog so it can be inlined in an HTML page. Set False for a standalone SVG file.

Returns

list[Path] The files written, in the order they were written.

export_proffunction#

View source

def export_prof(profiling_data: ProfilingResults | Sequence[ProfilingResults], filepath: str | Path, ranks: list[int] | int | None = None, include: list[str] | str | None = None, exclude: list[str] | str | None = None, call_paths: bool = True, verbose: bool = True) -> list[Path]

Write per-rank .prof files readable by pstats/snakeviz.

Parameters

profiling_data : ProfilingResults | Sequence[ProfilingResults] The run(s) to export: file runs, in-memory results from ProfileManager.finalize(return_results=True), or a mix. filepath : str | Path Base output path, e.g. figures/profile.prof. A _rank<N> suffix is appended per rank (and the input file’s stem too, when more than one file is exported), since .prof has no notion of ranks or runs. ranks : list[int] | int, optional Ranks to export (default: rank 0 only). include, exclude : list[str] | str, optional Region name filters, as for the plotting functions. call_paths : bool, optional Preserve each reconstructed parent/child path as a separate node in the exported pstats tree. This makes SnakeViz distinguish same-named regions called below different parents. Set to False for a compact, name-aggregated export. verbose : bool, optional Print each file written, with the snakeviz command that opens it.

Returns

list[Path] The files written, in the order they were written.

load_proffunction#

View source

def load_prof(filepath: str | Path) -> pstats.Stats

Read a .prof file written by :func:export_prof back as a Stats.

The .prof twin of :func:~scope_profiler.json_export.read_json.

to_pstatsfunction#

View source

def to_pstats(profiling_data: ProfilingResults | Sequence[ProfilingResults], ranks: list[int] | int | None = None, include: list[str] | str | None = None, exclude: list[str] | str | None = None, call_paths: bool = True) -> dict[tuple[str, int], pstats.Stats]

Build a :class:pstats.Stats per (run label, rank), without disk I/O.

The in-memory twin of :func:export_prof: same call reconstruction and filtering, but returns real pstats.Stats objects keyed by (run_label, rank) instead of writing .prof files.

Parameters

profiling_data : ProfilingResults | Sequence[ProfilingResults] The run(s) to export: file runs, in-memory results from ProfileManager.finalize(return_results=True), or a mix. ranks : list[int] | int, optional Ranks to export (default: rank 0 only). include, exclude : list[str] | str, optional Region name filters, as for the plotting functions. call_paths : bool, optional Preserve each reconstructed parent/child path as a separate node in the resulting tree; see :func:build_pstats_dict.

Returns

dict[tuple[str, int], pstats.Stats] Maps (run_label, rank) to its pstats.Stats, empty on ranks that own no files.

write_prof_filefunction#

View source

def write_prof_file(filepath: str | Path, stats: dict[tuple[str, int, str], tuple]) -> Path

Marshal a pstats dict to filepath, as cProfile would.