scope_profiler.native_trace

Reading the per-rank output written by the native (C and Fortran) region APIs.

The C and Fortran modules shipped in scope_profiler/c/ and scope_profiler/fortran/ record regions with no Python involved, and write one small file per rank at sp_finalize().

A C build compiled with SP_USE_HDF5 writes <prefix>_rank<NNNNN>.h5 directly, in the same schema-2 layout :mod:scope_profiler.h5writer produces: one rank’s profile, already readable by :func:~scope_profiler.read_h5 with no import step. Everything this module then does for those files is merging the ranks of one run into a single profile.

Every other native build - Fortran, and C without HDF5 - writes the compact binary trace documented below instead, one .spt per rank. Both write a compatible trace format, so a program built from either (or both) lands in one profile. This module turns those files into the same :class:~scope_profiler.results.ProfilingResults - and the same HDF5 layout — a Python run produces, so a Fortran run gets the whole post-processing stack (summaries, plots, exporters, plot) for free. The two kinds of input mix freely in one import.

Trace layout, little- or big-endian, as written by sp_finalize::

char[8]   "SCOPEPRF"
int32     format version (1 or 2)
int32     rank
int64     number of regions
per region:
    int32     length of the name in bytes
    char[]    name
    - version 2 only --
    int32     length of the source file path in bytes (0 if unknown)
    char[]    source file path
    int32     source line (-1 if unknown)
    - end version 2 only --
    int64     number of calls
    int64[]   start timestamps, nanoseconds
    int64[]   end timestamps, nanoseconds

Version 1 is what the Fortran API writes, and what older C releases wrote: no per-region source location. Version 2 is the current C API, which can attach one via sp_region_at(). Both are readable, per file, so a mixed C/Fortran run merges normally.

The timestamps come from the same clock as :func:time.perf_counter_ns, so Fortran and Python regions from one process tree share a timeline.

Attributes

Name	Description
C_DIR	No description.
FORMAT_VERSION	Newest layout this module writes metadata for; sp_finalize writes the
FORTRAN_DIR	No description.
HDF5_SUFFIX	Extension sp_finalize gives a directly-written HDF5 profile.
KNOWN_FORMAT_VERSIONS	Trace format versions this reader accepts.
MAGIC	First eight bytes of every trace file.
RANK_HDF5_GLOB	How :func:find_traces recognizes per-rank HDF5 output in a directory.
TRACE_SUFFIX	Extension sp_finalize gives a binary trace.

Classes

Name	Description
TraceFormatError	A file is not a scope-profiler Fortran trace, or is truncated.

Functions

Name	Description
c_include_dir	Directory holding scope_profiler.h, for the compiler's -I.
c_source_path	Path to scope_profiler.c, the implementation entry point to compile.
convert_traces	Convert Fortran traces into a standard scope-profiler HDF5 file.
find_traces	Collect native output files from paths, directories, or a mix of both.
fortran_source_path	Path to scope_profiler.f90, the module to compile into your program.
load_traces	Read native output into the standard post-processing API.
read_native_h5	Read the per-rank HDF5 profile an SP_USE_HDF5 C build writes.
read_native_ranks	Read one native output file, whichever of the two formats it is in.
read_trace	Read one rank's trace file.
write_results	Write any :class:ProfilingResults out as a standard HDF5 file.

C_DIRattributemodule attribute#

View source

C_DIR = Path(__file__).resolve().parent / 'c'

FORMAT_VERSIONattributemodule attribute#

View source

FORMAT_VERSION = 2

Newest layout this module writes metadata for; sp_finalize writes the same number. Version 1 (written by the Fortran API, and by older C releases) has no per-region source location; version 2 (the C API) adds one. Both are readable - see :func:read_trace.

FORTRAN_DIRattributemodule attribute#

View source

FORTRAN_DIR = Path(__file__).resolve().parent / 'fortran'

HDF5_SUFFIXattributemodule attribute#

View source

HDF5_SUFFIX = '.h5'

Extension sp_finalize gives a directly-written HDF5 profile.

KNOWN_FORMAT_VERSIONSattributemodule attribute#

View source

KNOWN_FORMAT_VERSIONS = (1, 2)

Trace format versions this reader accepts.

MAGICattributemodule attribute#

View source

MAGIC = b'SCOPEPRF'

First eight bytes of every trace file.

RANK_HDF5_GLOBattributemodule attribute#

View source

RANK_HDF5_GLOB = *_rank[0-9]*{HDF5_SUFFIX}

How :func:find_traces recognizes per-rank HDF5 output in a directory.

Matching the _rank<NNNNN> that sp_finalize writes, rather than every .h5, keeps a merged profile sitting in the same directory (the output of a previous import, say) from being swallowed as an input to the next one. Naming such a file explicitly on the command line still reads it.

TRACE_SUFFIXattributemodule attribute#

View source

TRACE_SUFFIX = '.spt'

Extension sp_finalize gives a binary trace.

TraceFormatErrorclass#

View source

class TraceFormatError(ValueError)

Bases: ValueError

A file is not a scope-profiler Fortran trace, or is truncated.

c_include_dirfunction#

View source

def c_include_dir() -> Path

Directory holding scope_profiler.h, for the compiler’s -I.

Returns

Path The include directory.

c_source_pathfunction#

View source

def c_source_path() -> Path

Path to scope_profiler.c, the implementation entry point to compile.

Its public and private implementation headers sit next to it; :func:c_include_dir is what to put on the compiler’s include path::

cc -c $(python -c             "import scope_profiler.native_trace as t; print(t.c_source_path())")            -I$(python -c             "import scope_profiler.native_trace as t; print(t.c_include_dir())")

Returns

Path The C source file.

convert_tracesfunction#

View source

def convert_traces(inputs, output_path, label: str | None = None)

Convert Fortran traces into a standard scope-profiler HDF5 file.

The result is indistinguishable from one a Python run wrote, so scope-profiler plot / inspect and :func:~scope_profiler.read_h5 work on it directly.

Parameters

inputs : path or sequence of paths Trace files and/or directories containing them. output_path : str or Path HDF5 file to write. label : str, optional Name for the run; defaults to the output file’s stem.

Returns

Path The file that was written.

find_tracesfunction#

View source

def find_traces(inputs) -> list

Collect native output files from paths, directories, or a mix of both.

Parameters

inputs : path or sequence of paths Files to read, and/or directories to search (non-recursively) for *.spt traces and *_rank<NNNNN>.h5 per-rank profiles.

Returns

list of Path The files, sorted, with duplicates removed.

Raises

FileNotFoundError If an input does not exist, or a directory holds neither kind of file.

fortran_source_pathfunction#

View source

def fortran_source_path() -> Path

Path to scope_profiler.f90, the module to compile into your program.

It ships with the package, so this works from an installed wheel::

gfortran -c $(python -c             "import scope_profiler.native_trace as t; print(t.fortran_source_path())")

Returns

Path The Fortran module source.

load_tracesfunction#

View source

def load_traces(inputs, label: str | None = None)

Read native output into the standard post-processing API.

Both formats are accepted, and mix: .spt traces from a Fortran or plain C build, and the per-rank .h5 profiles an SP_USE_HDF5 C build writes. Every rank of a run lands in one result either way.

Parameters

inputs : path or sequence of paths Files and/or directories containing them (see :func:find_traces). label : str, optional Name for the run in summaries, charts and exports.

Returns

ProfilingResults The same object a Python run produces, so every summary, plot and exporter works on it unchanged.

Raises

TraceFormatError If two files claim the same rank.

read_native_h5function#

View source

def read_native_h5(path) -> tuple

Read the per-rank HDF5 profile an SP_USE_HDF5 C build writes.

The file is an ordinary schema-2 profile holding one rank, so this is :func:~scope_profiler.read_h5 plus a regrouping into the (rank, regions) shape :func:read_trace returns - which is what lets one import mix directly-written HDF5 with .spt traces from other ranks.

Parameters

path : str or Path An .h5 file written by sp_finalize(). A file holding several ranks (a merged profile) is accepted too, and contributes all of them.

Returns

tuple (ranks, metadata), where ranks maps a rank to its {region name: Region} and metadata is the run metadata stored in the file.

read_native_ranksfunction#

View source

def read_native_ranks(path) -> tuple

Read one native output file, whichever of the two formats it is in.

The format-agnostic entry point: callers that must accept whatever a native build happened to write - :func:load_traces and ProfileManager.finalize(native_traces=...) - go through this rather than choosing :func:read_trace or :func:read_native_h5 by suffix themselves.

Parameters

path : str or Path A .spt trace or an .h5 profile written by sp_finalize().

Returns

tuple (ranks, metadata), where ranks maps a rank to its {region name: Region}. A .spt trace contributes exactly one rank and no metadata; an .h5 file contributes every rank it holds (one, as sp_finalize() writes it) and the metadata it stores.

read_tracefunction#

View source

def read_trace(path) -> tuple

Read one rank’s trace file.

Parameters

path : str or Path A .spt file written by sp_finalize().

Returns

tuple (rank, regions), where regions maps a region name to (start_times, end_times) int64 arrays in nanoseconds - exactly the shape :class:~scope_profiler.profile_manager.RankPayload carries. Each value also carries .source_file / .source_lineno attributes (both None on a version-1 trace, or a region registered without sp_region_at()), without changing how it unpacks.

Raises

TraceFormatError If the file is not a trace, is truncated, or was written by a newer format version.

write_resultsfunction#

View source

def write_results(results, output_path)

Write any :class:ProfilingResults out as a standard HDF5 file.

Goes through :class:~scope_profiler.h5writer.ProfilingWriter, so the result has exactly the layout a Python run produces - which is what lets an imported (or merged) run be read back by :func:~scope_profiler.read_h5 and fed to every plot and exporter.

Parameters

results : ProfilingResults The run to write. output_path : str or Path HDF5 file to create.

Returns

Path The file that was written.