Profiling C and C++ code

Profiling C and C++ code#

Install for this page: pip install scope-profiler supplies the C/C++ sources. Building the examples requires a C99 compiler (and a C++11 compiler for the RAII wrapper); no Python extra is required.

scope-profiler ships a C region API with the same model as the Python one. A C program marks regions and writes one file per rank at sp_finalize(), so a C run gets the same summaries, charts, exporters and plot workflow as a Python one.

Built with HDF5 — the preferred output format — that file is the profile: <prefix>_rank<NNNNN>.h5, the same layout a Python run writes, opened directly by scope-profiler inspect/plot and read_h5(). Without HDF5 the program writes a compact .spt trace instead, which scope-profiler import-native converts afterwards. See Output format.

The compiled backend is a small C99 entry source plus adjacent headers: with no dependencies beyond libc, no MPI, and nothing else to link unless you opt into HDF5. Keep the shipped scope_profiler_impl.h and scope_profiler_hdf5.h beside scope_profiler.c when vendoring them. The public header is extern "C", so C++ codes can include it directly - or reach for the optional scope_profiler.hpp RAII wrapper described below.

C++ projects can instead use the C++17 header-only backend, and CMake projects can consume exported scope-profiler::native, scope-profiler::cpp, and scope-profiler::header-only targets.

The trace format is shared with the Fortran API, so a program built from both lands in a single profile — and so does a run whose ranks wrote different formats.

Getting the source#

It ships inside the installed package:

cc -c $(python -c "import scope_profiler.native_trace as t; print(t.c_source_path())") \
   -I$(python -c "import scope_profiler.native_trace as t; print(t.c_include_dir())")

or from a checkout, src/scope_profiler/c/. A Makefile and a runnable example.c sit next to it.

Marking regions#

#include "scope_profiler.h"

int main(void)
{
    int step, solve, assemble;

    sp_init("profile", 0);              /* prefix, MPI rank */

    solve = sp_region("solve");         /* resolve names once... */
    assemble = sp_region("assemble");

    for (step = 0; step < nsteps; ++step) {
        sp_begin(assemble);             /* ...then use the handles */
        assemble_matrix();
        sp_end(assemble);

        sp_begin(solve);
        solve_system();
        sp_end(solve);
    }

    sp_finalize();                      /* writes profile_rank00000.h5 */
    return 0;
}

Regions may nest, and a region may re-enter itself recursively — each entry reserves its own slot, exactly as in the Python API.

Nesting must be proper: a region entered inside another has to be left before it. Post-processing reconstructs the call graph from the recorded intervals, so a sp_begin/sp_end pair that crosses another region’s boundary — rather than sitting inside or outside it — has no call stack to recover, and reading the trace back raises NestingError. The timings themselves are still recorded and readable; it is the nesting, the flame chart and the exclusive times that cannot be derived.

The API#

The default-context convenience functions:

Call

Purpose

int sp_init(const char *prefix, int rank)

Start profiling. Returns non-zero if no monotonic clock is available.

int sp_region(const char *name)

Handle for a region name, created on first use.

int sp_region_at(const char *name, const char *file, int line)

Like sp_region(), also recording where it is defined.

void sp_begin(int region) / void sp_end(int region)

Enter and leave a region.

int64_t sp_num_calls(int region)

How often a region was entered; readable after sp_finalize().

int64_t sp_now_ns(void)

The raw clock, in nanoseconds. Shared by every profiler.

int sp_is_active(void)

Whether profiling is running.

int sp_finalize(void)

Write <prefix>_rank<NNNNN>.h5 (or .spt) and stop. Returns non-zero on write failure.

int sp_set_output_format(sp_output_format)

Choose HDF5 or trace output; see Output format.

sp_output_format sp_current_output_format(void) / const char *sp_output_path(void)

What will be written, and where.

Calls made before sp_init() are harmless: sp_region() returns SP_INVALID_REGION and sp_begin/sp_end ignore it. Instrumentation can stay in a build that never profiles, with no #ifdef at the call sites.

The explicit-context API, each taking an sp_profiler * from sp_create():

Call

Purpose

sp_profiler *sp_create(const char *prefix, int rank) / void sp_destroy(sp_profiler *)

Create and release a profiler instance.

int sp_profiler_region(sp_profiler *, const char *name)

Handle for a region name, created on first use.

int sp_profiler_region_at(sp_profiler *, const char *name, const char *file, int line)

Like above, also recording where it is defined.

void sp_profiler_begin(sp_profiler *, int region) / void sp_profiler_end(sp_profiler *, int region)

Enter and leave a region.

int sp_profiler_end_last(sp_profiler *)

End whichever call, in any region, was opened most recently.

sp_scope sp_profiler_scope_begin(sp_profiler *, int region) / int sp_scope_end(sp_scope *)

A checked token for one exact call.

int sp_profiler_reset(sp_profiler *)

Discard recorded calls and statistics; keep region handles valid.

int sp_profiler_get_region_stats(const sp_profiler *, int region, sp_region_stats *)

Call count, total/min/max duration.

int sp_profiler_num_regions(const sp_profiler *) / const char *sp_profiler_region_name(const sp_profiler *, int)

Enumerate registered regions.

int sp_profiler_flush(sp_profiler *)

Write what has been recorded so far, without stopping profiling.

int sp_profiler_finalize(sp_profiler *)

Write the output file and stop.

int sp_profiler_set_output_format(sp_profiler *, sp_output_format) / sp_output_format sp_profiler_output_format(const sp_profiler *)

The format this profiler writes.

int sp_profiler_is_active(const sp_profiler *) / const char *sp_profiler_output_path(const sp_profiler *)

Introspection.

sp_status sp_profiler_last_error(const sp_profiler *) / const char *sp_error_string(sp_status)

What went wrong, if anything.

Every one of these accepts a NULL or inactive profiler harmlessly, the same way the default-context functions tolerate calls before sp_init().

Multiple profilers: explicit contexts#

sp_init()/sp_region()/sp_begin()/sp_end()/sp_finalize() all operate on one hidden default profiler per process. Library code that must not interfere with a caller’s own profiling - or with another instance of itself - creates its own instead:

sp_profiler *p = sp_create("solver", rank);

int solve = sp_profiler_region(p, "solve");
sp_profiler_begin(p, solve);
solve_system();
sp_profiler_end(p, solve);

sp_profiler_finalize(p);   /* writes solver_rank00000.h5 */
sp_destroy(p);

Every sp_create()d instance owns its own regions, buffers and open calls; nothing about one can corrupt another. sp_create() never returns NULL except on outright allocation failure - if no clock is available, it hands back a profiler that is simply inactive (every call against it a harmless no-op, same as the default context before sp_init()), so check sp_profiler_is_active() or sp_profiler_last_error() rather than a NULL return.

A checked call token#

sp_begin()/sp_end() identify a call only by region, so an incorrectly ordered sp_end() is caught only later, on import. sp_profiler_scope_begin() returns a token for the exact call instead:

sp_scope call = sp_profiler_scope_begin(p, solve);
solve_system();
sp_scope_end(&call);

Ending a token whose call is not the innermost one open in its region is refused (SP_ERR_UNMATCHED_END) rather than mistiming whichever call actually is - the token stays valid, so ending it again later, once it is on top, succeeds normally. Ending it twice is harmless: the second call finds nothing left to end. This is what a C++ RAII wrapper should store per scope, since a moved-from token is naturally inert.

For code that tracks “the currently open section” itself instead of naming a region to close it, sp_profiler_end_last(p) ends whichever call - in any region – was opened most recently and not yet closed.

Source-aware registration#

sp_region_at() records where a region is defined alongside its name, using the same (source_file, source_line) shape scope-profiler’s own source navigation and exports already understand:

int solve = sp_profiler_region_at(p, "solve", __FILE__, __LINE__);

It exists on both the default context (sp_region_at) and explicit ones (sp_profiler_region_at). A name registered without a source location (via sp_region(), or sp_region_at() called with NULL) can still pick one up later - the next sp_region_at() call for that name backfills it onto the existing handle - but once a handle has a location, later calls never overwrite it.

SP_REGION_AT(name) / SP_PROFILER_REGION_AT(profiler, name) are macros that fill in __FILE__/__LINE__ for you:

int solve = SP_REGION_AT("solve");                    /* default context */
int solve = SP_PROFILER_REGION_AT(p, "solve");         /* explicit context */

C++: RAII scopes#

scope_profiler.hpp, next to the C header, wraps a checked scope token in a move-only C++11 class so a region is left however its scope exits – return, break, or an exception - instead of a hand-written sp_begin/sp_end pair (or a hand-written wrapper of one’s own):

#include "scope_profiler.hpp"

sp_profiler *p = sp_create("solver", rank);
int solve = sp_profiler_region(p, "solve");

void step()
{
    sp::Scope timed(p, solve);
    solve_system();   // sp_scope_end() runs on every exit path, throw included
}

Against the default context instead of an explicit one, drop the profiler argument:

sp_init("profile", rank);
int solve = sp_region("solve");
...
sp::Scope timed(solve);

It is header-only and pulls in nothing beyond scope_profiler.h, so it never needs to be compiled or linked on its own - only included from C++ translation units. make hpp-check (in src/scope_profiler/c/) compile-checks it in CI.

Scoped instrumentation macros#

The recommended C++ call-site API captures source locations and closes regions on every exit path:

void solve()
{
    SP_PROFILE_FUNCTION();
    SP_PROFILE_SCOPE("assemble");
    assemble_matrix();
}

SP_PROFILE_SCOPE_CTX(profiler, "assemble") and SP_PROFILE_FUNCTION_CTX(profiler) select an explicit profiler. Give each macro its own lexical scope when two invocations would otherwise be on the same source line.

Define SP_DISABLE_PROFILING for an uninstrumented build. All four macros then expand to no-ops: their arguments are not evaluated, their strings are not retained, and no profiler symbol is referenced. The same source can therefore remain instrumented without linking scope-profiler:

c++ -DSP_DISABLE_PROFILING application.cpp -o application

C++17 header-only backend#

Defining SP_HEADER_ONLY before including scope_profiler.hpp supplies inline definitions of the complete recording backend. It is safe across multiple translation units and needs no separately compiled scope_profiler.c:

#define SP_HEADER_ONLY
#include "scope_profiler.hpp"

This mode requires C++17 for inline process-wide state. The ordinary compiled backend and RAII wrapper continue to support C++11.

Native LIKWID counters#

Define SP_USE_LIKWID and LIKWID_PERFMON, include LIKWID’s headers, and link liblikwid. SP_PROFILE_SCOPE* then starts and stops a LIKWID marker with the same name as its timing region. An RAII session owns LIKWID initialization and finalization without involving Python:

#define SP_USE_LIKWID
#include "scope_profiler.hpp"

int main()
{
    sp::LikwidSession counters;
    sp_init("profile", 0);
    {
        SP_PROFILE_SCOPE("solve");
        solve();
    }
    return sp_finalize();
}

Run the executable under likwid-perfctr -m. Worker threads must call sp::likwid_thread_init() once before entering markers. The optional CMake target described below supplies the definitions, include directory, and library automatically.

CMake package#

An installed source distribution exports namespaced targets:

find_package(scope-profiler CONFIG REQUIRED)
target_link_libraries(simulation PRIVATE scope-profiler::cpp)

  • scope-profiler::native is the compiled C99 recorder.

  • scope-profiler::cpp adds the C++11 RAII/macro API and links the recorder.

  • scope-profiler::header-only selects the C++17 header-only recorder.

  • scope-profiler::likwid is available when scope-profiler was configured with -DSCOPE_PROFILER_ENABLE_LIKWID=ON; link this add-on alongside either scope-profiler::cpp or scope-profiler::header-only.

  • scope-profiler::mpi similarly adds scope_profiler_mpi.hpp and MPI linkage when configured with -DSCOPE_PROFILER_ENABLE_MPI=ON.

  • -DSCOPE_PROFILER_ENABLE_HDF5=ON needs no separate target: it puts SP_USE_HDF5 and the HDF5 link on scope-profiler::native and scope-profiler::header-only, so everything built against them writes .h5 profiles. See Output format.

The same targets work through FetchContent_MakeAvailable(scope-profiler). See examples/cmake/ for both the installed-package and FetchContent paths.

Reset, statistics and flushing#

sp_profiler_reset() discards every recorded call and each region’s statistics but keeps every region name and handle valid, so a long-running application can report periodically without re-resolving anything:

sp_region_stats stats;
sp_profiler_get_region_stats(p, solve, &stats);
printf("solve: %lld calls, %lld ns total\n",
       (long long)stats.calls, (long long)stats.total_ns);

sp_profiler_reset(p);   /* refuses with SP_ERR_OPEN_SCOPES if solve is still open */

sp_profiler_flush(p) writes everything recorded so far to the trace file without stopping profiling - a call still open is simply left out until a later flush or sp_profiler_finalize() includes it.

Errors#

Every explicit-context function accepts a NULL or inactive profiler harmlessly, and records what went wrong on the profiler itself:

if (sp_profiler_last_error(p) != SP_OK) {
    fprintf(stderr, "scope_profiler: %s\n",
            sp_error_string(sp_profiler_last_error(p)));
}

sp_profiler_output_path(p) returns the path sp_profiler_finalize() / sp_profiler_flush() write (or wrote), so callers do not need to reconstruct <prefix>_rank<NNNNN>.h5 — or .spt — themselves.

Output format#

sp_finalize() writes one file per rank, in one of two formats.

HDF5 — preferred. <prefix>_rank<NNNNN>.h5 is an ordinary scope-profiler profile, byte-layout-identical to what a Python run writes, so there is nothing to convert:

./simulation
scope-profiler inspect profile_rank00000.h5
scope-profiler plot default profile_rank00000.h5 -o figures

It is opt-in at compile time, because it is the only thing that gives the library a dependency. Define SP_USE_HDF5 and link libhdf5:

cc -std=c99 -O2 -DSP_USE_HDF5 -c scope_profiler.c $(pkg-config --cflags hdf5)
cc -O2 main.o scope_profiler.o $(pkg-config --libs hdf5) -o simulation

or, under CMake, -DSCOPE_PROFILER_ENABLE_HDF5=ON, which puts the definition and the link on the scope-profiler::native and scope-profiler::header-only targets for you. Serial HDF5 is enough — each rank writes its own file, so nothing here needs parallel HDF5 or MPI.

Trace — the fallback. Without SP_USE_HDF5 the program writes <prefix>_rank<NNNNN>.spt, the compact binary trace shared with the Fortran API, and scope-profiler import-native turns it into the same HDF5 output afterwards. It needs nothing but libc, which is what makes it the right choice where the profiled machine has no HDF5 to link, or where the run must not carry the library at all.

A build with HDF5 defaults to it, so most programs never touch the format API. To pin one explicitly:

sp_init("profile", rank);
sp_set_output_format(SP_OUTPUT_TRACE);   /* or SP_OUTPUT_HDF5 */
printf("writing %s\n", sp_output_path());

sp_hdf5_available() reports whether this build has the HDF5 backend at all. Asking a build without it for SP_OUTPUT_HDF5 returns SP_ERR_UNSUPPORTED and leaves the format alone, so instrumentation that always asks for HDF5 still runs — and still produces a trace — where it is not available.

Both formats record exactly the same measurements. The difference is what the file already is: an HDF5 profile carries the region names, source locations, per-event timestamps and the fixed-size summary statistics a reader needs, while a trace carries the names, source locations and timestamps and lets the import derive the rest.

Under MPI#

Pass each rank’s id so the output files do not collide:

int rank;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
sp_init("profile", rank);

Every rank writes its own file; nothing is communicated, so sp_finalize() is not collective and a rank that dies takes only its own output with it. Merging the ranks into one profile is a post-processing step, in either format:

mpirun -n 128 ./simulation
scope-profiler import-native . -o profiling_data.h5
scope-profiler plot default profiling_data.h5 -o figures

import-native reads both, and mixes them: it picks up *.spt traces and per-rank *_rank<NNNNN>.h5 profiles from a directory and merges every rank it finds into one file. A merged profile sitting in the same directory is left alone — only the per-rank naming is treated as input — so re-running the import does not fold a previous result back into the next one.

With one rank there is nothing to merge: an HDF5 build’s output file is the profile, and import-native is only worth running to rename it.

Python calling C#

The case this is really for. Compile the kernels and the profiler into one shared library and load it with ctypes — no binding generator, no build backend:

cc -std=c99 -O2 -fPIC -shared kernels.c $(python -c \
    "import scope_profiler.native_trace as t; print(t.c_source_path())") \
   -I$(python -c "import scope_profiler.native_trace as t; print(t.c_include_dir())") \
   -o libkernels.so

import ctypes
from scope_profiler import ProfileManager

kernels = ctypes.CDLL("./libkernels.so")
kernels.kernels_jacobi_solve.argtypes = [ctypes.c_int, ctypes.c_int]
kernels.kernels_jacobi_solve.restype = ctypes.c_double   # see the trap below

ProfileManager.setup(file_path="profiling_data.h5")
kernels.kernels_start_profiling(b"trace", 0)

for step in range(nsteps):
    with ProfileManager.region("python:step"):
        kernels.kernels_jacobi_solve(n, sweeps)          # records its own regions

kernels.kernels_stop_profiling()                          # writes the C output
ProfileManager.finalize(native_traces=".")                # ...and folds it in

One file comes out, profiling_data.h5, holding both languages — there is no merge step here, and none of this changes with the C side’s output format: native_traces= reads whichever of the two it wrote. The C side’s own file (trace_rank00000.h5 or .spt) is an intermediate, left beside the result.

Because both APIs read the same clock, the C regions nest inside the Python region that called them and call_stack() sees a single tree:

  python:setup                   11.970 ms   [python]
  python:timestep                 0.331 ms   [python]
    python:call_solver              0.320 ms   [python]
      c:stencil                       0.023 ms   [c]
      c:residual                      0.040 ms   [c]

Two rules:

  • Call the C sp_finalize() first. Its output file has to exist by the time finalize() reads it.

  • Give the two sides distinct region names. A name recorded by both raises, rather than silently double-counting a wrapper and the region inside it. A python: / c: prefix is the simplest convention.

One ctypes trap#

ctypes assumes a function returns int unless told otherwise, so a double return comes back as garbage without restype. The regions are still recorded correctly, which makes it easy to miss. Declare argtypes and restype for everything you call.

One timeline with Python#

The API reads the same OS clock CPython’s time.perf_counter_ns() uses — CLOCK_MONOTONIC on Linux, CLOCK_UPTIME_RAW on macOS — resolved by probing at run time.

The macOS case is worth knowing about if you ever vendor this file: defining _POSIX_C_SOURCE there hides CLOCK_UPTIME_RAW, and the fallback CLOCK_MONOTONIC is both microsecond-granular and on a different epoch. The implementation therefore defines _DARWIN_C_SOURCE on Apple platforms and _POSIX_C_SOURCE elsewhere — for opposite reasons on each.

The trace format#

This is the .spt fallback format; a build with HDF5 writes the schema described in HDF5 output & post-processing from Python instead. Compatible with the Fortran API’s; see Profiling Fortran code for the shared layout. 16 bytes per recorded call, the same as the Python side. The C API writes format version 2, adding an optional source file and line per region (empty when a region was registered with sp_region() rather than sp_region_at()). The Fortran API also writes version 2. Legacy version-1 traces have no source fields. Both versions remain readable from the same trace directory, so a mixed C/Fortran run still merges into one profile.

Limitations#

  • Not thread safe. A region must be entered and left by the same thread. OpenMP threads inside a region are fine — wrap the whole parallel construct in one sp_begin/sp_end from the master thread.

  • Counter storage is owned by LIKWID. Native C++ scopes write the ordinary timing output and LIKWID writes its marker output; neither native format carries counters.

  • The HDF5 output is timing-only, like the trace: no GPU durations, no line profile, no per-call parent ids. Post-processing reconstructs the call graph and exclusive times from the recorded intervals, exactly as it does for an imported trace.

  • Recursion depth per region is capped at SP_MAX_DEPTH (64); deeper nesting is reported on stderr and left untimed rather than corrupting the buffer.

  • A region still open at sp_finalize() is reported on stderr and dropped, rather than written with a missing end time.

Contents