scope_profiler.benchmark
Repeatable benchmark workflows for coding agents and CI.
The benchmark format is TOML so it works with Python 3.10+ without adding a configuration dependency. A benchmark run produces a JSON manifest containing the individual profile paths, robust summary statistics, and correctness status.
Classes
| Name | Description |
|---|---|
BenchmarkConfig | No description. |
BenchmarkError | A user-facing benchmark configuration or execution error. |
Functions
| Name | Description |
|---|---|
compare_benchmarks | Compare benchmark manifests using medians and report an agent decision. |
load_config | Load and validate a benchmark TOML file. |
run_benchmark | Run warmups, correctness, and repeated measured profiles. |
BenchmarkConfigclassdataclass#
class BenchmarkConfig(name: str, script: str, args: tuple[str, ...] = (), runs: int = 5, warmups: int = 1, timeout_seconds: float = 300.0, only_user_code: bool = True, output_dir: str = '.scope-profiler', correctness_command: tuple[str, ...] = (), correctness_timeout_seconds: float = 300.0, threshold_pct: float = 2.0)argsattributeclass attributeinstance attribute#
args: tuple[str, ...] = ()correctness_commandattributeclass attributeinstance attribute#
correctness_command: tuple[str, ...] = ()correctness_timeout_secondsattributeclass attributeinstance attribute#
correctness_timeout_seconds: float = 300.0nameattributeinstance attribute#
name: stronly_user_codeattributeclass attributeinstance attribute#
only_user_code: bool = Trueoutput_dirattributeclass attributeinstance attribute#
output_dir: str = '.scope-profiler'runsattributeclass attributeinstance attribute#
runs: int = 5scriptattributeinstance attribute#
script: strthreshold_pctattributeclass attributeinstance attribute#
threshold_pct: float = 2.0timeout_secondsattributeclass attributeinstance attribute#
timeout_seconds: float = 300.0warmupsattributeclass attributeinstance attribute#
warmups: int = 1BenchmarkErrorclass#
class BenchmarkError(Exception)Bases: Exception
A user-facing benchmark configuration or execution error.
compare_benchmarksfunction#
def compare_benchmarks(baseline: dict | str, candidate: dict | str) -> dictCompare benchmark manifests using medians and report an agent decision.
load_configfunction#
def load_config(path: str | os.PathLike[str]) -> BenchmarkConfigLoad and validate a benchmark TOML file.
run_benchmarkfunction#
def run_benchmark(config: BenchmarkConfig, label: str = 'candidate') -> dictRun warmups, correctness, and repeated measured profiles.