# scope_profiler.benchmark

*module*

Repeatable benchmark workflows for coding agents and CI.

The benchmark format is TOML so it works with Python 3.10+ without adding a
configuration dependency.  A benchmark run produces a JSON manifest containing
the individual profile paths, robust summary statistics, and correctness status.

[View source](https://github.com/max-models/scope-profiler/blob/devel/src/scope_profiler/benchmark.py#L1-L1)

## scope_profiler.benchmark.BenchmarkConfig

*class* · *dataclass*

```python
class BenchmarkConfig
```

[View source](https://github.com/max-models/scope-profiler/blob/devel/src/scope_profiler/benchmark.py#L31-L43)

### scope_profiler.benchmark.BenchmarkConfig.args

*attribute* · *class attribute* · *instance attribute*

```python
args: tuple[str, ...] = ()
```

[View source](https://github.com/max-models/scope-profiler/blob/devel/src/scope_profiler/benchmark.py#L35-L35)

### scope_profiler.benchmark.BenchmarkConfig.correctness_command

*attribute* · *class attribute* · *instance attribute*

```python
correctness_command: tuple[str, ...] = ()
```

[View source](https://github.com/max-models/scope-profiler/blob/devel/src/scope_profiler/benchmark.py#L41-L41)

### scope_profiler.benchmark.BenchmarkConfig.correctness_timeout_seconds

*attribute* · *class attribute* · *instance attribute*

```python
correctness_timeout_seconds: float = 300.0
```

[View source](https://github.com/max-models/scope-profiler/blob/devel/src/scope_profiler/benchmark.py#L42-L42)

### scope_profiler.benchmark.BenchmarkConfig.name

*attribute* · *instance attribute*

```python
name: str
```

[View source](https://github.com/max-models/scope-profiler/blob/devel/src/scope_profiler/benchmark.py#L33-L33)

### scope_profiler.benchmark.BenchmarkConfig.only_user_code

*attribute* · *class attribute* · *instance attribute*

```python
only_user_code: bool = True
```

[View source](https://github.com/max-models/scope-profiler/blob/devel/src/scope_profiler/benchmark.py#L39-L39)

### scope_profiler.benchmark.BenchmarkConfig.output_dir

*attribute* · *class attribute* · *instance attribute*

```python
output_dir: str = '.scope-profiler'
```

[View source](https://github.com/max-models/scope-profiler/blob/devel/src/scope_profiler/benchmark.py#L40-L40)

### scope_profiler.benchmark.BenchmarkConfig.runs

*attribute* · *class attribute* · *instance attribute*

```python
runs: int = 5
```

[View source](https://github.com/max-models/scope-profiler/blob/devel/src/scope_profiler/benchmark.py#L36-L36)

### scope_profiler.benchmark.BenchmarkConfig.script

*attribute* · *instance attribute*

```python
script: str
```

[View source](https://github.com/max-models/scope-profiler/blob/devel/src/scope_profiler/benchmark.py#L34-L34)

### scope_profiler.benchmark.BenchmarkConfig.threshold_pct

*attribute* · *class attribute* · *instance attribute*

```python
threshold_pct: float = 2.0
```

[View source](https://github.com/max-models/scope-profiler/blob/devel/src/scope_profiler/benchmark.py#L43-L43)

### scope_profiler.benchmark.BenchmarkConfig.timeout_seconds

*attribute* · *class attribute* · *instance attribute*

```python
timeout_seconds: float = 300.0
```

[View source](https://github.com/max-models/scope-profiler/blob/devel/src/scope_profiler/benchmark.py#L38-L38)

### scope_profiler.benchmark.BenchmarkConfig.warmups

*attribute* · *class attribute* · *instance attribute*

```python
warmups: int = 1
```

[View source](https://github.com/max-models/scope-profiler/blob/devel/src/scope_profiler/benchmark.py#L37-L37)

## scope_profiler.benchmark.BenchmarkError

*class*

```python
class BenchmarkError(Exception)
```

Bases: `Exception`

A user-facing benchmark configuration or execution error.

[View source](https://github.com/max-models/scope-profiler/blob/devel/src/scope_profiler/benchmark.py#L27-L28)

## scope_profiler.benchmark.compare_benchmarks

*function*

```python
def compare_benchmarks(baseline: dict | str, candidate: dict | str) -> dict
```

Compare benchmark manifests using medians and report an agent decision.

[View source](https://github.com/max-models/scope-profiler/blob/devel/src/scope_profiler/benchmark.py#L211-L238)

## scope_profiler.benchmark.load_config

*function*

```python
def load_config(path: str | os.PathLike[str]) -> BenchmarkConfig
```

Load and validate a benchmark TOML file.

[View source](https://github.com/max-models/scope-profiler/blob/devel/src/scope_profiler/benchmark.py#L46-L94)

## scope_profiler.benchmark.run_benchmark

*function*

```python
def run_benchmark(config: BenchmarkConfig, label: str = 'candidate') -> dict
```

Run warmups, correctness, and repeated measured profiles.

[View source](https://github.com/max-models/scope-profiler/blob/devel/src/scope_profiler/benchmark.py#L185-L208)