slurm_script_generator

slurm_script_generator#

Public package exports for slurm_script_generator.

class slurm_script_generator.SAcct(user: str | None = None, days: int = 7, partition: str | None = None, me: bool = False)[source]#

Bases: object

Interface to SLURM job accounting via sacct.

Parameters:

user (str, optional) – If given, fetch only jobs for this user.
days (int) – Number of days of history to look back (default: 7).
partition (str, optional) – If given, filter to this partition.
me (bool) – If True, fetch only jobs belonging to the current OS user. Mutually exclusive with user. Defaults to False.

Examples

>>> a = SAcct(user='alice', days=30)
>>> a.summary()
{'total': 42, 'completed': 30, 'failed': 5, ...}

jobs(user: str | None = None, state: str | None = None, partition: str | None = None) → List[SAcctJob][source]#: Return accounting records matching the given criteria.

jobs_by_partition() → Dict[str, List[SAcctJob]][source]#: Return a mapping of partition -> list of jobs in that partition.

jobs_by_state() → Dict[str, List[SAcctJob]][source]#: Return a mapping of state -> list of jobs in that state.

jobs_by_user() → Dict[str, List[SAcctJob]][source]#: Return a mapping of username -> list of their historical jobs.

refresh() → SAcct[source]#: Re-run sacct and update the cached job list.

summary() → dict[source]#

Return a summary dict of job counts and CPU usage.

Returns:: Keys: total, completed, failed, cancelled, timeout, cpu_hours, by_state, users.
Return type:: dict

class slurm_script_generator.SAcctJob(job_id: int, user: str, name: str, state: str, partition: str, num_nodes: int, num_cpus: int, elapsed: str, cpu_time_raw: int, exit_code: str)[source]#

Bases: object

A single job record from SLURM accounting (sacct).

property cpu_hours: float#

cpu_time_raw: int#

elapsed: str#

exit_code: str#

property is_cancelled: bool#

property is_completed: bool#

property is_failed: bool#

property is_timeout: bool#

job_id: int#

name: str#

num_cpus: int#

num_nodes: int#

partition: str#

state: str#

user: str#

class slurm_script_generator.SQueue(user: str | None = None, partition: str | None = None, me: bool = False)[source]#

Bases: object

Interface to the SLURM job queue via squeue.

Parameters:

user (str, optional) – If given, only fetch jobs belonging to this user by default.
me (bool) – If True, fetch only jobs belonging to the current OS user by default. Mutually exclusive with user. Defaults to False.

Examples

>>> q = SQueue()
>>> q.summary()
{'total_jobs': 42, 'running': 30, 'pending': 12, 'users': {...}, 'by_state': {...}}

>>> q.wait_until_done(job_name='training_*')
>>> q.wait_until_done(job_id=12345)
>>> q.wait_until_done(job_id=[12345, 12346])
>>> q.wait_until_done(user='alice')

>>> q.cancel(job_id=12345)
>>> q.cancel(job_name='training_*')

Cancel all matching jobs with scancel.

Supports glob patterns in job_name (* and ? wildcards). At least one filter argument must be provided, so that an accidental call cannot cancel the whole queue.

Parameters:

job_name (str, optional) – Job name or glob pattern, e.g. 'train_*'.
job_id (int, str, or list of int/str, optional) – A specific job ID, or a list of job IDs, to cancel.
user (str, optional) – Cancel all jobs belonging to this user.
me (bool) – Cancel all jobs belonging to the current OS user. Mutually exclusive with user. Defaults to False.
state (str, optional) – SLURM state code, e.g. 'PD' to cancel only pending jobs.
partition (str, optional) – Partition name to filter by.
verbose (bool) – Print progress messages. Defaults to True.

Returns:

The job IDs that were passed to scancel.

Return type:

list of int

Raises:

ValueError – If no filter is specified.
RuntimeError – If scancel exits with a non-zero status.

Examples

>>> q = SQueue()
>>> q.cancel(job_id=12345)
>>> q.cancel(job_name='train_*')
>>> q.cancel(user='alice', state='PD')

Return jobs matching the given criteria.

Parameters:

job_name (str, optional) – Job name or glob pattern (e.g. 'train_*').
job_id (int, str, or list of int/str, optional) – Exact job ID, or a list of job IDs.
user (str, optional) – Username to filter by.
state (str, optional) – SLURM state code, e.g. 'R' or 'PD'.
partition (str, optional) – Partition name to filter by.
me (bool) – Filter to jobs belonging to the current OS user. Mutually exclusive with user. Defaults to False.

Return type:

list of SQueueJob

jobs_by_partition() → Dict[str, List[SQueueJob]][source]#: Return a mapping of partition name -> list of jobs in that partition.

jobs_by_state() → Dict[str, List[SQueueJob]][source]#: Return a mapping of state code -> list of jobs in that state.

jobs_by_user() → Dict[str, List[SQueueJob]][source]#: Return a mapping of username -> list of their jobs.

pending_jobs() → List[SQueueJob][source]#: Return all jobs currently in the PD (Pending) state.

refresh() → SQueue[source]#

Re-run squeue and update the cached job list.

Returns:: self, for chaining.
Return type:: SQueue

running_jobs() → List[SQueueJob][source]#: Return all jobs currently in the R (Running) state.

summary() → dict[source]#

Return a summary dict with total counts, per-user counts, and per-state counts.

Returns:: Keys: total_jobs, running, pending, users (dict of user -> job count), by_state (dict of state code -> job count).
Return type:: dict

users() → List[str][source]#: Return a sorted list of unique users with jobs in the queue.

Block until all matching jobs leave the active queue.

Supports glob patterns in job_name (* and ? wildcards). At least one filter argument must be provided.

Once the jobs are gone the final states are looked up with a single sacct call (see job_states()), so the caller learns whether the jobs it waited for actually succeeded.

Parameters:

job_name (str, optional) – Job name or glob pattern, e.g. 'train_*'.
job_id (int, str, or list of int/str, optional) – A specific job ID, or a list of job IDs, to wait for.
user (str, optional) – Wait for all jobs belonging to this user to finish.
me (bool) – Wait for all jobs belonging to the current OS user. Mutually exclusive with user. Defaults to False.
poll_interval (float) – Seconds between queue polls. Defaults to 30.
timeout (float, optional) – Maximum seconds to wait before raising TimeoutError.
verbose (bool) – Print progress messages. Defaults to True.
check (bool) – Raise RuntimeError if any job ended in a state from FAILED_JOB_STATES. Jobs whose state is undetermined (None) never trigger this. Defaults to False.

Returns:

Final accounting state per job ID that was waited on. A None value means the state could not be determined, not that the job failed.

Return type:

dict of int -> (str or None)

Raises:

ValueError – If no filter is specified.
TimeoutError – If timeout is exceeded before all jobs finish.
RuntimeError – If check is True and a job ended in a failure state.

Examples

>>> q.wait_until_done(job_id=12345)
{12345: 'COMPLETED'}
>>> q.wait_until_done(job_name='train_*', check=True)
{12345: 'COMPLETED', 12346: 'COMPLETED'}

class slurm_script_generator.SQueueJob(job_id: int, user: str, name: str, state: str, partition: str, num_nodes: int, num_cpus: int, time_used: str, time_limit: str, reason: str, priority: int, submit_time: str = '')[source]#

Bases: object

A single job entry from the SLURM queue.

cancel(verbose: bool = True) → None[source]#

Cancel this specific job with scancel.

Parameters:: verbose (bool) – Print a confirmation message. Defaults to True.

final_state() → str | None[source]#

Return this job’s accounting state via sacct.

See job_state(). Returns None when the state cannot be determined; that is not evidence of failure.

property is_active: bool#

property is_pending: bool#

property is_running: bool#

job_id: int#

name: str#

num_cpus: int#

num_nodes: int#

partition: str#

priority: int#

reason: str#

state: str#

property state_name: str#

property submit_datetime: datetime | None#: The job’s submission time, or None if squeue didn’t report one.

submit_time: str = ''#

time_limit: str#

time_used: str#

user: str#

wait_until_done(poll_interval: float = 30.0, timeout: float | None = None, verbose: bool = True, check: bool = False) → str | None[source]#

Block until this specific job leaves the active queue.

Parameters:

poll_interval (float) – Seconds between queue polls. Defaults to 30.
timeout (float, optional) – Maximum seconds to wait before raising TimeoutError.
verbose (bool) – Print progress messages. Defaults to True.
check (bool) – Raise RuntimeError if the job ends in a failure state. Defaults to False.

Returns:

The job’s final accounting state, or None if undetermined.

Return type:

str or None

property waiting_seconds: float | None#

Seconds elapsed since submission — how long a pending job has waited.

None when the submission time could not be determined.

Bases: object

Class representing a Slurm batch script with pragmas, modules, and custom commands.

Parameters:

account (str, optional) – The account to charge for the job.
array (str, optional) – The job array specification.
begin (str, optional) – The time to begin the job.
bell (str, optional) – Ring terminal bell when job is allocated.
burst_buffer (str, optional) – Burst buffer specifications.
bb_file (str, optional) – Burst buffer specification file.
cpus_per_task (int, optional) – Number of CPUs required per task.
comment (str, optional) – Arbitrary comment for the job.
container (str, optional) – Path to OCI container bundle.
container_id (str, optional) – OCI container ID.
cpu_freq (str, optional) – Requested CPU frequency and governor.
delay_boot (str, optional) – Delay boot for desired node features.
dependency (str, optional) – Job dependency specification.
deadline (str, optional) – Remove job if no ending possible before deadline.
chdir (str, optional) – Change working directory for the job.
get_user_env (str, optional) – Used by Moab for environment setup.
gres (str, optional) – Required generic resources.
gres_flags (str, optional) – Flags related to GRES management.
hold (str, optional) – Submit job in held state.
immediate (str, optional) – Exit if resources not available within seconds.
job_name (str, optional) – Name of the job.
no_kill (str, optional) – Do not kill job on node failure.
kill_command (str, optional) – Signal to send terminating job.
licenses (str, optional) – Required licenses, comma separated.
clusters (str, optional) – Comma separated list of clusters.
distribution (str, optional) – Distribution method for processes.
mail_type (str, optional) – Notify on state change.
mail_user (str, optional) – Email for job state changes.
mcs_label (str, optional) – MCS label if mcs plugin is used.
ntasks (str, optional) – Number of processors required.
nice (str, optional) – Decrease scheduling priority by value.
nodes (int, optional) – Number of nodes to allocate.
ntasks_per_node (int, optional) – Number of tasks to invoke on each node.
oom_kill_step (str, optional) – Set OOMKillStep behaviour.
overcommit (str, optional) – Overcommit resources.
power (str, optional) – Power management options.
priority (str, optional) – Set job priority.
profile (str, optional) – Enable acct_gather_profile for detailed data.
partition (str, optional) – Partition requested.
qos (str, optional) – Quality of service.
quiet (str, optional) – Suppress informational messages.
reboot (str, optional) – Reboot compute nodes before starting job.
oversubscribe (str, optional) – Oversubscribe resources with other jobs.
signal (str, optional) – Send signal when time limit within seconds.
spread_job (str, optional) – Spread job across as many nodes as possible.
error (str, optional) – Redirect stderr to file.
output (str, optional) – Redirect stdout to file.
switches (str, optional) – Optimum switches and max wait time.
core_spec (str, optional) – Count of reserved cores.
thread_spec (str, optional) – Count of reserved threads.
time (str, optional) – Time limit for the job.
time_min (str, optional) – Minimum time limit.
tres_bind (str, optional) – Task to TRES binding options.
tres_per_task (str, optional) – TRES required per task.
use_min_nodes (str, optional) – Prefer smaller node count.
wckey (str, optional) – Wckey to run job under.
cluster_constraint (str, optional) – List of cluster constraints.
contiguous (str, optional) – Demand contiguous range of nodes.
constraint (str, optional) – List of constraints.
nodefile (str, optional) – Request specific list of hosts from file.
mem (str, optional) – Minimum real memory required.
mincpus (str, optional) – Minimum logical processors per node.
reservation (str, optional) – Allocate resources from named reservation.
tmp (str, optional) – Minimum temporary disk required.
nodelist (str, optional) – Request specific list of hosts.
exclude (str, optional) – Exclude specific list of hosts.
exclusive_user (str, optional) – Allocate nodes in exclusive mode.
exclusive_mcs (str, optional) – Exclusive mode when mcs plugin enabled.
mem_per_cpu (str, optional) – Real memory per allocated CPU.
resv_ports (str, optional) – Reserve communication ports.
sockets_per_node (int, optional) – Number of sockets per node to allocate.
cores_per_socket (int, optional) – Number of cores per socket to allocate.
threads_per_core (int, optional) – Number of threads per core to allocate.
extra_node_info (str, optional) – Combine sockets, cores, threads.
ntasks_per_core (int, optional) – Number of tasks per core.
ntasks_per_socket (int, optional) – Number of tasks per socket.
hint (str, optional) – Application binding hints.
mem_bind (str, optional) – Bind memory to locality domains.
cpus_per_gpu (int, optional) – Number of CPUs required per allocated GPU.
gpus (str, optional) – Count of GPUs required.
gpu_bind (str, optional) – Task to GPU binding options.
gpu_freq (str, optional) – Frequency and voltage of GPUs.
gpus_per_node (str, optional) – GPUs per allocated node.
gpus_per_socket (str, optional) – GPUs per allocated socket.
gpus_per_task (str, optional) – GPUs per spawned task.
mem_per_gpu (str, optional) – Real memory per allocated GPU.
disable_output_job_summary (str, optional) – Disable job summary in output file.
nvmps (str, optional) – Launch NVIDIA MPS for job.
pragmas (List[Pragma], optional) – List of pragmas to add to the script.
modules (List[str], optional) – List of modules to load in the script.
custom_command (str, optional) – Custom command to run in the script.
custom_commands (list, optional) – List of custom commands to run in the script.
inlined_script (str, optional) – Inline script to include in the batch script.
inlined_scripts (list, optional) – List of inline scripts to include in the batch script.
line_length (int, optional) – Line length for formatting output.

add_custom_command(command: str) → None[source]#

Add a single custom command to the script.

Parameters:: command (str) – The custom command to add.

add_custom_commands(commands: List[str] | None) → None[source]#

Add multiple custom commands to the script.

Parameters:: commands (list of str, optional) – List of custom commands to add.

add_inlined_script(path: str) → None[source]#

Add lines from an inlined script file to the custom commands.

Parameters:: path (str) – Path to the script file to inline.

add_inlined_scripts(paths: List[str] | None) → None[source]#

Add lines from multiple inlined script files to the custom commands.

Parameters:: paths (list of str, optional) – List of script file paths to inline.

add_module(module: str) → None[source]#

Add a single module to the script.

Parameters:: module (str) – The module to add.

add_modules(modules: List[str] | None) → None[source]#

Add multiple modules to the script.

Parameters:: modules (list of str, optional) – List of modules to add.

add_param(key: str, value: Any) → None[source]#

Add a non-pragma parameter to the script.

Parameters:

key (str) – The parameter key.
value (Any) – The parameter value.

add_pragma(pragma: Pragma) → None[source]#

Add a Pragma object to the script, replacing any existing pragma with the same destination.

Parameters:: pragma (Pragma) – The Pragma object to add.

add_pragmas(pragmas: List[Pragma] | None) → None[source]#

Add multiple Pragma objects to the script.

Parameters:: pragmas (list of Pragma, optional) – List of Pragma objects to add.

check() → None[source]#: Raise ValueError if validate() finds any problems.

property custom_commands: List[str]#

Get the list of custom commands to run in the script.

Returns:: List of custom command strings.
Return type:: List[str]

static from_dict(data: dict[str, Any]) → SlurmScript[source]#

Create a SlurmScript instance from a dictionary.

Parameters:: data (dict[str, Any]) – Dictionary containing the SlurmScript data.
Returns:: The constructed SlurmScript object.
Return type:: SlurmScript

static from_json(path: str) → SlurmScript[source]#

Load a SlurmScript instance from a JSON file.

Parameters:: path (str) – Path to the JSON file to load.
Returns:: The constructed SlurmScript object.
Return type:: SlurmScript

static from_script(script: str, verbose: bool = False) → SlurmScript[source]#

Parse a SLURM script string and create a SlurmScript instance.

Parameters:

script (str) – SLURM script content.
verbose (bool :) – (Default value = False)

Returns:

The constructed SlurmScript object.

Return type:

SlurmScript

generate_script(line_length: int = 54, include_header: bool = False) → str[source]#

Parameters:

line_length – int: (Default value = 54)
include_header – bool: (Default value = False)

property line_length: int#

Get the maximum line length for the script, used for formatting the output. This value is used to determine how many characters fit on a line when generating the script string, and is also used for formatting the header and section separators.

Returns:: The line length value.
Return type:: int

property modules: List[str]#

Get the list of modules to load in the script.

Returns:: List of module names.
Return type:: List[str]

property pragmas: List[Pragma]#

Get the list of Pragma objects in the script.

Returns:: List of all Pragma instances.
Return type:: List[Pragma]

static read_script(path: str, verbose: bool = False) → SlurmScript[source]#

Read a SLURM script from a file and parse it into a SlurmScript instance.

Parameters:

path (str) – Path to the script file.
verbose (bool) – Whether to enable verbose output. (Default value = False)

Returns:

The parsed SlurmScript object.

Return type:

SlurmScript

save(path: str | Path, include_header: bool = True, verbose: bool = False) → None[source]#

Save the generated SLURM script to a file.

Parameters:

path (str) – Path to save the script file.
include_header (bool) – Whether to include the script header.
verbose (bool) – Whether to enable verbose output. (Default value = False)

submit_job(path: str, verbose: bool = False) → int[source]#

Submit the SLURM script as a job using sbatch.

Parameters:

path (str) – Path to the script file to submit.

Raises:

ValueError – If validate() finds a combination of pragmas sbatch will reject (e.g. both --mem and --mem-per-cpu set).
RuntimeError – If sbatch fails to submit the job.

to_dict() → dict[str, Any][source]#

Convert the SlurmScript instance to a dictionary representation.

Returns:: Dictionary with keys ‘pragmas’, ‘modules’, and ‘custom_commands’.
Return type:: dict[str, Any]

to_json(path: str) → None[source]#

Save the SlurmScript instance as a JSON file.

Parameters:: path (str) – Path to save the JSON file.

to_string(include_header: bool = True) → str[source]#

Generate the SLURM script as a string.

Parameters:: include_header (bool) – Whether to include the header in the generated script. (Default value = True)
Returns:: The generated script string.
Return type:: str

validate() → List[str][source]#

Check for combinations of pragmas that sbatch will reject.

These are mistakes that only surface once a job is actually submitted (or silently misbehave), so it is worth catching them in Python first. This is not an exhaustive validator of sbatch’s rules, just the handful of cross-pragma mistakes that are easy to make.

Returns:: Human-readable problem descriptions. Empty if nothing was found.
Return type:: List[str]

slurm_script_generator.job_state(job_id: int | str, timeout: float = 30.0) → str | None[source]#

Return the state of a job from SLURM accounting (sacct).

squeue only says whether a job is still in the queue, not how it ended, so this is what distinguishes a crashed run from one that simply wrote no output. States are normalized, e.g. 'CANCELLED by 1234' -> 'CANCELLED'.

Parameters:

job_id (int or str) – The job ID to look up.
timeout (float) – Seconds to wait for sacct before giving up. Defaults to 30.

Returns:

The job state ('COMPLETED', 'FAILED', 'RUNNING', …), or None when it cannot be determined — no accounting configured, sacct missing or unresponsive, or the job not yet in the accounting database. A None result is not evidence of failure and callers should not report one.

Return type:

str or None

`clusters`	Per-cluster presets for `slurm-template`'s `--cluster` flag.
`main`
`pragmas`
`salloc`	Turn a batch script into an interactive allocation.
`slurm_script`
`slurm_template`	Generate a SLURM batch script from a built-in template.
`squeue`
`templates`	Built-in SLURM script templates for common job shapes.
`utils`

slurm_script_generator

Contents

slurm_script_generator#