slurm_script_generator.squeue

slurm_script_generator.squeue#

Functions

`current_user`()	Return the current OS username (as `whoami`/`$USER` would report it).
`job_state`(job_id[, timeout])	Return the state of a job from SLURM accounting (`sacct`).
`job_states`(job_ids[, timeout])	Return the states of several jobs from SLURM accounting (`sacct`).
`main`()	Entry point for the `slurm-queue` command-line tool.
`main_history`()	Entry point for the `slurm-history` command-line tool.
`main_stats`()	Entry point for the `slurm-stats` command-line tool.
`main_wait`()	Entry point for the `slurm-wait` command-line tool.

Classes

`SAcct`([user, days, partition, me])	Interface to SLURM job accounting via `sacct`.
`SAcctJob`(job_id, user, name, state, ...)	A single job record from SLURM accounting (`sacct`).
`SQueue`([user, partition, me])	Interface to the SLURM job queue via `squeue`.
`SQueueJob`(job_id, user, name, state, ...[, ...])	A single job entry from the SLURM queue.

class slurm_script_generator.squeue.SAcct(user: str | None = None, days: int = 7, partition: str | None = None, me: bool = False)[source]#

Bases: object

Interface to SLURM job accounting via sacct.

Parameters:

user (str, optional) – If given, fetch only jobs for this user.
days (int) – Number of days of history to look back (default: 7).
partition (str, optional) – If given, filter to this partition.
me (bool) – If True, fetch only jobs belonging to the current OS user. Mutually exclusive with user. Defaults to False.

Examples

>>> a = SAcct(user='alice', days=30)
>>> a.summary()
{'total': 42, 'completed': 30, 'failed': 5, ...}

jobs(user: str | None = None, state: str | None = None, partition: str | None = None) → List[SAcctJob][source]#: Return accounting records matching the given criteria.

jobs_by_partition() → Dict[str, List[SAcctJob]][source]#: Return a mapping of partition -> list of jobs in that partition.

jobs_by_state() → Dict[str, List[SAcctJob]][source]#: Return a mapping of state -> list of jobs in that state.

jobs_by_user() → Dict[str, List[SAcctJob]][source]#: Return a mapping of username -> list of their historical jobs.

refresh() → SAcct[source]#: Re-run sacct and update the cached job list.

summary() → dict[source]#

Return a summary dict of job counts and CPU usage.

Returns:: Keys: total, completed, failed, cancelled, timeout, cpu_hours, by_state, users.
Return type:: dict

class slurm_script_generator.squeue.SAcctJob(job_id: int, user: str, name: str, state: str, partition: str, num_nodes: int, num_cpus: int, elapsed: str, cpu_time_raw: int, exit_code: str)[source]#

Bases: object

A single job record from SLURM accounting (sacct).

property cpu_hours: float#

cpu_time_raw: int#

elapsed: str#

exit_code: str#

property is_cancelled: bool#

property is_completed: bool#

property is_failed: bool#

property is_timeout: bool#

job_id: int#

name: str#

num_cpus: int#

num_nodes: int#

partition: str#

state: str#

user: str#

class slurm_script_generator.squeue.SQueue(user: str | None = None, partition: str | None = None, me: bool = False)[source]#

Bases: object

Interface to the SLURM job queue via squeue.

Parameters:

user (str, optional) – If given, only fetch jobs belonging to this user by default.
me (bool) – If True, fetch only jobs belonging to the current OS user by default. Mutually exclusive with user. Defaults to False.

Examples

>>> q = SQueue()
>>> q.summary()
{'total_jobs': 42, 'running': 30, 'pending': 12, 'users': {...}, 'by_state': {...}}

>>> q.wait_until_done(job_name='training_*')
>>> q.wait_until_done(job_id=12345)
>>> q.wait_until_done(job_id=[12345, 12346])
>>> q.wait_until_done(user='alice')

>>> q.cancel(job_id=12345)
>>> q.cancel(job_name='training_*')

Cancel all matching jobs with scancel.

Supports glob patterns in job_name (* and ? wildcards). At least one filter argument must be provided, so that an accidental call cannot cancel the whole queue.

Parameters:

job_name (str, optional) – Job name or glob pattern, e.g. 'train_*'.
job_id (int, str, or list of int/str, optional) – A specific job ID, or a list of job IDs, to cancel.
user (str, optional) – Cancel all jobs belonging to this user.
me (bool) – Cancel all jobs belonging to the current OS user. Mutually exclusive with user. Defaults to False.
state (str, optional) – SLURM state code, e.g. 'PD' to cancel only pending jobs.
partition (str, optional) – Partition name to filter by.
verbose (bool) – Print progress messages. Defaults to True.

Returns:

The job IDs that were passed to scancel.

Return type:

list of int

Raises:

ValueError – If no filter is specified.
RuntimeError – If scancel exits with a non-zero status.

Examples

>>> q = SQueue()
>>> q.cancel(job_id=12345)
>>> q.cancel(job_name='train_*')
>>> q.cancel(user='alice', state='PD')

Return jobs matching the given criteria.

Parameters:

job_name (str, optional) – Job name or glob pattern (e.g. 'train_*').
job_id (int, str, or list of int/str, optional) – Exact job ID, or a list of job IDs.
user (str, optional) – Username to filter by.
state (str, optional) – SLURM state code, e.g. 'R' or 'PD'.
partition (str, optional) – Partition name to filter by.
me (bool) – Filter to jobs belonging to the current OS user. Mutually exclusive with user. Defaults to False.

Return type:

list of SQueueJob

jobs_by_partition() → Dict[str, List[SQueueJob]][source]#: Return a mapping of partition name -> list of jobs in that partition.

jobs_by_state() → Dict[str, List[SQueueJob]][source]#: Return a mapping of state code -> list of jobs in that state.

jobs_by_user() → Dict[str, List[SQueueJob]][source]#: Return a mapping of username -> list of their jobs.

pending_jobs() → List[SQueueJob][source]#: Return all jobs currently in the PD (Pending) state.

refresh() → SQueue[source]#

Re-run squeue and update the cached job list.

Returns:: self, for chaining.
Return type:: SQueue

running_jobs() → List[SQueueJob][source]#: Return all jobs currently in the R (Running) state.

summary() → dict[source]#

Return a summary dict with total counts, per-user counts, and per-state counts.

Returns:: Keys: total_jobs, running, pending, users (dict of user -> job count), by_state (dict of state code -> job count).
Return type:: dict

users() → List[str][source]#: Return a sorted list of unique users with jobs in the queue.

Block until all matching jobs leave the active queue.

Supports glob patterns in job_name (* and ? wildcards). At least one filter argument must be provided.

Once the jobs are gone the final states are looked up with a single sacct call (see job_states()), so the caller learns whether the jobs it waited for actually succeeded.

Parameters:

job_name (str, optional) – Job name or glob pattern, e.g. 'train_*'.
job_id (int, str, or list of int/str, optional) – A specific job ID, or a list of job IDs, to wait for.
user (str, optional) – Wait for all jobs belonging to this user to finish.
me (bool) – Wait for all jobs belonging to the current OS user. Mutually exclusive with user. Defaults to False.
poll_interval (float) – Seconds between queue polls. Defaults to 30.
timeout (float, optional) – Maximum seconds to wait before raising TimeoutError.
verbose (bool) – Print progress messages. Defaults to True.
check (bool) – Raise RuntimeError if any job ended in a state from FAILED_JOB_STATES. Jobs whose state is undetermined (None) never trigger this. Defaults to False.

Returns:

Final accounting state per job ID that was waited on. A None value means the state could not be determined, not that the job failed.

Return type:

dict of int -> (str or None)

Raises:

ValueError – If no filter is specified.
TimeoutError – If timeout is exceeded before all jobs finish.
RuntimeError – If check is True and a job ended in a failure state.

Examples

>>> q.wait_until_done(job_id=12345)
{12345: 'COMPLETED'}
>>> q.wait_until_done(job_name='train_*', check=True)
{12345: 'COMPLETED', 12346: 'COMPLETED'}

class slurm_script_generator.squeue.SQueueJob(job_id: int, user: str, name: str, state: str, partition: str, num_nodes: int, num_cpus: int, time_used: str, time_limit: str, reason: str, priority: int, submit_time: str = '')[source]#

Bases: object

A single job entry from the SLURM queue.

cancel(verbose: bool = True) → None[source]#

Cancel this specific job with scancel.

Parameters:: verbose (bool) – Print a confirmation message. Defaults to True.

final_state() → str | None[source]#

Return this job’s accounting state via sacct.

See job_state(). Returns None when the state cannot be determined; that is not evidence of failure.

property is_active: bool#

property is_pending: bool#

property is_running: bool#

job_id: int#

name: str#

num_cpus: int#

num_nodes: int#

partition: str#

priority: int#

reason: str#

state: str#

property state_name: str#

property submit_datetime: datetime | None#: The job’s submission time, or None if squeue didn’t report one.

submit_time: str = ''#

time_limit: str#

time_used: str#

user: str#

wait_until_done(poll_interval: float = 30.0, timeout: float | None = None, verbose: bool = True, check: bool = False) → str | None[source]#

Block until this specific job leaves the active queue.

Parameters:

poll_interval (float) – Seconds between queue polls. Defaults to 30.
timeout (float, optional) – Maximum seconds to wait before raising TimeoutError.
verbose (bool) – Print progress messages. Defaults to True.
check (bool) – Raise RuntimeError if the job ends in a failure state. Defaults to False.

Returns:

The job’s final accounting state, or None if undetermined.

Return type:

str or None

property waiting_seconds: float | None#

Seconds elapsed since submission — how long a pending job has waited.

None when the submission time could not be determined.

slurm_script_generator.squeue.current_user() → str[source]#: Return the current OS username (as whoami/$USER would report it).

slurm_script_generator.squeue.job_state(job_id: int | str, timeout: float = 30.0) → str | None[source]#

Return the state of a job from SLURM accounting (sacct).

squeue only says whether a job is still in the queue, not how it ended, so this is what distinguishes a crashed run from one that simply wrote no output. States are normalized, e.g. 'CANCELLED by 1234' -> 'CANCELLED'.

Parameters:

job_id (int or str) – The job ID to look up.
timeout (float) – Seconds to wait for sacct before giving up. Defaults to 30.

Returns:

The job state ('COMPLETED', 'FAILED', 'RUNNING', …), or None when it cannot be determined — no accounting configured, sacct missing or unresponsive, or the job not yet in the accounting database. A None result is not evidence of failure and callers should not report one.

Return type:

str or None

slurm_script_generator.squeue

Contents

slurm_script_generator.squeue#