Batch Systems

🪵 How to use different known batch systems for scalable job execution.

Currently Isabl supports out-of-the-box the following Batch Systems to submit jobs with isabl_cli:

Batch System	Import string	Resources
LSF	isabl_cli.batch_systems.submit_lsf	See more IBM Spectrum LSF Reference
Slurm	isabl_cli.batch_systems.submit_slurm	See more Slurm - Worload Manager
SGE	isabl_cli.batch_systems.submit_sge	See more Sun Grid Engine - queueing system

By default, all submissions are run locally using isabl_cli.batch_systems.submit_local

What about other systems?

To support other systems or to customize your own submission steps, you may need just to create your own method and define it on your isabl_cli's SUBMIT_ANALYSES setting.

Example:

# my_isabl_extension/batch_systems.py
def submit_aws_batch(app, command_tuples):
    ...

// Isabl CLI settings:
{..., "SUBMIT_ANALYSES": "my_isabl_extension.batch_systems.submit_aws_batch"}

For help, creating your custom submit method, follow direction from the existing ones.

Please consider contributing any new one to the Isabl Project! 💌

Submit Configuration

isabl_cli's settings have a SUBMIT_CONFIGURATION dictionary to provide extra arguments to the submission methods. Current parameters used are:

Attribute	Type	Description	Example value
extra_args	String	Any additional extra arguments passed to the batch submission command.	" --time=48:00:00 ". Add a maximum job array time.
get_requirements	Import String	To define custom resources needs for different applications or methods.	See the example above.
throttle_by	Integer	To control the maximum number of jobs running simultaneously in a job array submission.	50
unbuffer	Boolean	redirect stdout and stderr to same file with ascii characters that allow colored logs.	True. If not defined, it is False by default. See Colored Logs.

Example Settings:

// isabl cli's settings
{
    ...
    "SUBMIT_ANALYSES": "isabl_cli.batch_systems.submit_slurm",
    "SUBMIT_CONFIGURATION": {
        "extra_args": " -p GPU_PARTITION ",
        "get_requirements": "my_isabl_extension.batch_systems.get_slurm_requirements",
        "throttle_by": 50,
        "unbuffer": True,
    },
    "SUBMIT_MERGE_ANALYSIS": "my_isabl_extension.batch_systems.submit_merge_analysis_to_slurm",
    ...
}

# my_isabl_extension/batch_systems.py

from my_isabl_extension import apps


def get_slurm_requirements(app, targets_methods):
    """Get custom requirements for any app or method."""
    
    # Defaults
    runtime = 240 # 6 hours
    cores = 1
    memory = 6
    
    # Technique.method-specific    
    method = targets_methods[0]
    if method == "WG":
        runtime = 10080 # 7 days

    # Application-specific
    if isinstance(app, apps.BwaMemGRCh37):
        cores = 32
        runtime = 1440 # 24 hours
        memory = 8

    return (
        f"--ntasks=1 "
        f"--cpus-per-task={cores} "
        f"--time={runtime} "
        f"--mem-per-cpu={memory}G "
    )


def submit_merge_analysis_to_slurm(instance, application, command):
    """Submit command to run merge in slurm."""

    command = f"sbatch --ntasks=1 --cpus-per-task=1 --mem=32G {command}"
    subprocess.check_call(command.split())
    click.secho(f"Submitted project level merge with: {command}", fg="yellow")