Batch Systems

🪵 How to use different known batch systems for scalable job execution.

Currently Isabl supports out-of-the-box the following Batch Systems to submit jobs with isabl_cli:

Batch SystemImport stringResources

LSF

isabl_cli.batch_systems.submit_lsf

Slurm

isabl_cli.batch_systems.submit_slurm

SGE

isabl_cli.batch_systems.submit_sge

By default, all submissions are run locally using isabl_cli.batch_systems.submit_local

What about other systems?

To support other systems or to customize your own submission steps, you may need just to create your own method and define it on your isabl_cli's SUBMIT_ANALYSES setting.

Example:

# my_isabl_extension/batch_systems.py
def submit_aws_batch(app, command_tuples):
    ...
// Isabl CLI settings:
{..., "SUBMIT_ANALYSES": "my_isabl_extension.batch_systems.submit_aws_batch"}

For help, creating your custom submit method, follow direction from the existing ones.

Please consider contributing any new one to the Isabl Project! 💌

Submit Configuration

isabl_cli's settings have a SUBMIT_CONFIGURATION dictionary to provide extra arguments to the submission methods. Current parameters used are:

AttributeTypeDescriptionExample value

extra_args

String

Any additional extra arguments passed to the batch submission command.

" --time=48:00:00 ". Add a maximum job array time.

get_requirements

Import String

To define custom resources needs for different applications or methods.

See the example above.

throttle_by

Integer

To control the maximum number of jobs running simultaneously in a job array submission.

50

unbuffer

Boolean

redirect stdout and stderr to same file with ascii characters that allow colored logs.

True. If not defined, it is False by default. See Colored Logs.

Example Settings:

// isabl cli's settings
{
    ...
    "SUBMIT_ANALYSES": "isabl_cli.batch_systems.submit_slurm",
    "SUBMIT_CONFIGURATION": {
        "extra_args": " -p GPU_PARTITION ",
        "get_requirements": "my_isabl_extension.batch_systems.get_slurm_requirements",
        "throttle_by": 50,
        "unbuffer": True,
    },
    "SUBMIT_MERGE_ANALYSIS": "my_isabl_extension.batch_systems.submit_merge_analysis_to_slurm",
    ...
}
# my_isabl_extension/batch_systems.py

from my_isabl_extension import apps


def get_slurm_requirements(app, targets_methods):
    """Get custom requirements for any app or method."""
    
    # Defaults
    runtime = 240 # 6 hours
    cores = 1
    memory = 6
    
    # Technique.method-specific    
    method = targets_methods[0]
    if method == "WG":
        runtime = 10080 # 7 days

    # Application-specific
    if isinstance(app, apps.BwaMemGRCh37):
        cores = 32
        runtime = 1440 # 24 hours
        memory = 8

    return (
        f"--ntasks=1 "
        f"--cpus-per-task={cores} "
        f"--time={runtime} "
        f"--mem-per-cpu={memory}G "
    )


def submit_merge_analysis_to_slurm(instance, application, command):
    """Submit command to run merge in slurm."""

    command = f"sbatch --ntasks=1 --cpus-per-task=1 --mem=32G {command}"
    subprocess.check_call(command.split())
    click.secho(f"Submitted project level merge with: {command}", fg="yellow")

Last updated