Batch Systems

🪵 How to use different known batch systems for scalable job execution.

Currently Isabl supports out-of-the-box the following Batch Systems to submit jobs with isabl_cli:

Batch System	Import string	Resources
LSF	isabl_cli.batch_systems.submit_lsf	See more IBM Spectrum LSF Reference
Slurm	isabl_cli.batch_systems.submit_slurm	See more Slurm - Worload Manager
SGE	isabl_cli.batch_systems.submit_sge	See more Sun Grid Engine - queueing system

By default, all submissions are run locally using isabl_cli.batch_systems.submit_local

What about other systems?

To support other systems or to customize your own submission steps, you may need just to create your own method and define it on your isabl_cli's SUBMIT_ANALYSES setting.

Example:

# my_isabl_extension/batch_systems.py
def submit_aws_batch(app, command_tuples):
    ...

// Isabl CLI settings:
{..., "SUBMIT_ANALYSES": "my_isabl_extension.batch_systems.submit_aws_batch"}

For help, creating your custom submit method, follow direction from the existing ones.

Please consider contributing any new one to the Isabl Project! 💌

Submit Configuration

isabl_cli's settings have a SUBMIT_CONFIGURATION dictionary to provide extra arguments to the submission methods. Current parameters used are:

Attribute	Type	Description	Example value
extra_args	String	Any additional extra arguments passed to the batch submission command.	" --time=48:00:00 ". Add a maximum job array time.
get_requirements	Import String	To define custom resources needs for different applications or methods.	See the example above.
throttle_by	Integer	To control the maximum number of jobs running simultaneously in a job array submission.	50
unbuffer	Boolean	redirect stdout and stderr to same file with ascii characters that allow colored logs.	True. If not defined, it is False by default. See Colored Logs.

Example Settings:

// isabl cli's settings
{
    ...
    "SUBMIT_ANALYSES": "isabl_cli.batch_systems.submit_slurm",
    "SUBMIT_CONFIGURATION": {
        "extra_args": " -p GPU_PARTITION ",
        "get_requirements": "my_isabl_extension.batch_systems.get_slurm_requirements",
        "throttle_by": 50,
        "unbuffer": True,
    },
    "SUBMIT_MERGE_ANALYSIS": "my_isabl_extension.batch_systems.submit_merge_analysis_to_slurm",
    ...
}

# my_isabl_extension/batch_systems.py

from my_isabl_extension import apps


def get_slurm_requirements(app, targets_methods):
    """Get custom requirements for any app or method."""
    
    # Defaults
    runtime = 240 # 6 hours
    cores = 1
    memory = 6
    
    # Technique.method-specific    
    method = targets_methods[0]
    if method == "WG":
        runtime = 10080 # 7 days

    # Application-specific
    if isinstance(app, apps.BwaMemGRCh37):
        cores = 32
        runtime = 1440 # 24 hours
        memory = 8

    return (
        f"--ntasks=1 "
        f"--cpus-per-task={cores} "
        f"--time={runtime} "
        f"--mem-per-cpu={memory}G "
    )


def submit_merge_analysis_to_slurm(instance, application, command):
    """Submit command to run merge in slurm."""

    command = f"sbatch --ntasks=1 --cpus-per-task=1 --mem=32G {command}"
    subprocess.check_call(command.split())
    click.secho(f"Submitted project level merge with: {command}", fg="yellow")

🌈 Colored Logs with Batch Systems

Default: No colored logs

Colored unbuffered logs

When running jobs using batch systems like LSF or Slurm, many programs detect whether their output is being sent to a terminal (TTY) or a file and may disable features like color formatting or progress bars if they detect non-interactive output.

One workaround, using it carefully, is to use the unbuffer utility from the expect package. unbuffer simulates a terminal by running the program in a pseudo-terminal (PTY), tricking the program into thinking it is running interactively in a terminal, so it continues to produce colored output.

Requirements:

• Have unbuffer available. See tips on how to install on an HPC.

• unbuffer: True, in your isabl_clis SUBMIT_CONFIGURATION.

• >=0.3.28 version of @papaemmelab/isabl_web

• 8c8ad75 commit of isabl_cli

Unbuffered output will redirect both stdout/stderr to the samehead_job.logfile, causing the file head_job.err output to be empty from errors of the running application.

By simulating a pseudo-PTY, you can gain desired colored outputs, but some outputs like progress bars print undesired characters that can clutter your logs. Be aware to disable those if possible (i.e. toil ... --disableProgress),or for specific applications add a dummy envTERM=dumb to tell the program there's no effective terminal attached.

➕ Bonus tip: Install unbuffer in an HPC with miniconda.

conda create -n myenv
conda activate myenv
conda install -c conda-forge expect

Having $OPT_DIR where you install your packages and $BIN_DIR where you add your executable binaries, create a runnable script called unbuffer, where you add your miniconda env's unbuffer executable and add its shared libraries:

${BIN_DIR}/unbuffer

#!/bin/bash

OPT_DIR=<your-installation-dir>
export LD_LIBRARY_PATH="${OPT_DIR}/miniconda3/envs/myenv/lib:$LD_LIBRARY_PATH"
${OPT_DIR}/miniconda3/envs/myenv/bin/unbuffer "$@"

Test it works:

$ unbuffer echo "Hello World!" 
Hello World