# Batch Systems

Currently Isabl supports out-of-the-box the following **Batch Systems** to submit jobs with `isabl_cli`:

<table><thead><tr><th width="156">Batch System</th><th width="219">Import string</th><th>Resources</th></tr></thead><tbody><tr><td><strong>LSF</strong></td><td><code>isabl_cli.batch_systems.submit_lsf</code></td><td>See more <a href="https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=started-quick-start-guide">IBM Spectrum LSF Reference</a></td></tr><tr><td><strong>Slurm</strong></td><td><code>isabl_cli.batch_systems.submit_slurm</code></td><td>See more <a href="https://slurm.schedmd.com/documentation.html">Slurm - Worload Manager</a></td></tr><tr><td><strong>SGE</strong></td><td><code>isabl_cli.batch_systems.submit_sge</code></td><td>See more <a href="http://star.mit.edu/cluster/docs/0.93.3/guides/sge.html">Sun Grid Engine - queueing system</a></td></tr></tbody></table>

By default, all submissions are run **locally** using `isabl_cli.batch_systems.submit_local`&#x20;

{% hint style="success" %}
**What about other systems?**

To support other systems or to customize your own submission steps, you may need just to create your own method and define it on your `isabl_cli`'s `SUBMIT_ANALYSES` setting.

**Example:**

```python
# my_isabl_extension/batch_systems.py
def submit_aws_batch(app, command_tuples):
    ...
```

```json
// Isabl CLI settings:
{..., "SUBMIT_ANALYSES": "my_isabl_extension.batch_systems.submit_aws_batch"}
```

For help, creating your custom submit method, follow direction from the [existing ones](https://github.com/papaemmelab/isabl_cli/tree/master/isabl_cli/batch_systems).&#x20;

Please consider contributing any new one to the **Isabl Project!** :love\_letter:
{% endhint %}

### **Submit Configuration**

`isabl_cli`'s settings have a `SUBMIT_CONFIGURATION` dictionary to provide extra arguments to the submission methods. Current parameters used are:

<table><thead><tr><th width="213">Attribute</th><th width="93">Type</th><th width="209">Description</th><th>Example value</th></tr></thead><tbody><tr><td><strong><code>extra_args</code></strong></td><td><em>String</em></td><td>Any additional extra arguments passed to the batch submission command.</td><td><code>" --time=48:00:00 "</code>. Add a maximum job array time.</td></tr><tr><td><strong><code>get_requirements</code></strong></td><td><em>Import String</em></td><td>To define custom resources needs for different applications or methods.</td><td>See the example above.</td></tr><tr><td><strong><code>throttle_by</code></strong></td><td><em>Integer</em></td><td>To control the maximum number of jobs running simultaneously in a job array submission.</td><td><code>50</code></td></tr><tr><td><strong><code>unbuffer</code></strong></td><td><em>Boolean</em></td><td>redirect stdout and stderr to same file with ascii characters that allow colored logs.</td><td><code>True</code>. If not defined, it is <code>False</code> by default. See <a href="#colored-logs-with-batch-systems">Colored Logs</a>.</td></tr></tbody></table>

#### Example Settings:

```json
// isabl cli's settings
{
    ...
    "SUBMIT_ANALYSES": "isabl_cli.batch_systems.submit_slurm",
    "SUBMIT_CONFIGURATION": {
        "extra_args": " -p GPU_PARTITION ",
        "get_requirements": "my_isabl_extension.batch_systems.get_slurm_requirements",
        "throttle_by": 50,
        "unbuffer": True,
    },
    "SUBMIT_MERGE_ANALYSIS": "my_isabl_extension.batch_systems.submit_merge_analysis_to_slurm",
    ...
}
```

```python
# my_isabl_extension/batch_systems.py

from my_isabl_extension import apps


def get_slurm_requirements(app, targets_methods):
    """Get custom requirements for any app or method."""
    
    # Defaults
    runtime = 240 # 6 hours
    cores = 1
    memory = 6
    
    # Technique.method-specific    
    method = targets_methods[0]
    if method == "WG":
        runtime = 10080 # 7 days

    # Application-specific
    if isinstance(app, apps.BwaMemGRCh37):
        cores = 32
        runtime = 1440 # 24 hours
        memory = 8

    return (
        f"--ntasks=1 "
        f"--cpus-per-task={cores} "
        f"--time={runtime} "
        f"--mem-per-cpu={memory}G "
    )


def submit_merge_analysis_to_slurm(instance, application, command):
    """Submit command to run merge in slurm."""

    command = f"sbatch --ntasks=1 --cpus-per-task=1 --mem=32G {command}"
    subprocess.check_call(command.split())
    click.secho(f"Submitted project level merge with: {command}", fg="yellow")

```