Writing Applications
⚡️Isabl Applications enable you to systematically deploy data science tools across thousands of Experiments in a metadata driven approach. Learn how to build them here.
Isabl Applications enable you to systematically deploy data science tools across thousands of Experiments in a metadata driven approach. The most important things to know about applications are:
- Applications are agnostic to the underlying tools being utilized.
- Applications can submit analyses to multiple compute environments (local, cluster, cloud).
- Results are stored as analyses for which uniqueness is a function of the experiments used.
- Once implemented, applications can be deployed across any subset of experiments in the database.
Isabl applications are not Workflow Management Systems (see what Isabl is not trying to solve). Instead they use metadata to systematically build and deploy any type of execution commands across thousands of experiments.
During this tutorial we will build a hello world application that show cases the functionalities and advantages of processing data with Isabl. Here is a really simple example of an Isabl application that echoes an experiment's sample identifier and it's raw data.
hello_world/apps.py
from isabl_cli import AbstractApplication
from isabl_cli import options
class HelloWorldApp(AbstractApplication):
NAME = "HELLO WORLD"
VERSION = "1.0.0"
cli_options = [options.TARGETS]
def get_command(self, analysis):
experiment = analysis.targets[0]
return f"echo {experiment.sample.identifier} {experiment.raw_data} "
This application can now be executed system-wide using:
isabl apps hello-world --filters projects 102
cli_options
enabled us to run the app across multiple experiments using RESTful API filters (i.e. --filters
). We will learn more about how to link experiments with analyses later.Results produced by applications are stored as analyses. The uniqueness of an analysis is determined by the experiments associated with it. Specifically, analyses can be linked to multiple targets and references experiments (e.g. tumor-normal pairs). The possibility of linking analyses to multiple experiments allow for a wide variety of experimental designs:
- Single target analyses (e.g. quality control applications).
- Tumor-normal pairs (e.g. variant calling applications).
- One target vs. a pool of references (e.g. copy number applications).
- Multiple targets agains multiple references (e.g. all vs. all contamination testing).
Importantly, if someone tries to run the same application over the same experiments, a new analysis won't be created and but the existing one will be retrieved.

Isabl applications are python classes with the role of constructing, validating, and deploying commands for tools (or pipelines) into compute environments across several samples, all guided by metadata retrieved from Isabl API.
You should store your applications and custom Isabl logic in your own python package. Cookiecutter Apps will help you bootstrap your own Isabl project:
# first make sure you have cookiecutter installed
pip install cookiecutter
# now lets bootstrap your project
cookiecutter https://github.com/isabl-io/cookiecutter-apps
# finally install you project in a new virtual environment
cd <project-name> && pip install -r requirements.txt
An example of a project generated with Cookiecutter Apps its available here. This project, and every project created with Cookiecutter Apps includes the hello world application described in this tutorial, check it out here. Now let's learn about writing apps!
To make sure your applications are available when running
isabl --help
, make sure you add them to the client setting INSTALLED_APPLICATIONS
: "INSTALLED_APPLICATIONS": [
"example_apps.apps.HelloWorldApp",
]
All Isabl Applications inherit from
isabl_cli.AbstractApplication
and are configured using a class based approach. Your role is to override attributes and methods to drive the behavior of your app. Applications are represented both with a python class and a database object. The database object is created and updated automatically when the application is run.
Applications are uniquely versioned by setting the
NAME
and VERSION
attributes. The version of an application is not necessarily the version of the underlying tool being executed:class HelloWorldApp(AbstractApplication):
NAME = "Hello World"
VERSION = "1.0.0"
A good strategy to version applications is to ask the question: are results comparable across experiments? An optimization (or bug fix) that doesn't change outputs might not require a version change.
Optionally you can also set
ASSEMBLY
and SPECIES
to version the application as a function of a given genome assembly. This is particularly useful for NGS applications as often results are only comparable if data was analyzed against the same version of the genome: class HelloWorldApp(AbstractApplication):
NAME = "Hello World"
VERSION = "1.0.0"
ASSEMBLY = "GRCh37"
SPECIES = "HUMAN"
You can add additional metadata to be attached to the database object, such as an application description and URLs (or comma separated URLs):
class HelloWorldApp(AbstractApplication):
application_description = "An App to show case different Isabl functionalities."
application_url = "https://docs.isabl.io/writing-applications"
Applications can depend on multiple configurations such as paths to executables, references files, compute requirements, etc. These settings are explicitly defined using the
application_settings
dictionary:class HelloWorldApp(AbstractApplication):
application_import_strings = {"sym_link"}
application_settings = {
"echo_path": "echo",
"default_message": "Hello World",
"sym_link": "isabl_cli.utils.force_symlink"
}
If a setting is meant to be imported, include it in
application_settings_import_strings
.Optional settings can be set to
None
whilst required but not yet defined settings can be set to theNotImplemented
python object. Settings defined in the application python class are considered to be the default settings, yet they can be overridden using the database application field settings
. > HelloWorldApp().application.settings
{
# settings as a function of Isabl CLI client's primary key
1: {"echo_path": "/usr/local/bin/echo"}
}
Note that
application.settings
are a function of the client's primary key. This enables you to run the sample application in different compute architectures. You can configure application.settings
using the Django Admin site or the application method patch_application_settings
:HelloWorldApp().patch_application_settings(
client_id=1,
echo_path="/usr/bin/echo"
)
If the value of a setting is a dictionary, the schema (i.e. keys) of that setting will be validated unless the dictionary contains
"skip_check": True.
You can make sure applications are properly configured by performing settings validation. To do so, simply define
validate_settings
and raise an AssertionError
if something is not set properly:from shutil import which
class HelloWorldApp(AbstractApplication):
def validate_settings(self, settings):
assert which(settings.echo_path), f"{settings.echo_path} not in PATH"
The
application_settings
dictionary defines default settings, but during execution your app may have different settings for clients or environments. For example, you may have a small test reference file for testing and the real one for production. That's why you can define NotImplemented
by default, but validate that it's in fact implemented on execution.Applications can be launched from both the command line and from python (we will learn more about the latter in the operational automations guide).
To support CLI capabilities you have to tell the application how to link analyses to experiments using command line options:
from isabl_cli import options
class HelloWorldApp(AbstractApplication):
cli_help = "This is the Hello World App - a way to learn Isabl applications."
cli_options = [options.TARGETS]
The attribute
cli_options
is set to a list of Click Options that will be used to retrieve experiments from the API and link them to new analyses. Out of the box, Isabl supports the following CLI options to retrieve experiments:isabl.options | Description |
TARGETS | Enable --filters (-fi) to provide key value pair of RESTful API filters used to retrieve experiments (e.g. -fi sample.category TUMOR ). Each experiment will be linked to a new analysis in a one-to-one basis using the analysis.targets field. |
PAIRS , PAIR , PARS_FROM_FILE | Enable --pairs (-p), --pair (-p), --paris-from-file (-pf) to provide pairs of target, reference experiments (e.g. -p TUMOR-ID NORMAL-ID ). Each pair will be linked to a new analysis (targets list is one experiment, references list is one experiment). |
REFERENCES , NULLABLE_REFERENCES | Enable --references-filters (-rfi) to provide filters to retrieve reference experiments. This has to be coupled withTARGETS , each analysis will then be linked to one target, and to as many references. |
When these options are not adequate for your experimental design, you can implement
get_experiments_from_cli_options
. This function takes the evaluated cli_options
and must return a list of tuples: one tuple per analysis, each tuple with 2 lists: the target experiments and the reference experiments. Here is a an example of an application that creates only one analysis linked to all Whole Genome experiments in a project:from isabl_cli import api
import click
class HelloWorldApp(AbstractApplication):
cli_options = [
click.option("--project", help="project key", required=True),
click.option("--method", help="technique method", default="WG"),
]
def get_experiments_from_cli_options(self, **cli_options):
project = cli_options["project"]
method = cli_options["method"]
filters = dict(projects=project, technique__method=method)
targets = api.get_instances("experiments", **filters)
return [(targets, [])] # will result in only one analysis
By default, applications come with
--force
to remove and start analyses from scratch, --restart
to run failed analyses again without trashing them, --local
to run analyses locally, one after the other:class HelloWorldApp(AbstractApplication):
cli_allow_force = True
cli_allow_restart = True
cli_allow_local = True
As such, the CLI configuration for our Hello World app will result in the following help message:
$ isabl apps hello-world-1.0.0 --help
Usage: isabl apps hello-world-1.0.0 [OPTIONS]
This is the Hello World App - a way to learn Isabl applications.
Options:
--targets-filters API filters for target experiments [required] <TEXT TEXT>...
--commit Submit application analyses. [default: False]
--force Wipe unfinished analyses and start from scratch.
--restart Attempt restarting failed analyses from previous checkpoint.
--help Show this message and exit.
The
--force
flag will not completely remove the analyses, but it will move them to a temporary trash directory within the BASE_STORAGE_DIRECTORY
. You may want to clean this location periodically using crontab -e
:crontab -e
# Clears trash directory.
0 0 * * * source ~/.bash_profile &> /dev/null; rm -rf <replace with BASE_STORAGE_DIRECTORY>/.analyses_trash/* &> /dev/null;
You can use
cli_options
to include any other argument your app may need in order to successfully build and deploy data processing tools.Some of the advantages of metadata-driven applications is that we can prevent analyses that don't make sense, for example running a variant calling application on imaging data. Simply raise an
AssertionError
if something doesn't make sense, and the error message will be provided to the user:class HelloWorldApp(AbstractApplication):
def validate_experiments(self, targets, references):
assert len(targets) == 1, "only one target experiment per analysis"
assert targets[0].raw_data, "target experiment has no linked raw data"
self.validate_dna_only(targets) # multiple validators are readily available
Analyses are understood to be unique if their targets, references, and application are the same (as well as previously linked dependencies). If you need custom get or create logic, you can override the
get_or_create_analyses
method.AbstractApplication
comes with readily available validators that you may want to use. Here are some examples of commonly used ones:validate_dna_only
Check technique category isDNA.
validate_same_technique
Validate experiments have same experimental technique.validate_same_individual
Check experiments come from same individual.
Now that we know how to link analyses to experiments, lets dive into creating data processing commands. Our only objective is to use the analysis and settings objects to build a
shell
command and return it as a string (ignore inputs
for now, we will learn more about it when specifying application dependencies).from os.path import join
import click
class HelloWorldApp(AbstractApplication):
cli_options = [options.TARGETS, click.option("--message")]
def get_command(self, analysis, inputs, settings):
echo = settings.echo_path
target = analysis.targets[0]
message = settings.run_args.message or settings.default_message
output_file = join(analysis.storage_url, "output.txt")
input_file = join(analysis.storage_url, "input.txt")
settings.sym_link(target.raw_data[0].file_url, input_file)
return (
f"bash -c '{echo} Sample: {target.sample.identifier} > {output_file}' && "
f"bash -c '{echo} Message: {message} >> {output_file}' && "
f"bash -c '{echo} Data: >> {output_file}' && "
f"bash -c 'cat {input_file} >> {output_file}' "
)
All options passed in
cli_options
are available during get_command
using the settings attribute run_args
. In this simple example, we allowed the user to pass a custom --message
.This also means we can perform actions when restarting an analysis, which can be useful for cleaning up files:
from os.path import join, exists
from os import remove
class HelloWorldApp(AbstractApplication):
cli_options = [options.TARGETS]
def get_command(self, analysis, inputs, settings):
if settings.restart:
to_rm_path = join(analysis.storage_url, "run.lock")
if exists(to_rm_path):
remove(to_rm_path)
return "echo 'Hello World!'"
Isabl is agnostic to compute architecture,
get_command
does not need to worry about HPC schedulers, or cloud architecture (e.g. LSF
, AWS
), its only role is to return a shell command.Isabl
is agnostic of the compute infrastructure you're working on and can be configured to work with different batch systems (e.g. local, HPC, cloud). Currently, Isabl
supports local
, LSF
, SGE
, and Slurm
submissions, how ever you can create a submitter for other schedulers.Importantly
Isabl
is not a workflow management system or language like Toil, Bpipe, CWL, etc. Isabl however, can submit head jobs per analysis to a compute infrastructure.Isabl comes with prebuilt logic to submit thousands of analyses to
LSF
, SGE
, and Slurm
using Job Arrays. To do so simply set the Isabl CLI setting SUBMIT_ANALYSES
as follows:// IBM's LSF
"SUBMIT_ANALYSES": "isabl_cli.batch_systems.lsf.submit_lsf",
// Sun Grid Engine
"SUBMIT_ANALYSES": "isabl_cli.batch_systems.sge.submit_sge",
// Slurm
"SUBMIT_ANALYSES": "isabl_cli.batch_systems.slurm.submit_slurm",
Configuration Name | Type | Description |
get_requirements | Import String | An import string to a function that will determine LSF requirements as a function of the experimental methods, see below. |
extra_args | String | Default qsub , bsub , or sbatch args to be used across all submissions. |
throttle_by | Integer | The total number of analyses that are allowed to run at the same time (default is 50). |
The method
get_requirements
must take the application and a list of targets' technique methods (which are submitted together in the same job array):def get_lsf_requirements(app, targets_methods):
if isinstance(app, apps.HelloWorldApp):
memory = 10 if "WG" in targets_methods else 1
return f"-n {1} -R 'rusage[mem={memory}]'"
You can implement
SUBMIT_ANALYSES
functions for other schedulers, the function must take a list of tuples, each tuple being an analysis and the analysis head job script.Isabl applications can be run by multiple users in the same unix group. However, if applications are run by users different than the
ADMIN_USER
and are not re-runnable, then analyses will be set to FINISHED
instead of SUCCEEDED
. isabl process-finished
can be run by the ADMIN_USER
to copy and own the results and set the permissions to read-only whilst updating analyses status to SUCCEEDED
. We recommend you add the following cron task in the ADMIN_USER
profile using crontab -e
: crontab -e
# Change analyses permissions and updates them to SUCCEEDED.
*/30 * * * * source ~/.bash_profile &> /dev/null; isabl process-finished &>> ~/moving.log
Apps can programmatically be triggered from python using the
run
method:HelloWorldApp().run(
tuples=[([target_experiment], [])],
run_args=dict(message="custom message"),
commit=True,
)
You can provide an specification your application results using the
application_results
dictionary. Each key is a result id and the value is a dictionary with specs of the result:class HelloWorldApp(AbstractApplication):
application_results = {
"input": {
"frontend_type": "text-file",
"description": "Symlink to experiment raw data.",
"verbose_name": "Hello World Input",
"external_link": "https://en.wikipedia.org/wiki/Symbolic_link",
},
"output": {
"frontend_type": "text-file",
"description": "Sample id, hello world message, and content of raw data.",
"verbose_name": "Hello World Result",
"external_link": "https://hello.world/",
},
"count": {
"frontend_type": "number",
"description": "Count of characters in output file.",
"verbose_name": "Characters Count",
},
}
By default, all applications come with 3 default settings
command_script
, command_log
, and command_err
. These point to the standard output, standard error, and analysis head job command, respectively.Results can be paths to files, strings (e.g. MD5s), numbers, and any other serializable value. Here is a full list of the different specifications a result can have:
description
Information about the result (required)verbose_name
Name displayed for the result in the results list (required)optional
IfFalse
and result is missing, an alert will be shown online (optional)external_link
URL to a resource that may explain about the result (optional)
By default, analysis results are protected upon completion (i.e. permissions are set to read only). If you want your application to be re-runnable indefinitely, set
application_protect_results = False
.Here is a full list of the result types that are supported for rendering in Isabl Web:
Frontend Type | Description |
text-file | It's shown as a raw file, and its content is streamed as the user requests it. |
tsv-file | It can be shown as raw text or tabulated for easier inspection (i.e. VCF, TSV). |
string , number | It's shown as a string and can't be downloaded. |
image | Previews are displayed in a gallery in the analysis view. |
html , pdf | Rendered as html in an iframe . |
igv_bam[:index] | Can be streamed to visualized in an embedded IGV viewer. If another result called bai is the BAM index, you can set it toigv_bam:bai . |
By default, analysis results are protected upon completion (i.e. permissions are set to read only). If you want your application to be re-runnable indefinitely, set:
class HelloWorldApp(AbstractApplication):
application_protect_results = False
When
application_results
is defined, you must implement get_analysis_results
. This method must return a serializable dictionary of results and its only run after the analysis has been completed successfully. For our example it can be something like:class HelloWorldApp(AbstractApplication):
def get_analysis_results(self, analysis):
output = join(analysis.storage_url, "output.txt")
with open(output) as f:
count = sum(len(i) for i in f)
return {
"input": join(analysis.storage_url, "input.txt"),
"output": output,
"count": count
}
Isabl applications can produce auto-merge analyses at a project and individual level. For example, you may want to merge variants whenever new results are available for a given project, or update quality control reports when a new sample is added to an individual. A newly versioned analysis will be created for each type of auto-merge and your role is to take a list of succeeded analysis and implement the merge logic.
class HelloWorldApp(AbstractApplication):
application_project_level_results = {
"merged": {
"frontend_type": "text-file",
"description": "Merged output files.",
"verbose_name": "Merged Output Files",
},
"count": {
"frontend_type": "number",
"description": "Count of characters for merged output.",
"verbose_name": "Merged Outth put Characters Count",
},
}
def merge_project_analyses(self, analysis, analyses):
with open(join(analysis.storage_url, "merged.txt"), "w") as f:
for i in analyses:
with open(i.results.output) as output:
f.write(output.read())
def get_project_analysis_results(self, analysis):
merged = join(analysis.storage_url, "merged.txt")
with open(merged) as f:
count = sum(len(i) for i in f)
return {"merged": merged, "count": count}
The first argument in
merge_project_analyses
, is the project level analysis
, which is unique per project and application. The second argument is a list of all completed analyses
of this application for a given project. Your role is to merge analyses
output into the project level analysis
directory. We need to define similar methods for the Individual level auto merge. Lets say that our project-level merge logic is the same for individuals, then we can simply do:class HelloWorldApp(AbstractApplication):
# reuse the project merge logic at the individual level
application_individual_level_results = application_project_level_results
merge_individual_analyses = merge_project_analyses
get_individual_analysis_results = get_project_analysis_results
If at any arbitrary time you want to test the auto-merge logic, use any of these two commands:
# for project level auto merge
isabl merge-project-analyses --project <project id> --application <application id>
# for individual level automerge
isabl merge-individual-analyses --individual <individual id> --application <application id>
Please note that merged output will always be stored in the same analysis for a given project or individual and application. Furthermore, you can validate analyses before running the merge operation by implementing
validate_project_analyses
, and validate_individual_analyses
.Merge operations are triggered automatically when the last analysis that is meant to be merged finish running. By default, the merge operation will be conducted right after the analysis is patched to
SUCCEEDED
. However, you can define how merge analyses are submitted using Isabl CLI setting SUBMIT_MERGE_ANALYSIS
. For example in LSF
:import subprocess
def submit_merge_analysis_to_lsf(instance, application, command):
"""Submit project merge to LSF."""
command = ["bsub", "-n", "1", "-W", "40000", "-M", "32"] + command.split()
subprocess.check_call(command)
print("Submited merge analysis using: " + " ".join(command))
Here is an example for
SGE
:import subprocess
def submit_merge_analysis_to_sge(instance, application, command):
"""Submit project merge to SGE."""
command = f"qsub -l h_vmem=[32G] << EOF\n{command}\nEOF\n"
subprocess.check_call(command, shell=True)
print(f"Submited project level merge with: {command}")
This section lists additional optional functionality supported by Isabl applications. Particularly, dependencies on other applications, after completion analyses status, and unique analyses per individual.
Application inputs are analysis-specific settings (settings are the same for all analyses, yet inputs are potentially different for each analysis). Inputs can be formally defined using
application_inputs
, inputs set to NotImplemented
are considered required and must be resolved using get_dependencies
:from isabl_cli import utils
class HelloWorldApp(AbstractApplication):
application_inputs = {"previously_generated_result": NotImplemented}
# get dictionary of inputs this function is run before get_command.
# must return a tuple: (list of analyses dependencies primary keys, inputs dict)
def get_dependencies(self, targets, references, settings):
result, analysis_key = utils.get_result( # return result value and key of analysis that generated it
experiment=targets[0], # get result for the first target experiment
application_key=123, # primary key of previous app in database
result_key="a_result_name", # name of result in the app definition
)
return [analysis_key], {"previously_generated_result": result}
The main objective of
application_inputs
and get_dependencies
is to retrieve results and analyses that should be linked to the newly created analysis. Linked analyses are accessible from the analysis detail frontend view.In certain cases you don't want your analyses to be marked as
SUCCEEDED
after completion, as you may want to flag them for manual review or leave them to know that you need to run an extra step on them. For these cases, you may want to set the after-completion status to IN_PROGRESS
:class HelloWorldApp(AbstractApplication):
def get_after_completion_status(self, analysis):
return "IN_PROGRESS"
It is possible to create applications that are unique at the individual level. To do so set
unique_analysis_per_individual = True
. A good example of a unique per individual application could be a patient centric report that aggregates results across all samples. If you are interested on how analyses for these applications are created, give a look at AbstractApplication.get_individual_level_analyses.
Individual Level Auto-Merge and Unique Analyses Per Individual are different concepts. Applications that require a unique analysis per individual don't support individual level auto-merge.
You can configure Isabl API to periodically check if any analysis has failed and send you email notifications. To do so, head to the admin site at
/admin/django_celery_beat/periodictask/add/
and in Task (registered) select isabl_api.tasks.report_status_change_task
, then create a 1 hour interval, and provide the following Keyword arguments {"status": "FAILED", "seconds": 3600}
(i.e. every hour check how many analyses failed in the past hour):
Every hour check how many analyses failed in the past hour.
Our goal is to make it extremely easy to test your applications. Ideal apps can be tested locally, with fake/dummy data using factory-created database instances. Isabl CLI and Cookiecutter Apps come with a range of utilities to help you test your applications.
If you created your project using Cookiecutter Apps, the following
pytest
fixtures are available to you: Name | Description |
commit | Enables you to run pytest using a --commit flag, this flag can later be used to actually commit the application. |
datadir | Path to the dummy data directory. Your tests directory comes with a data folder, which can be populated with dummy, small files - useful to run your apps. |
tmpdir | This is like the regular pytest tmpdir yet it comes with some perks. First, it sets the current DATA_STORAGE_DIRECTORY to a temporary directory. Second, it comes with tmpdir.docker , a method to create executable scripts to docker containers and with specific entrypoints . For example tmpdir.docker("ubuntu", "echo") creates an executable script that calls echo using an Ubuntu image. |
Isabl CLI comes with a full set of factories that facilitate the creation of fake metadata. Here is an example of how to create two experiments for the same sample:
from isabl_cli import api
from isabl_cli import factories
meta_data = factories.ExperimentFactory()
meta_data["sample"]["individual"]["species"] = "HUMAN"
experiment = api.create_instance("experiments", **meta_data)
assert experiment.pk > 0, "failed to create database instance"
We recommend limiting use of these factories to development instances of Isabl API. By default,
ISABL_API_URL
is set to http://0.0.0.0:8000/api/v1/. Being able to actually run the applications (i.e. passing
--commit
) during testing might be something valuable to you. In the case of Next Generation Sequencing, for example, you could create fake BAMs and really small reference genomes (few KBs) to test variant calling applications.Here is a comprehensive example to test our
HelloWorldApp
, projects created Cookiecutter Apps will include this test:def test_hello_world_app(tmpdir, datadir, commit):
# path to hello_world test data
hello_world_datadir = join(datadir, "hello_world")
raw_data = [
dict(
file_url=join(hello_world_datadir, "test.txt"),
file_data=dict(extra="annotations"),
file_type="FASTQ_R1",
),
]
# overwrite default configuration for the default client
meta_data = factories.ExperimentFactory(raw_data=raw_data)
meta_data["sample"]["individual"]["species"] = "HUMAN"
meta_data["storage_url"] = hello_world_datadir
experiment = api.create_instance("experiments", identifier="a", **meta_data)
# create files if you may test with real data at some point
app = HelloWorldApp()
app.application.settings.default_client = {
"default_message": "Hello from Elephant Island.",
"echo_path": tmpdir.docker("ubuntu", "echo")
}
# run application and make sure results are reported
utils.assert_run(
application=app,
tuples=[([experiment], [])],
commit=commit,
results=["output", "count", "input"],
)
The actual test is much more comprehensive. It creates two experiments per individual and validates that auto-merge was actually conducted, that the application is re-runnable, and that the command line configuration works well.
Last modified 5mo ago