Operational Automations

Isabl Docs

Search…

Isabl Docs

Home

Isabl Settings

Production Deployment

Contributing Guide

Other CLI commands

Operational Automations

🤖 Once you have set up your Isabl instance and created a few applications you can now automate your processes! In Isabl, this is achieved using signals.

Operational Signals
Signals are python functions that take one argument: an experiment or an analysis. Signals for experiments are triggered on data import, whilst signals for analyses are triggered on status change (i.e.  analysis failure or completion). 
Registering Signals
Register signals by including the function import string in either ON_STATUS_CHANGE or ON_DATA_IMPORT Isabl CLI settings: 
# experiments signals are triggered on data import
"ON_DATA_IMPORT": [
    "my_apps.signals.trigger_apps"
]
​
# analyses signals are triggered on status change
"ON_STATUS_CHANGE": [
    "my_apps.signals.trigger_dependencies"
]
Signals on Data Import
Signals for experiments are triggered on data import and receive the experiment object as its only argument. You can use the metadata of the experiment to determine what automation should be applied. 
from isabl_apps import apps
​
​
def signal_data_import(experiment):
    """Run upon data import using the cli."""
    species = experiment.sample.individual.species
    category = experiment.technique.category
    dna_aligner = {"HUMAN": apps.BwaMemGRCh37, "MOUSE": apps.BwaMemGRCm38}.get(species)
    tuples = [([experiment], [])]
​
    if category == "DNA" and dna_aligner:
        dna_aligner().run(tuples=tuples, commit=True)
Some examples are:
Trigger assembly/species/category aware alignment
Perform Gene quantification or Fusion calling in RNA
Create symlinks to the raw data that are more human accessible
Signals on Analysis Status Change
Analyses signals are triggered on status change. Each signal will receive the analysis object as its only argument. You can use the metadata of the experiment to determine what automation should be applied. 
from isabl_apps import apps
​
​
def signal_apps_automation(analysis):
    """Run upon an analysis status update in the database."""
    qc_app = {
        "GRCh37": apps.QualityControlGRCh37,
        "GRCm38": apps.QualityControlGRCm38,
    }.get(analysis.application.assembly.name)
​
    if (
        analysis.status == "SUCCEEDED"
        and analysis.application.name in ["DISAMBIGUATE", "BWA_MEM", "STAR"]
        and qc_app
    ):
        qc_app().run(tuples=[(analysis.targets, [])], commit=True)
Some examples are:
Trigger Quality Control/Coverage calculation after alignment has successfully been run
Trigger Variant Calling after alignment
Trigger Report Generation after analyses have been completed
Working with Signals
Here are a few examples of how to work with signals, trigger them, and get notified if signals fail.
Running Signals Manually
You can trigger signals manually with Isabl CLI:
# experiment signals
isabl run-signals experiments -s my_apps.signals.trigger_apps -fi projects 100 
​
# analyses signals
isabl run-signals analyses -s my_apps.signals.trigger_dependencies -fi projects 100 
Rerunning Failed Signals
When signals fail during automation, database records are created to keep track of this event. Rerun failed signals with:
# rerun all failed signals
isabl rerun-signals
​
# rerun failed signals using filters
isabl rerun-signals \
    -fi import_string my_apps.signals.trigger_apps \
    -fi target_endpoint analyses \
    -fi target_id <an analysis primary key>
Get Notified When Signals Fail
You can configure Isabl API to periodically check if any signal has failed and send you email notifications. To do so, head to the admin site at /admin/django_celery_beat/periodictask/add/ and in Task (registered) select isabl_api.tasks.report_failed_signals_task, then create a 15 minutes interval, and hit save:
​

Writing Applications

Isabl Settings

Last modified 2yr ago

Copy link

Outline

Operational Signals

Registering Signals

Signals on Data Import

Signals on Analysis Status Change

Working with Signals

Running Signals Manually

Rerunning Failed Signals

Get Notified When Signals Fail