# Quick Start
{% hint style="warning" %}
The Isabl Platform is **freely available for research and academic purposes**. For commercial use, a license is required. To access the source code, please [get in touch with us](https://app.gitbook.com/u/N5rUcsWkbhbAHG3vQPsQQ2Tj40J2).
{% endhint %}
Welcome to the *10 Minutes to Isabl* guide! This tutorial will walk you through installation, meta data registration, data import, and automated data processing.
{% hint style="info" %}
* Also check the [Retrieve Data](https://docs.isabl.io/retrieve-data) and [Register Metadata](https://docs.isabl.io/data-model) guides!
* Join us on [Gitter](https://gitter.im/isabl-io/community) if you have questions
* Submit an [issue](https://github.com/isabl-io/demo/issues/new) 🐛 if you are having problems with this guide
{% endhint %}
## Intro to Isabl
Checkout the documentation [home](https://docs.isabl.io/master) page for an intro to Isabl.
## Prerequisites
* [Docker Compose](https://docs.docker.com/compose/install/) for building and running the application.
{% hint style="warning" %}
Make sure your installation doesn't require `sudo` to run `docker` and `docker-compose`. Otherwise you will have issues running this demo. Check `docker run hello-world` runs without problem. If you have permissions issues, see [how to run docker as-non-root user](https://docs.docker.com/install/linux/linux-postinstall/#manage-docker-as-a-non-root-user).
{% endhint %}
## Demo Setup
Let's start by clone the demo:
```bash
git clone https://github.com/papaemmelab/isabl_demo.git --recurse-submodules
cd isabl_demo
```
{% hint style="warning" %}
Make sure the `git submodule` folders `isabl_api` and `isabl_cli` are not empty. If they are, probably the `--recurse-submodules` flag didn't work.
As a workaround, run:
`git submodule update --recursive` --init
{% endhint %}
Next, source a simple initiation profile:
```bash
source .demo-profile
```
{% hint style="info" %}
If you are **redoing** the tutorial, we recommend to remove the demo directory and clone it again:
```bash
chmod -R u+w isabl_demo && rm -rf isabl_demo
```
Also remove the Docker volume:
```bash
docker volume rm isabl_demo_local_postgres_data
```
{% endhint %}
## Installation
{% hint style="warning" %}
This installation relies on docker private images. Please make sure with the *isabl team admins* that you have proper access to them.
{% endhint %}
Build and run the application (this might take a few minutes):
{% hint style="success" %}
Before running `docker-compose build,` specify your **platform** architecture if it's different from the standard intel `linux/amd64`, i.e. you have an **Apple M1/M2**:
```bash
export DOCKER_DEFAULT_PLATFORM=linux/arm64
```
If you are not sure, you can check your **platform** by running:
uname -m
{% endhint %}
```bash
demo-compose build
```
Now we can run the application in the background:
```bash
demo-compose up -d
```
{% hint style="info" %}
You can type `demo-compose down` to stop the application. And use `demo-compose logs -f` in a new session to see logs.
{% endhint %}
Create a superuser by running `demo-django createsuperuser`, for example using credentials: `username=admin password=admin email=admin@demo.io`
Now, visit and log in!
{% hint style="info" %}
`demo-compose`, `demo-django`, and `demo-cli` are simple wrappers around `docker-compose` - check them out. The `isabl_demo` directory was bootstrapped using [cookiecutter-isabl](https://isabl-io.github.io/docs/#/api/settings), a proud fork of [cookiecutter-django](https://github.com/pydanny/cookiecutter-django)! Many topics from their [guide](https://cookiecutter-django.readthedocs.io/en/latest/developing-locally-docker.html) will be relevant to your project.
{% endhint %}
## Create Project
Creating a project in Isabl is as simple as adding a title. You can also specify optional fields:
## Configure Isabl CLI
We need to let `isabl-cli` know where can the API be reached, and what CLI settings we should use (if you are using `demo-compose`, these variables are already set):
```bash
# install isabl-cli from github (Use a virtualenv as a good practice)
pip install -e ./isabl_cli
pip install -e ./isabl_apps
# point to your API URL
export ISABL_API_URL=http://localhost:8000/api/v1/
# set environment variable for demo client
export ISABL_CLIENT_ID="demo-cli-client"
```
Now we can create a client for `isabl-cli`:
```bash
# create and update the client object
demo-cli python3.6 ./assets/metadata/create_cli_client.py
# check the file used to create the client
cat ./assets/metadata/create_cli_client.py
```
## Register Samples
Before we create samples, let's use `isabl-cli` to add choices for *Center*, *Disease*, *Sequencing Technique*, and *Data Generating Platform*:
```bash
demo-cli python3.6 ./assets/metadata/create_choices.py
```
{% hint style="info" %}
New options can also be easily created using the admin site:
{% endhint %}
We will use *Excel submissions* to register samples through the web interface. To do so, the demo project comes with a pre-filled metadata form available at:
```bash
open assets/metadata/demo_submission.xlsm
```
{% hint style="info" %}
When prompted to allow *macros*, say yes. This will enable you to toggle between optional and required columns. By the way, Isabl has multiple mechanisms for metadata ingestion! Learn more [here](https://docs.isabl.io/data-model).
{% endhint %}
Now let's proceed to submit this excel form. First open the directory:
```bash
open assets/metadata
```
And drag the `demo_submission.xlsm` file into the submit samples form:
We can also review metadata at the command line:
```bash
isabl get-metadata experiments --fx
```
{% hint style="info" %}
Expand and navigate with arrow keys, press e to *expand all* and E to minimize. Learn more at [`fx` documentation](https://github.com/antonmedv/fx/blob/master/docs.md#interactive-mode). Use `--help` to learn about other ways to visualize metadata (e.g. `tsv`).
{% endhint %}
For this particular demo, we wanted to create a *sample tree* that showcased the flexibility of Isabl's data model. Our demo individual has two samples, one normal and one tumor. The tumor sample is further divided into two biological replicates (or *aliquots*), with two experiments conducted on the second aliquot:
## RESTful API
Although not required for this tutorial, you are welcome to checkout the RESTful API documentation at: or .
## **Import** Reference Data
Given that `isabl-cli` will move our test data, let's copy original assets into a *staging* directory:
```bash
mkdir -p assets/staging && cp -r assets/data/* assets/staging
```
Now let's import the genome:
demo-isabl import-reference-genome \
--assembly GRCh37 \
--genome-path assets/staging/reference/reference.fasta
We can also import BED files for our demo *Sequencing Technique*:
```bash
demo-isabl import-bedfiles \
--technique DEMO_TECHNIQUE \
--targets-path assets/staging/bed/targets.bed \
--baits-path assets/staging/bed/baits.bed \
--assembly GRCh37 \
--species HUMAN \
--description "Demo BED files"
```
Check that import was successful:
isabl get-bed DEMO_TECHNIQUE # retrieve BED file
isabl get-reference GRCh37 # retrieve reference genome
By means of the `--data-id` flag, the command `get-reference` also allows you to retrieve the indexes generated during import. To get a list of available files per assembly run:
```bash
isabl get-reference GRCh37 --resources
```
{% hint style="info" %}
Learn more about importing data into Isabl [here](https://docs.isabl.io/import-data).
On this previous steps of the demo, we used `demo-isabl` to import references and bedfiles, because it is a wrapper to run `isabl` inside a container with `samtools`, `bwa` and `tabix` . These tools are needed to index these imported genomic files.
{% endhint %}
## Import Experimental Data
Next step is to import data for the samples we just created:
```bash
isabl import-data \
-di ./assets/staging `# provide data location ` \
-id identifier `# match files using experiment id` \
-fi identifier.contains "demo" `# filter samples to be imported ` \
--symlink `# to not move the original demo data`
```
{% hint style="success" %}
Add `--commit` to complete the operation.
{% endhint %}
Retrieve imported data for the normal to see how directories are created:
```bash
isabl get-data -fi sample.identifier "demo normal"
```
The front end will also reflect that data has been imported.
## Writing Applications
Isabl is a language agnostic platform and can deploy any pipeline. To get started, we will use some applications from [isabl-io/apps](https://github.com/isabl-io/apps). Precisely we will run alignment, quality control, and variant calling. Applications were [previously registered](#configure-isabl-cli) in client object. Once registered, they are available in the client:
```bash
isabl apps-grch37
```
{% hint style="info" %}
Learn more about customizing your instance with [Isabl Settings](https://docs.isabl.io/isabl-settings).
{% endhint %}
First we'll run alignment (pass `--commit` to deploy):
```bash
isabl apps-grch37 `# apps are grouped by assembly ` \
bwa-mem-0.7.17.r1188 `# run bwa-mem version 0.7.17.r1188 ` \
-fi tags.contains data `# filter using tags, feel free to try others `
```
{% hint style="info" %}
Note that if you try to re-run the same command, Isabl will notify you that results are already available. If for some reason the analyses fail, you can force a re-run using `--force`.
{% endhint %}
Now we can retrieve bams from the command line:
```bash
isabl get-bams -fi sample.individual.identifier "demo individual"
```
We can also visualize aligned bams online:
{% hint style="info" %}
Insert `2:123,028-123,995` in the locus bar, that's were our test data has reads. Learn more about default BAMs in the [Writing Applications](https://docs.isabl.io/writing-applications) guide.
{% endhint %}
## Auto-merge Analyses
Let's get some stats for our experiments with a quality control [application](https://github.com/isabl-io/apps/blob/4f893b8995c110c1d685f49a04737533173907c4/isabl_apps/apps/qc_data/apps.py#L20):
```bash
isabl apps-grch37 qc-data-0.1.0 -fi identifier.icontains demo --commit
```
This quality control application has defined logic to merge results at a project and individual level. Upon completion of analyses execution, Isabl automatically runs the auto-merge logic:
Isabl-web can render multiple types of results, in this case we will check at HTML reports. Results for our `qc-data` application are available at an *experiment*, *individual*, and *project* level. In this example we are looking at the *project-level* auto-merge analysis:
{% hint style="info" %}
[Applications](https://docs.isabl.io/writing-applications) can define any custom logic to merge analyses.
{% endhint %}
## Multi-experiment Analyses
Up until now we've run applications that are linked to one experiment only. However, analyses can be related to any number of *target* and *reference* experiments. For example [this implementation of *Strelka*](https://github.com/isabl-io/apps/blob/4f893b8995c110c1d685f49a04737533173907c4/isabl_apps/apps/strelka/apps.py#L15) uses *tumor-normal* pairs. Before you can run this command you will need to retrieve the system id of your experiments, let's try:
```bash
isabl get-metadata experiments -f system_id
```
Now insert those identifiers in the following command:
```bash
isabl apps-grch37 strelka-2.9.1 \
--pairs {TUMOR 1 ID} {NORMAL ID} `# replace tumor 1 system id and normal system id` \
--pairs {TUMOR 2 ID} {NORMAL ID} `# replace tumor 2 system id and normal system id` \
--pairs {TUMOR 3 ID} {NORMAL ID} `# replace tumor 3 system id and normal system id`
```
You can retrieve registered results for the analysis, for instance the *indels* VCF:
```bash
isabl get-results -fi name STRELKA --result-key indels
```
{% hint style="info" %}
To find out what other results are available use:
```bash
# app-primary-key can be retrieved from the frontend
isabl get-results --app-results {app-primary-key}
# when writing this tutorial, the app key for strelka was 5
isabl get-results --app-results 5
```
{% endhint %}
Furthermore, you can get paths for any instance in the database using `get-paths`:
```bash
isabl get-outdirs -fi name STRELKA
```
Lastly, lets check the indels VCFs through the web portal:
## Software Development Kit
To finalize the tutorial, we'll use Isabl as an SDK with `ipython`:
```bash
# we'll access ipython using the cli container
demo-cli ipython
```
Then lets check the output directories of Strelka:
```python
import isabl_cli as ii
# retrieve analyses from API using filters
analyses = ii.get_analyses(name="STRELKA")
# list the strelka ouput directories
for i in analyses:
!ls {i.storage_url}/strelka
```
The analysis objects are [`Munch`](https://github.com/Infinidat/munch), in other words they are dot-dicts (like javascript):
```python
analysis = analyses[0]
# get the target experiment or tumor
target = analysis.targets[0]
# print the parent sample class
print(target.sample.category)
# see available fields
print(target.keys())
```
## Wrap up and Next Steps
Learn about [CLI advanced configuration](https://github.com/isabl-io/docs/tree/c6cbd729fc1d9696332a3c78ea48aa4c7f409066/guides/cli/README.md#configuration) to customize functionality:
{% content-ref url="isabl-settings" %}
[isabl-settings](https://docs.isabl.io/isabl-settings)
{% endcontent-ref %}
Learn about writing applications:
{% content-ref url="writing-applications" %}
[writing-applications](https://docs.isabl.io/writing-applications)
{% endcontent-ref %}
Ready for **production**? learn more about [deployment](https://github.com/isabl-io/docs/tree/c6cbd729fc1d9696332a3c78ea48aa4c7f409066/tutorials/deployment/README.md):
{% content-ref url="production-deployment" %}
[production-deployment](https://docs.isabl.io/production-deployment)
{% endcontent-ref %}