# Quick Start
{% hint style="warning" %}
The Isabl Platform is **freely available for research and academic purposes**. For commercial use, a license is required. To access the source code, please [get in touch with us](mailto:undefined).
{% endhint %}
Welcome to the *10 Minutes to Isabl* guide! This tutorial will walk you through installation, meta data registration, data import, and automated data processing.
{% hint style="info" %}
* Also check the [Retrieve Data](/retrieve-data.md) and [Register Metadata](/data-model.md) guides!
* Join us on [Gitter](https://gitter.im/isabl-io/community) if you have questions
* Submit an [issue](https://github.com/isabl-io/demo/issues/new) 🐛 if you are having problems with this guide
{% endhint %}
## Intro to Isabl
Checkout the documentation [home](/master.md) page for an intro to Isabl.
## Prerequisites
* [Docker Compose](https://docs.docker.com/compose/install/) for building and running the application.
{% hint style="warning" %}
Make sure your installation doesn't require `sudo` to run `docker` and `docker-compose`. Otherwise you will have issues running this demo. Check `docker run hello-world` runs without problem. If you have permissions issues, see [how to run docker as-non-root user](https://docs.docker.com/install/linux/linux-postinstall/#manage-docker-as-a-non-root-user).
{% endhint %}
## Demo Setup
Let's start by clone the demo:
```bash
git clone https://github.com/papaemmelab/isabl_demo.git --recurse-submodules
cd isabl_demo
```
{% hint style="warning" %}
Make sure the `git submodule` folders `isabl_api` and `isabl_cli` are not empty. If they are, probably the `--recurse-submodules` flag didn't work.
As a workaround, run:
`git submodule update --recursive` --init
{% endhint %}
Next, source a simple initiation profile:
```bash
source .demo-profile
```
{% hint style="info" %}
If you are **redoing** the tutorial, we recommend to remove the demo directory and clone it again:
```bash
chmod -R u+w isabl_demo && rm -rf isabl_demo
```
Also remove the Docker volume:
```bash
docker volume rm isabl_demo_local_postgres_data
```
{% endhint %}
## Installation
{% hint style="warning" %}
This installation relies on docker private images. Please make sure with the *isabl team admins* that you have proper access to them.
{% endhint %}
Build and run the application (this might take a few minutes):
{% hint style="success" %}
Before running `docker-compose build,` specify your **platform** architecture if it's different from the standard intel `linux/amd64`, i.e. you have an **Apple M1/M2**:
```bash
export DOCKER_DEFAULT_PLATFORM=linux/arm64
```
If you are not sure, you can check your **platform** by running:
uname -m
{% endhint %}
```bash
demo-compose build
```
Now we can run the application in the background:
```bash
demo-compose up -d
```
{% hint style="info" %}
You can type `demo-compose down` to stop the application. And use `demo-compose logs -f` in a new session to see logs.
{% endhint %}
Create a superuser by running `demo-django createsuperuser`, for example using credentials: `username=admin password=admin email=admin@demo.io`
Now, visit and log in!
{% hint style="info" %}
`demo-compose`, `demo-django`, and `demo-cli` are simple wrappers around `docker-compose` - check them out. The `isabl_demo` directory was bootstrapped using [cookiecutter-isabl](https://isabl-io.github.io/docs/#/api/settings), a proud fork of [cookiecutter-django](https://github.com/pydanny/cookiecutter-django)! Many topics from their [guide](https://cookiecutter-django.readthedocs.io/en/latest/developing-locally-docker.html#) will be relevant to your project.
{% endhint %}
## Create Project
Creating a project in Isabl is as simple as adding a title. You can also specify optional fields:
## Configure Isabl CLI
We need to let `isabl-cli` know where can the API be reached, and what CLI settings we should use (if you are using `demo-compose`, these variables are already set):
```bash
# install isabl-cli from github (Use a virtualenv as a good practice)
pip install -e ./isabl_cli
pip install -e ./isabl_apps
# point to your API URL
export ISABL_API_URL=http://localhost:8000/api/v1/
# set environment variable for demo client
export ISABL_CLIENT_ID="demo-cli-client"
```
Now we can create a client for `isabl-cli`:
```bash
# create and update the client object
demo-cli python3.6 ./assets/metadata/create_cli_client.py
# check the file used to create the client
cat ./assets/metadata/create_cli_client.py
```
## Register Samples
Before we create samples, let's use `isabl-cli` to add choices for *Center*, *Disease*, *Sequencing Technique*, and *Data Generating Platform*:
```bash
demo-cli python3.6 ./assets/metadata/create_choices.py
```
{% hint style="info" %}
New options can also be easily created using the admin site:
{% endhint %}
We will use *Excel submissions* to register samples through the web interface. To do so, the demo project comes with a pre-filled metadata form available at:
```bash
open assets/metadata/demo_submission.xlsm
```
{% hint style="info" %}
When prompted to allow *macros*, say yes. This will enable you to toggle between optional and required columns. By the way, Isabl has multiple mechanisms for metadata ingestion! Learn more [here](/data-model.md).
{% endhint %}
Now let's proceed to submit this excel form. First open the directory:
```bash
open assets/metadata
```
And drag the `demo_submission.xlsm` file into the submit samples form:
We can also review metadata at the command line:
```bash
isabl get-metadata experiments --fx
```
{% hint style="info" %}
Expand and navigate with arrow keys, press e to *expand all* and E to minimize. Learn more at [`fx` documentation](https://github.com/antonmedv/fx/blob/master/docs.md#interactive-mode). Use `--help` to learn about other ways to visualize metadata (e.g. `tsv`).
{% endhint %}
For this particular demo, we wanted to create a *sample tree* that showcased the flexibility of Isabl's data model. Our demo individual has two samples, one normal and one tumor. The tumor sample is further divided into two biological replicates (or *aliquots*), with two experiments conducted on the second aliquot:
## RESTful API
Although not required for this tutorial, you are welcome to checkout the RESTful API documentation at: or .
## **Import** Reference Data
Given that `isabl-cli` will move our test data, let's copy original assets into a *staging* directory:
```bash
mkdir -p assets/staging && cp -r assets/data/* assets/staging
```
Now let's import the genome:
demo-isabl import-reference-genome \
--assembly GRCh37 \
--genome-path assets/staging/reference/reference.fasta
We can also import BED files for our demo *Sequencing Technique*:
```bash
demo-isabl import-bedfiles \
--technique DEMO_TECHNIQUE \
--targets-path assets/staging/bed/targets.bed \
--baits-path assets/staging/bed/baits.bed \
--assembly GRCh37 \
--species HUMAN \
--description "Demo BED files"
```
Check that import was successful:
isabl get-bed DEMO_TECHNIQUE # retrieve BED file
isabl get-reference GRCh37 # retrieve reference genome
By means of the `--data-id` flag, the command `get-reference` also allows you to retrieve the indexes generated during import. To get a list of available files per assembly run:
```bash
isabl get-reference GRCh37 --resources
```
{% hint style="info" %}
Learn more about importing data into Isabl [here](/import-data.md).
On this previous steps of the demo, we used `demo-isabl` to import references and bedfiles, because it is a wrapper to run `isabl` inside a container with `samtools`, `bwa` and `tabix` . These tools are needed to index these imported genomic files.
{% endhint %}
## Import Experimental Data
Next step is to import data for the samples we just created:
```bash
isabl import-data \
-di ./assets/staging `# provide data location ` \
-id identifier `# match files using experiment id` \
-fi identifier.contains "demo" `# filter samples to be imported ` \
--symlink `# to not move the original demo data`
```
{% hint style="success" %}
Add `--commit` to complete the operation.
{% endhint %}
Retrieve imported data for the normal to see how directories are created:
```bash
isabl get-data -fi sample.identifier "demo normal"
```
The front end will also reflect that data has been imported.
## Writing Applications
Isabl is a language agnostic platform and can deploy any pipeline. To get started, we will use some applications from [isabl-io/apps](https://github.com/isabl-io/apps). Precisely we will run alignment, quality control, and variant calling. Applications were [previously registered](/quick-start.md#configure-isabl-cli) in client object. Once registered, they are available in the client:
```bash
isabl apps-grch37
```
{% hint style="info" %}
Learn more about customizing your instance with [Isabl Settings](/isabl-settings.md).
{% endhint %}
First we'll run alignment (pass `--commit` to deploy):
```bash
isabl apps-grch37 `# apps are grouped by assembly ` \
bwa-mem-0.7.17.r1188 `# run bwa-mem version 0.7.17.r1188 ` \
-fi tags.contains data `# filter using tags, feel free to try others `
```
{% hint style="info" %}
Note that if you try to re-run the same command, Isabl will notify you that results are already available. If for some reason the analyses fail, you can force a re-run using `--force`.
{% endhint %}
Now we can retrieve bams from the command line:
```bash
isabl get-bams -fi sample.individual.identifier "demo individual"
```
We can also visualize aligned bams online:
{% hint style="info" %}
Insert `2:123,028-123,995` in the locus bar, that's were our test data has reads. Learn more about default BAMs in the [Writing Applications](/writing-applications.md) guide.
{% endhint %}
## Auto-merge Analyses
Let's get some stats for our experiments with a quality control [application](https://github.com/isabl-io/apps/blob/4f893b8995c110c1d685f49a04737533173907c4/isabl_apps/apps/qc_data/apps.py#L20):
```bash
isabl apps-grch37 qc-data-0.1.0 -fi identifier.icontains demo --commit
```
This quality control application has defined logic to merge results at a project and individual level. Upon completion of analyses execution, Isabl automatically runs the auto-merge logic:
Isabl-web can render multiple types of results, in this case we will check at HTML reports. Results for our `qc-data` application are available at an *experiment*, *individual*, and *project* level. In this example we are looking at the *project-level* auto-merge analysis:
{% hint style="info" %}
[Applications](/writing-applications.md) can define any custom logic to merge analyses.
{% endhint %}
## Multi-experiment Analyses
Up until now we've run applications that are linked to one experiment only. However, analyses can be related to any number of *target* and *reference* experiments. For example [this implementation of *Strelka*](https://github.com/isabl-io/apps/blob/4f893b8995c110c1d685f49a04737533173907c4/isabl_apps/apps/strelka/apps.py#L15) uses *tumor-normal* pairs. Before you can run this command you will need to retrieve the system id of your experiments, let's try:
```bash
isabl get-metadata experiments -f system_id
```
Now insert those identifiers in the following command:
```bash
isabl apps-grch37 strelka-2.9.1 \
--pairs {TUMOR 1 ID} {NORMAL ID} `# replace tumor 1 system id and normal system id` \
--pairs {TUMOR 2 ID} {NORMAL ID} `# replace tumor 2 system id and normal system id` \
--pairs {TUMOR 3 ID} {NORMAL ID} `# replace tumor 3 system id and normal system id`
```
You can retrieve registered results for the analysis, for instance the *indels* VCF:
```bash
isabl get-results -fi name STRELKA --result-key indels
```
{% hint style="info" %}
To find out what other results are available use:
```bash
# app-primary-key can be retrieved from the frontend
isabl get-results --app-results {app-primary-key}
# when writing this tutorial, the app key for strelka was 5
isabl get-results --app-results 5
```
{% endhint %}
Furthermore, you can get paths for any instance in the database using `get-paths`:
```bash
isabl get-outdirs -fi name STRELKA
```
Lastly, lets check the indels VCFs through the web portal:
## Software Development Kit
To finalize the tutorial, we'll use Isabl as an SDK with `ipython`:
```bash
# we'll access ipython using the cli container
demo-cli ipython
```
Then lets check the output directories of Strelka:
```python
import isabl_cli as ii
# retrieve analyses from API using filters
analyses = ii.get_analyses(name="STRELKA")
# list the strelka ouput directories
for i in analyses:
!ls {i.storage_url}/strelka
```
The analysis objects are [`Munch`](https://github.com/Infinidat/munch), in other words they are dot-dicts (like javascript):
```python
analysis = analyses[0]
# get the target experiment or tumor
target = analysis.targets[0]
# print the parent sample class
print(target.sample.category)
# see available fields
print(target.keys())
```
## Wrap up and Next Steps
Learn about [CLI advanced configuration](https://github.com/isabl-io/docs/tree/c6cbd729fc1d9696332a3c78ea48aa4c7f409066/guides/cli/README.md#configuration) to customize functionality:
{% content-ref url="/pages/-LgzI4B7ORXzpBOP-j8S" %}
[Isabl Settings](/isabl-settings.md)
{% endcontent-ref %}
Learn about writing applications:
{% content-ref url="/pages/-LgZjlyvKfTRM6B0HnMl" %}
[Writing Applications](/writing-applications.md)
{% endcontent-ref %}
Ready for **production**? learn more about [deployment](https://github.com/isabl-io/docs/tree/c6cbd729fc1d9696332a3c78ea48aa4c7f409066/tutorials/deployment/README.md):
{% content-ref url="/pages/-LgZjrA075oF5YZLVGsU" %}
[Production Deployment](/production-deployment.md)
{% endcontent-ref %}
---
# Agent Instructions: Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://docs.isabl.io/quick-start.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.