Retrieving Data

Introduction to Filters

Filters enable you to subset the data of your interest. For example you can use filters to retrieve all the BAM files of a given project, or get all VCFs from a given variant calling application. Filters are field-value pairs and can be used both in the Command Line and within Python. Check out this examples:

curl http://{my-isabl-instance}/api/v1/experiments?sample__identifier={the-sample-id}

isabl get-outdirs -fi application.name BWA_MEM -fi status SUCCEEDED

import isabl_cli as ii
samples = ii.get_instances('samples', individual__species="HUMAN")

Filters Modifiers

As indicated in the previous hint, filter fields can traverse database relationships. However, all filters can be augmented using lookups:

Related Filters

Lookup Types

Datetime Lookups

Full Relational Model

Copy

{related_field}__{related_filter}="query"

Here is a quick representation of Isabl's relational model, hence related filters:

Furthermore, all query parameters in this API support advanced lookup types:

Lookup Type

Description

Example

!

Negate any query

name!=isabel

[i]exact

Exact match

name__exact=isabl name__iexact=IsAbL

[i]contains

Value contains query

name__contains=isa name__icontains=iSa

[i]startswith

Value starts with query

name__startswith=isab name__istartswith=iSab

[i]endswith

Value starts with query

name__endswith=bl name__iendswith=bL

in

Comma separated query

name__in=isabl,besuhof

isnull

Value is null

name__isnull=true

regex

Use regex pattern

name__regex=isabl

gt

Greater than

total__gt=1

gte

Greater or equal

total__gte=1

lt

Less than

total__lt=1

lte

Less than or equal

total__lte=1

Moreover, Datetime query parameters support extra lookups:

Lookup Type

Description

Example

``

No lookup, ISO format required

created=

date

Filter by date YYYY-MM-DD.

created__date=2016-06-04 created__date__gt=2016-06-04

day

Filter by day DD

created__day=04

month

Filter by month MM

created__month=06

year

Filter by year YYYY

created__year=2016

time

Filter by time HH-MM-SS

created__time=21:00:51

Common Filters

Here are some common and useful filters for Isabl.

Limit vs Count Limit

The filter count_limit enables you to limit the total number of instances that will be retrieved. For example to get the output directory for the first 10 successful analyses you could do:

isabl get-outdirs -fi status SUCCEEDED -fi count_limit 10

On the other side, limit will determine how many instances should be retrieved at the same time. For example, the following command would retrieve paths to all successful analyses in batches of 10000:

isabl get-outdirs -fi status SUCCEEDED -fi limit 100000

Has BAM File

To get for example all experiments that have a BAM file for GRCh37 you could do:

experiments = ii.experiments(has_bam_for="GRCh37")

Performance Filters

The following filters can be used to (quite dramatically) improve the performance for some queries:

Isabl Command Line Client

Filters in the command line are usually provided using the -fi or --filters flags. Relations or lookups can be provided using double underscores or dots (e.g. application.name or application__name). Here is a list of Isabl commands available to retrieve information:

Dynamically Explore Metadata

Another useful way to explore the relational model is by using isabl get-metadata:

isabl get-metadata experiments --fx

Furthermore, you can limit the amount of information you are retrieving by passing the list of fields you are interested in:

isabl get-metadata analyses -f application.name -f status

Assembly Resources

By default, the command get-reference helps you retrieve the assembly reference genome.

isabl get-reference GRCh37    # retrieve reference genome

However, by means of the --data-id flag, the command get-reference also allows you to retrieve the indexes generated during import. To get a list of available files per assembly use --resources:

$ isabl get-reference GRCh37 --resources

    genome_fasta         Reference Genome Fasta File.
    genome_fasta_fai     Index generated by: samtools faidx
    ...

Then get the one you are interested in with:

isabl get-reference GRCh37 --data-id genome_fasta_fai

Retrieving Application Results

You can use get-outdirs within the command line to systematically explore output directories. For example:

isabl get-outdirs -fi status FAILED | xargs tree -L 2

Further more you can retrieve files within those directories by using --pattern:

isabl get-outdirs -fi status FAILED --pattern 'head_job.*'

Additionally, you can retrieve results directly registered by the application:

for i in `isabl get-results -fi status FAILED -r command_err`; do
   echo exploring $i;
   cat $i;
done

To visualize what results are available for a given application run:

isabl get-results --app-results <application primary key>

Isabl Software Development Kit

Importantly, isabl-cli can also be used as a Software Development Kit within python:

import isabl_cli as ii  # ii stands for `interactive isabl` 😎

Getting Instances

To get started, we can retrieve specific instances from the database:

# retrieve an experiment with a system id (primary keys also work)
experiment = ii.Experiment("DEM_10000_T01_01_TD1")

# we can also get an analysis using it's primary key (we'll limit retrieved fields)
analysis = ii.Analysis(10235, fields="status,application")

# same thing for assemblies
assembly = ii.Assembly("GRCh37")

A more general way to retrieve any object in the database is using get_instance:

project = ii.get_instance("projects", 100)  # the signature is (endpoint, identifier)

Some examples of things you can do with these instances:

# get the target experiment or tumor
target = analysis.targets[0]

# print the experiment sample class
print(experiment.sample.category)

# see available analysis fields
print(analysis.keys())

# see all available data
print(vars(assembly))

To get multiple instances you can do:

# get all TUMOR experiments in project 102
experiments = ii.get_experiments(projects=102, sample__category="TUMOR")

# get the first 10 SUCCEEDED analyses in the same project
analyses = ii.get_experiments(projects=102, status="SUCCEEDED", count_limit=10)

# get all the projects where I'm the owner
projects = ii.get_projects(owner__startswith="besuhof")

Similarly to isabl get-count , you can determine the number of available results for a given query:

ii.get_instances_count("analyses", status="FAILED")

Getting all Samples from an Individual

To retrieve all samples and experiments for a given individual:

# you can also use the individual system_id
individual = ii.get_tree(10000)

# them all samples are available at
samples = individual.sample_set

# and all experiments for a given sample
experiments = samples[0].experiment_set

You can also retrieve multiple trees:

individuals = ii.get_trees(projects=267)

Create, Delete, and Modify Instances

If you have permissions, you will be able to systematically alter instances in the database:

# create a disease
ii.create_instance("diseases", name="Osteosarcoma", acrynom="OS")

# update an individual's gender
ii.patch_instance(individual.pk, gender="UNKNOWN")

# delete an analysis
ii.delete_instance("analyses", analysis.pk)

Isabl SDK Utils

Here are other useful utilities available in isabl-cli:

Isabl Web

Isabl Web is a great tool to retrieve information and understand the state of affairs within the system. Simply type something in the search bar to retrieve instances across multiple schemas:

Multiple panels will be stacked horizontally as you request more information:

Projects Detail Panel

The projects detail panel conveys all assets and stakeholders linked to a particular project:

Live Tables are directly wired to the API and will enable you to search and filter on specific columns. For the later, simply click in the column name:

The Samples View

The samples tree panel provides access to all assets generated on a given individual.

By clicking on a given node in the tree, you can retrieve more metadata, filter out available analyses on that instance, and even get access to BAM files:

Analyses Results

We can retrieve different types of results for all analyses generated by Isabl. For example accessing a project level quality control report:

Similarly we can retrieve other types of results such as a VCF: