Importing Data
📦 Learn how to import raw data into Isabl using existing metadata.
Isabl-CLI enables tracking and managing of raw data, as well as reference resources that are a function of a genome assembly or an experimental technique.
Isabl-CLI supports automated data import by recursively exploring data deposition directories and matching raw data files with identifiers registered in the database. For example, the client can be instructed to explore the
/projects directory (A), retrieving only samples from Project 393, and match files using Sample Identifiers (B).Isabl supports automatic import from data deposition directories.
Upon
--commit, Isabl-CLI proceeds to move (or symlink) matched files into scalable directory structures (C). The experiments data path is created by hashing the four last digits of the its primary key. For instance, data for Experiment 57395 will be stored at {storage-directory}/experiments/73/95/57395/. This hashing approach ensures a maximum of 1000 subdirectories in any folder at a worst case scenario of 10 million experiments.Isabl experiments can be linked to any kind of data. Be default Isabl will match the following data types:
RAW_DATA_FORMATS = [
("CRAM", "CRAM"),
("FASTQ_R1", "FASTQ_R1"),
("FASTQ_R2", "FASTQ_R2"),
("FASTQ_I1", "FASTQ_I1"),
("BAM", "BAM"),
("PNG", "PNG"),
("JPEG", "JPEG"),
("TXT", "TXT"),
("TSV", "TSV"),
("CSV", "CSV"),
("PDF", "PDF"),
("DICOM", "DICOM"),
("MD5", "MD5"),
]
If you need to support more raw data formats, adding the EXTRA_RAW_DATA_FORMATS both in the api and client settings, you can extend the valid data format choices in the backend, and provide a new format file validator in the client settings or a new data importer.
ie. to support
MAF format:# In the api settings
EXTRA_RAW_DATA_FORMATS = [("MAF", "MAF")]
# In the cli client settings
EXTRA_RAW_DATA_FORMATS = [("\\.maf(\\.gz)?$", "MAF")]
Tip: subclassing
isabl_cli.data.LocalDataImporter and overwriting RAW_DATA_INSPECTORSmight be enough to support new data formats.Isabl-CLI also supports explicit importing into a single experiment by specifying absolute file paths and metadata in a yaml file via the
import-data-from-yaml command. The metadata will be added to the file_data field in an experiment's raw_data.The two main parameters to be specified when importing are:
- -fi: an argument that takes a pair of values (field, field value) to identify an experiment. For example, if you had an experiment with a
system_idofTEST_EXPERIMENT_T01, the argument would look like:-fi system_id TEST_EXPERIMENT_T01 - --files-data: an argument that takes an absolute file path to the yaml file containing absolute file paths and metadata. For example, if you had a yaml file
/absolute/path/to/files_data.yamlwith the following contents:/absolute/path/to/files_data.yaml/absolute/path/to/file_1.fastq.gz:metadata1: value1metadata2: value2/absolute/path/to/file_2.fastq.gz:metadata3: value3metadata4: value4the argument would look like:--files-data /absolute/path/to/files_data.yaml
Full command using examples above:
isabl import-data-from-yaml \
-fi system_id TEST_EXPERIMENT_T01 \
--files-data /absolute/path/to/files_data.yaml \
--commit
View command details by running:
isabl import-data-from-yaml --helpYou can link reference data to assemblies and techniques. Here are a few ways of how to go about it.
The need to register arbitrary resources for any assembly or technique (e.g. gene annotations) is also supported:
isabl import-reference-data --help
# extra resources are included in the assembly directory
assemblies/
├── GRCh37
│ ├── chr_alias
│ │ └── hg19_alias.tab
│ ├── cytoband
│ │ └── cytoBand.txt
│ ├── genes
│ │ └── refGene.txt
│ └── genome_fasta
│ └── GRCh37.fasta ...
└── GRCm38
Isabl supports the ability to track resources for assemblies and techniques. For instance, ensuring that reference FASTA files are uniformly index, named, and tracked across genome builds:
# indexes are created with `bwa index`, `samtools faidx`, `samtools dict`
isabl import-reference-genome --help
# example of isabl assemblies directories
assemblies/
├── GRCh37
│ └── genome_fasta
│ ├── GRCh37.fasta
│ ├── GRCh37.fasta.amb
│ ├── GRCh37.fasta.ann
│ ├── GRCh37.fasta.bwt
│ ├── GRCh37.fasta.dict
│ ├── GRCh37.fasta.fai
│ ├── GRCh37.fasta.pac
│ └── GRCh37.fasta.sa
└── ...
Lastly, you can register BED files for any sequencing technique, which will be compressed, indexed, moved to the technique data directory, and registered in the database:
# compressed with bgzip, indexed with tabix
isabl import-bedfiles --help
# example of isabl technique directories
techniques/
├── 34
│ └── bed_files
│ └── GRCh37
│ ├── dna-td-hemepact-v4-grch37.baits.bed
│ ├── dna-td-hemepact-v4-grch37.baits.bed.gz
│ ├── dna-td-hemepact-v4-grch37.baits.bed.gz.tbi
│ ├── dna-td-hemepact-v4-grch37.targets.bed
│ ├── dna-td-hemepact-v4-grch37.targets.bed.gz
│ └── dna-td-hemepact-v4-grch37.targets.bed.gz.tbi
└── ...
Imported assets are available for systematic processing by Isabl applications.
All registration mechanisms are configurable and can be customized by providing an alternative python
sub class:Setting Name | Default |
DATA_IMPORTER | isabl_cli.data.LocalDataImporter |
REFERENCE_GENOME_IMPORTER | isabl_cli.data.LocalReferenceGenomeImporter |
REFERENCE_DATA_IMPORTER | isabl_cli.data.LocalReferenceDataImporter |
BED_IMPORTER | isabl_cli.data.LocalBedImporter |
Although only local storage is supported at the time of writing, Isabl-CLI capability can be extrapolated to cloud solutions including integration with cloud workbenches such as Arvados.
Last modified 1yr ago